cerebras.modelzoo.data_preparation.huggingface.CSDataCollatorForLanguageModeling.CSDataCollatorForLanguageModeling#

class cerebras.modelzoo.data_preparation.huggingface.CSDataCollatorForLanguageModeling.CSDataCollatorForLanguageModeling(*args, **kwargs)[source]#

Bases: transformers.DataCollatorForLanguageModeling

Overrides DataCollatorForLanguageModeling from HF to shift the inputs/labels in the dataloader

Methods

torch_call

previous

cerebras.modelzoo.data_preparation.huggingface.CSDataCollatorForLanguageModeling

next

cerebras.modelzoo.data_preparation.huggingface.HF_converter_example_BookCorpus