cerebras.modelzoo.data_preparation.huggingface#

CSDataCollatorForLanguageModeling

HF_converter_example_BookCorpus

Example script to convert HF BookCorpus dataset to HDF5 dataset

HF_converter_example_Eli5

Example script to convert HF Eli5 dataset to HDF5 dataset

HuggingFaceDataProcessor

Pytorch HuggingFace Dataloader

HuggingFace_BookCorpus

HuggingFace BookCorpus Dataset

HuggingFace_Eli5

HuggingFace Eli5 Dataset