cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing#

convert_dataset_to_HDF5

create_hdf5_dataset

Script that generates a dataset in HDF5 format for GPT Models.

hdf5_base_preprocessor

hdf5_curation_corpus_preprocessor

hdf5_dataset_preprocessors

hdf5_nlg_preprocessor

utils