cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.verify_saved_hdf5_files_mp#

cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.verify_saved_hdf5_files_mp(files, args, vocab_size)[source]#

Verify the generated HDF5 dataset. :param files: List of files to process. :type files: list :param args: Arguments for verifying HDF5 dataset. :type args: VerificationArgs :param vocab_size: Size of the vocabulary from data_processor. :type vocab_size: int

previous

cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.verify_saved_hdf5_files

next

cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.wikitext_detokenizer