cerebras.modelzoo.data_preparation.nlp.slimpajama.preprocessing.datasets.Dataset#
- class cerebras.modelzoo.data_preparation.nlp.slimpajama.preprocessing.datasets.Dataset[source]#
Bases:
abc.ABC
Methods
Datasets where the source is already shuffled should override this to return True so that it isn't shuffled again.
Path to the directory
A generator producing all documents in the dataset.
Human-readable name of tfhe dataset
num_docs
Path to the file with short documents
Return an estimate of the dataset size.
- documents(process_id, n_process, dup_sh, short_sh)[source]#
A generator producing all documents in the dataset.