cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.verify_saved_hdf5_files#
- cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.verify_saved_hdf5_files(params)[source]#
This function is used to do sanity checks at the end of the creation of hdf5 files. This function loads every .h5 files generated and checks:
The data type
Shape of the dataset
Fact that labels and inputs are as expected