cerebras.modelzoo.data.nlp.bert.BertSumCSVDataProcessor.BertSumCSVDataProcessor#
- class cerebras.modelzoo.data.nlp.bert.BertSumCSVDataProcessor.BertSumCSVDataProcessor(*args, **kwargs)[source]#
Bases:
torch.utils.data.IterableDataset
Reads csv file containing the input_token_ids, and label_ids. Creates attention_mask and segment_ids on the fly :param <dict> params: dict containing input parameters for creating dataset.
Methods
Classmethod to create the dataloader object.
Generator to read the data in chunks of size of data_buffer.