cerebras.modelzoo.data_preparation.data_preprocessing.data_reader.DataFrame#

class cerebras.modelzoo.data_preparation.data_preprocessing.data_reader.DataFrame(keys=None, format_hook_fn=None)[source]#

Bases: object

Initialize the DataFrame object.

Parameters

keys (Dict) – Keys for the data entries.

Methods

add

Add an entry to the DataFrame.

clear

Clear the raw data after tokenizing.

tokenize

Tokenize the data values.

add(value)[source]#

Add an entry to the DataFrame.

Parameters

value (Union[Dict[str, Any], Any]) – Entry to be added.

clear()[source]#

Clear the raw data after tokenizing.

tokenize(token_generator)[source]#

Tokenize the data values.

Parameters

token_generator (Any) – Token generator to be used for processing the data.