cerebras.modelzoo.data_preparation.data_preprocessing.utils.wikitext_detokenizer#

cerebras.modelzoo.data_preparation.data_preprocessing.utils.wikitext_detokenizer(string)[source]#

Detokenizer for wikitext. Used for special handling of data for substrings.

Parameters

string (str) – String to detoknize before tokenization.

Returns

Detokenized string

previous

cerebras.modelzoo.data_preparation.data_preprocessing.utils.validate_tokens

next

cerebras.modelzoo.data_preparation.data_preprocessing.vsl_finetuning_token_generator