cerebras.modelzoo.data_preparation.utils.whitespace_tokenize

cerebras.modelzoo.data_preparation.utils.whitespace_tokenize(text, lower=False)

Splits a piece of text based on whitespace characters

previous

cerebras.modelzoo.data_preparation.utils.text_to_tokenized_documents

next

cerebras.modelzoo.data_preparation.utils.maskedLmInstance