cerebras.modelzoo.data_preparation.data_preprocessing.pretraining_token_generator.PretrainingTokenGenerator#
- class cerebras.modelzoo.data_preparation.data_preprocessing.pretraining_token_generator.PretrainingTokenGenerator(params, tokenizer, eos_id, pad_id)[source]#
Bases:
object
Initialize the PretrainingTokenGenerator class.
- Parameters
params (Dict[str, Any]) – Parameters for the dataset and processing.
tokenizer (Any) – Tokenizer to use for tokenization.
eos_id (int) – End-of-sequence token ID.
pad_id (int) – Padding token ID.
Methods
Clean the provided text.
Tokenize and encode the data for auto-regressive language modeling.
Processes the leftover prefix which is a list of ndarray tokens into chunks based on max sequence length.
Generate a list of token IDs that can be masked.
Get data statistics from the sample.
Masks tokens in a single sequence according to the MLM strategy.
Parse semantic data dictionary.
Processes chunks of tokenized text and returns processed features along with the total padding added.
Processes chunks of tokenized text and returns processed features along with the total padding added.
Tokenize the text and create features for auto-regressive language modeling.
- get_data_stats(sample, lvt=None)[source]#
Get data statistics from the sample.
- Parameters
sample (np.ndarray) – Tokenized sample.
- Returns
Data statistics.
- Return type
Dict[str, int]
- mask_single_sequence(input_ids)[source]#
Masks tokens in a single sequence according to the MLM strategy. When self.mlm_with_gather is False, the returning len(labels) == len(input_ids) When self.mlm_with_gather is True, the returning len(labels) == self.max_predictions
- Parameters
input_ids (List[int]) – Original sequence of token IDs.
- Returns
input_ids: Modified sequence with masked tokens.
masked_lm_positions: Positions of the masked tokens, empty if not self.mlm_with_gather.
masked_lm_mask: Binary indicators (1s) for positions that were masked, empty if not self.mlm_with_gather.
labels: Original token IDs of the masked tokens for label purposes.
- Return type
Tuple[List[int], List[int], List[int], List[int]]
- process_chunks(tokenized_text_chunks)[source]#
Processes chunks of tokenized text and returns processed features along with the total padding added.
- Parameters
tokenized_text_chunks (List[List[int]]) – A list of tokenized text chunks, where each chunk is represented as a list of integers.
- Returns
A tuple containing a list of processed results and dataset stats.
- Return type
Tuple[List[np.ndarray], Dict[str, int]]
- process_chunks_mlm(tokenized_text_chunks)[source]#
Processes chunks of tokenized text and returns processed features along with the total padding added.
Args: tokenized_text_chunks (List[List[int]]): A list of tokenized text chunks, where each chunk is represented as a list of integers.
Returns: Tuple[List[Any], Dict]: A tuple containing a list of processed results and dataset stats.
- clean_text(data)[source]#
Clean the provided text.
- Parameters
data (str) – Text to clean.
- Returns
Cleaned text.
- Return type
str
- tokenize_data(semantic_data_array)[source]#
Tokenize the text and create features for auto-regressive language modeling.
- Parameters
semantic_data_dict (Union[Dict[str, Any], List[Dict[str, Any]]]) – Data to tokenize.
- Returns
Tuple of encoded features for auto-regressive language modeling and dataset stats.
- Return type
Tuple[List[np.ndarray], Dict[str, int]]
- parse_semantic_data_array(semantic_data_array)[source]#
Parse semantic data dictionary.
- Parameters
entry (Union[Dict[str, Any], List[Dict[str, Any]]]) – Data entry.
- Returns
Parsed text and raw data statistics.
- Return type
Tuple[str, Dict[str, int]]
- encode(semantic_data_array)[source]#
Tokenize and encode the data for auto-regressive language modeling.
- Parameters
semantic_data_array (Union[Dict[str, Any], List[Dict[str, Any]]]) – Data to encode.
- Returns
Tuple of encoded features for auto-regressive language modeling and dataset stats.
- Return type
Tuple[Dict[str, Any], Dict[str, int]]
- encode_leftover_prefix(prefix)[source]#
Processes the leftover prefix which is a list of ndarray tokens into chunks based on max sequence length.
The last chunk is handled specifically if it’s shorter than the max sequence length. If the last chunk has less than two tokens, it’s discarded.
- Parameters
prefix (List[np.ndarray]) – The prefix list of token arrays to process.
- Returns
A tuple containing the processed token chunks as a list of ndarrays and the dataset stats.
- Return type
Tuple[Dict[str, Any], Dict[str, int]]