cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.create_features_auto_lm#

cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.create_features_auto_lm(token_ids, max_sequence_length, short_seq_prob=0, inverted_mask=False, pad_id=0, min_len=10, input_ids_dtype='int32', input_mask_dtype='int32', labels_dtype='int32', rng=None)[source]#

Given a list of token_ids, generate input sequence and labels.

Parameters
  • token_ids (sequence) – List containing token ids for creating features, labels and input mask from.

  • max_sequence_length (int) – Maximum sequence length for data writes.

  • short_seq_prob (float) – Probability of generating short sequences from data. Defaults to 0.

  • inverted_mask (bool) – Invert mask if specified for runtime execution. Defaults to False.

  • min_len (int) – Minimum length of token_ids to be considered a valid sequence.

  • pad_id (int) – Id for pad token. Defaults to 0.

  • input_ids_dtype (str) – Dtype as string for input ids. Defaults to int32.

  • input_mask_dtype (str) – Dtype as string for input mask. Defaults to int32.

  • labels_dtype (str) – Dtype as string for labels. Defaults to int32.

  • rng (random.Random obj) – Instance of random object, with states set. Defaults to None.

Returns

Tuple containing features and labels