cerebras.modelzoo.data.nlp.gpt.InferenceDataProcessor#
Functions
Get encoded token ids from a string using the specified tokenizer. |
|
Helper to construct a list of stop token sequences from the given list of stop words using the specified tokenizer. |
Classes
Subclass for processing BigCode data, i.e. bigcode_eh requests. |
|
Subclass for processing EEH generate_until requests. |
|
Subclass for processing EEH loglikelihood requests. |
|
An enumeration. |