cerebras.modelzoo.data_preparation.data_preprocessing.hooks.nlg_read_hook#

cerebras.modelzoo.data_preparation.data_preprocessing.hooks.nlg_read_hook(example, **read_hook_kwargs)[source]#

Transforms natural language generation (NLG) data for reading.

Parameters

  • example (Dict[str, Any]) – The input data containing NLG information.

  • **read_hook_kwargs (Any) – Additional keyword arguments containing data_keys.

Returns

Transformed data suitable for reading.

Return type

List[Dict[str, Any]]

Raises

AssertionError – If required keys are not provided in read_hook_kwargs.

previous

cerebras.modelzoo.data_preparation.data_preprocessing.hooks.finetuning_llava_hook_prompt_completion

next

cerebras.modelzoo.data_preparation.data_preprocessing.hooks.pretraining_image_captions_hook