cerebras.modelzoo.data_preparation.nlp.pile.download#
Functions
Download a single file from url to specified filepath. |
|
Download The Pile dataset from eye.ai website. |
|
Download files needed for tokenization for dataset creation. |
|
Get urls for downloading files for tokenization. |
|
Get urls given split of dataset. |
|
Main function for execution. |
|
Argparser definition for command line arguments from user. |