cerebras.modelzoo.data_preparation.nlp.pubmed.TextFormatting.TextFormatting#

class cerebras.modelzoo.data_preparation.nlp.pubmed.TextFormatting.TextFormatting(pubmed_path, output_filename, filesize_limit=5000000000, recursive=False)[source]#

Bases: object

Parameters

pubmed_path (str) – Path to folder containing PubMed files

:param str output_folder : Path to where the txt file to be written :param Optional[int] filesize_limit: Max size of each text file :param Optional[bool] recursive: Flag if true, searches for nxml/xml files recursively within subfolders

Methods

merge

merge_abstracts

merge_fulltext

previous

cerebras.modelzoo.data_preparation.nlp.pubmed.TextFormatting

next

cerebras.modelzoo.data_preparation.nlp.pubmed.TextSharding