cerebras.modelzoo.data_preparation.nlp.slimpajama.dedup.generate_duplicate_pairs.get_hashes

cerebras.modelzoo.data_preparation.nlp.slimpajama.dedup.generate_duplicate_pairs.get_hashes(files, doc_queues, r)

previous

cerebras.modelzoo.data_preparation.nlp.slimpajama.dedup.generate_duplicate_pairs.generate_pairs

next

cerebras.modelzoo.data_preparation.nlp.slimpajama.dedup.generate_duplicate_pairs.lsh