cerebras.modelzoo.tools.checkpoint_converters.streaming_checkpoints.StreamingShardedHFReader#
- class cerebras.modelzoo.tools.checkpoint_converters.streaming_checkpoints.StreamingShardedHFReader(index_file)[source]#
Bases:
object
Allows sharded HuggingFace checkpoints to be read in a streaming manner rather than loading all shards into memory all at once. The underlying checkpoint is read-only.
Only one shard is stored into memory at a time. For this reason, accessing random keys may slow due to the switching cost (loading) between shards. For this reason, it is recommend that keys are accessed in the order given by self.keys() or self.__iter__() as keys that appear in the same shard are in consecutive order.
- Parameters
index_file (str) – Path to .index.json file.
Methods
items
keys
load_shard
values