cerebras.modelzoo.data_preparation.data_preprocessing.data_preprocessor#

This module implements a generic data preprocessor called DataPreprocessor. It internally uses DataFrame and DataReader to read and process data.

Functions

check_and_create_dir

Ensures a directory exists, optionally handling a subdirectory.

default_hook

format_time

Format seconds into a human-readable string showing hours:minutes:seconds, minutes:seconds, or seconds.

get_available_memory

Returns available memory in bytes.

get_compression_factor

Calculate and return the compression factor based on a file's extension.

save_image_locally

update_progress

Update the progress bar based on the current progress.

Classes

DataPreprocessor

Initialize the class with given parameters.