composer.datasets.ffcv_utils#

composer.datasets.ffcv_utils

Functions

ffcv_monkey_patches

composer.datasets.ffcv_utils.ffcv_monkey_patches

write_ffcv_dataset

Converts PyTorch dataset or webdataset at remote into FFCV format at filepath write_path.
composer.datasets.ffcv_utils.write_ffcv_dataset(dataset=None, remote=None, write_path='/tmp/dataset.ffcv', max_resolution=None, num_workers=16, write_mode='raw', compress_probability=0.5, jpeg_quality=90, chunk_size=100)[source]#

Converts PyTorch dataset or webdataset at remote into FFCV format at filepath write_path.

Parameters

  • dataset (Iterable[Sample]) โ€“ A PyTorch dataset. Default: None.

  • remote (str) โ€“ A remote path for webdataset. Default: None.

  • write_path (str) โ€“ Write results to this file. Default: "/tmp/dataset.ffcv".

  • max_resolution (int) โ€“ Limit resolution if provided. Default: None.

  • num_workers (int) โ€“ Numbers of workers to use. Default: 16.

  • write_mode (str) โ€“ Write mode for the dataset. Default: 'raw'.

  • compress_probability (float) โ€“ Probability with which image is JPEG-compressed. Default: 0.5.

  • jpeg_quality (float) โ€“ Quality to use for jpeg compression. Default: 90.

  • chunk_size (int) โ€“ Size of chunks processed by each worker during conversion. Default: 100.