composer.datasets.ffcv_utils#
composer.datasets.ffcv_utils
Functions
|
composer.datasets.ffcv_utils.ffcv_monkey_patches |
Converts PyTorch compatible |
- composer.datasets.ffcv_utils.write_ffcv_dataset(dataset=None, write_path='/tmp/dataset.ffcv', max_resolution=None, num_workers=16, write_mode='raw', compress_probability=0.5, jpeg_quality=90, chunk_size=100)[source]#
Converts PyTorch compatible
dataset
into FFCV format at filepathwrite_path
.- Parameters
dataset (Iterable[Sample]) โ A PyTorch dataset. Default:
None
.write_path (str) โ Write results to this file. Default:
"/tmp/dataset.ffcv"
.max_resolution (int) โ Limit resolution if provided. Default:
None
.num_workers (int) โ Numbers of workers to use. Default:
16
.write_mode (str) โ Write mode for the dataset. Default:
'raw'
.compress_probability (float) โ Probability with which image is JPEG-compressed. Default:
0.5
.jpeg_quality (float) โ Quality to use for jpeg compression. Default:
90
.chunk_size (int) โ Size of chunks processed by each worker during conversion. Default:
100
.