composer.datasets.hparams#
Dataset Hyperparameter classes.
Hparams
These classes are used with yahp
for YAML
-based configuration.
Abstract base class for hyperparameters to initialize a dataset. |
|
Synthetic dataset parameter mixin for |
- class composer.datasets.hparams.DatasetHparams(is_train=True, drop_last=True, shuffle=True, datadir=None)[source]#
Bases:
yahp.hparams.Hparams
,abc.ABC
Abstract base class for hyperparameters to initialize a dataset.
- Parameters
datadir (str) โ The path to the data directory.
is_train (bool) โ Whether to load the training data or validation data. Default:
True
.drop_last (bool) โ If the number of samples is not divisible by the batch size, whether to drop the last batch or pad the last batch with zeros. Default:
True
.shuffle (bool) โ Whether to shuffle the dataset. Default:
True
.
- abstract initialize_object(batch_size, dataloader_hparams)[source]#
Creates a
DataLoader
orDataSpec
for this dataset.- Parameters
batch_size (int) โ The size of the batch the dataloader should yield. This batch size is device-specific and already incorporates the world size.
dataloader_hparams (DataLoaderHparams) โ The dataset-independent hparams for the dataloader.
- Returns
DataLoader or DataSpec โ The
DataLoader
, or if the dataloader yields batches of custom types, aDataSpec
.
- class composer.datasets.hparams.SyntheticHparamsMixin(use_synthetic=False, synthetic_num_unique_samples=100, synthetic_device='cpu', synthetic_memory_format=MemoryFormat.CONTIGUOUS_FORMAT)[source]#
Bases:
yahp.hparams.Hparams
,abc.ABC
Synthetic dataset parameter mixin for
DatasetHparams
.- Parameters
use_synthetic (bool, optional) โ Whether to use synthetic data. Default:
False
.synthetic_num_unique_samples (int, optional) โ The number of unique samples to allocate memory for. Ignored if
use_synthetic
isFalse
. Default:100
.synthetic_device (str, optional) โ The device to store the sample pool on. Set to
'cuda'
to store samples on the GPU and eliminate PCI-e bandwidth with the dataloader. Set to'cpu'
to move data between host memory and the device on every batch. Ignored ifuse_synthetic
isFalse
. Default:'cpu'
.synthetic_memory_format โ The
MemoryFormat
to use. Ignored ifuse_synthetic
isFalse
. Default:'CONTIGUOUS_FORMAT'
.
- class composer.datasets.hparams.WebDatasetHparams(is_train=True, drop_last=True, shuffle=True, datadir=None, webdataset_cache_dir='/tmp/webdataset_cache/', webdataset_cache_verbose=False, shuffle_buffer=256)[source]#
Bases:
composer.datasets.hparams.DatasetHparams
,abc.ABC
Abstract base class for hyperparameters to initialize a webdataset.
- Parameters
datadir (str) โ The path to the data directory.
is_train (bool) โ Whether to load the training data or validation data. Default:
True
.drop_last (bool) โ If the number of samples is not divisible by the batch size, whether to drop the last batch or pad the last batch with zeros. Default:
True
.shuffle (bool) โ Whether to shuffle the dataset. Default:
True
.webdataset_cache_dir (str) โ WebDataset cache directory.
webdataset_cache_verbose (str) โ WebDataset cache verbosity.