composer.datasets.ade20k#
ADE20K Semantic segmentation and scene parsing dataset.
Please refer to the ADE20K dataset for more details about this dataset.
Classes
PyTorch Dataset for ADE20k. |
Hparams
These classes are used with yahp
for YAML
-based configuration.
Defines an instance of the ADE20k dataset for semantic segmentation from a local disk. |
|
Defines an instance of the ADE20k dataset for semantic segmentation from a remote blob store. |
- class composer.datasets.ade20k.ADE20k(datadir, split='train', both_transforms=None, image_transforms=None, target_transforms=None)[source]#
Bases:
torch.utils.data.dataset.Dataset
PyTorch Dataset for ADE20k.
- Parameters
datadir (str) โ the path to the ADE20k folder.
split (str) โ the dataset split to use, either โtrainโ, โvalโ, or โtestโ. Default:
'train'
.both_transforms (Module) โ transformations to apply to the image and target simultaneously. Default:
None
.image_transforms (Module) โ transformations to apply to the image only. Default:
None
.target_transforms (Module) โ transformations to apply to the target only. Default
None
.
- class composer.datasets.ade20k.ADE20kDatasetHparams(use_synthetic=False, synthetic_num_unique_samples=100, synthetic_device='cpu', synthetic_memory_format=MemoryFormat.CONTIGUOUS_FORMAT, is_train=True, drop_last=True, shuffle=True, datadir=None, split='train', base_size=512, min_resize_scale=0.5, max_resize_scale=2.0, final_size=512, ignore_background=True)[source]#
Bases:
composer.datasets.hparams.DatasetHparams
,composer.datasets.hparams.SyntheticHparamsMixin
Defines an instance of the ADE20k dataset for semantic segmentation from a local disk.
- Parameters
use_synthetic (bool, optional) โ Whether to use synthetic data. Default:
False
.synthetic_num_unique_samples (int, optional) โ The number of unique samples to allocate memory for. Ignored if
use_synthetic
isFalse
. Default:100
.synthetic_device (str, optional) โ The device to store the sample pool on. Set to
'cuda'
to store samples on the GPU and eliminate PCI-e bandwidth with the dataloader. Set to'cpu'
to move data between host memory and the device on every batch. Ignored ifuse_synthetic
isFalse
. Default:'cpu'
.synthetic_memory_format โ The
MemoryFormat
to use. Ignored ifuse_synthetic
isFalse
. Default:'CONTIGUOUS_FORMAT'
.datadir (str) โ The path to the data directory.
is_train (bool) โ Whether to load the training data or validation data. Default:
True
.drop_last (bool) โ If the number of samples is not divisible by the batch size, whether to drop the last batch or pad the last batch with zeros. Default:
True
.shuffle (bool) โ Whether to shuffle the dataset. Default:
True
.split (str) โ the dataset split to use either โtrainโ, โvalโ, or โtestโ. Default:
'train`
.base_size (int) โ initial size of the image and target before other augmentations. Default:
512
.min_resize_scale (float) โ the minimum value the samples can be rescaled. Default:
0.5
.max_resize_scale (float) โ the maximum value the samples can be rescaled. Default:
2.0
.final_size (int) โ the final size of the image and target. Default:
512
.ignore_background (bool) โ if true, ignore the background class when calculating the training loss. Default:
true
.
- initialize_object(batch_size, dataloader_hparams)[source]#
Creates a
DataLoader
orDataSpec
for this dataset.- Parameters
batch_size (int) โ The size of the batch the dataloader should yield. This batch size is device-specific and already incorporates the world size.
dataloader_hparams (DataLoaderHparams) โ The dataset-independent hparams for the dataloader.
- Returns
DataLoader or DataSpec โ The
DataLoader
, or if the dataloader yields batches of custom types, aDataSpec
.
- class composer.datasets.ade20k.ADE20kWebDatasetHparams(is_train=True, drop_last=True, shuffle=True, datadir=None, webdataset_cache_dir='/tmp/webdataset_cache/', webdataset_cache_verbose=False, shuffle_buffer=256, remote='s3://mosaicml-internal-dataset-ade20k', name='ade20k', split='train', base_size=512, min_resize_scale=0.5, max_resize_scale=2.0, final_size=512, ignore_background=True)[source]#
Bases:
composer.datasets.hparams.WebDatasetHparams
Defines an instance of the ADE20k dataset for semantic segmentation from a remote blob store.
- Parameters
datadir (str) โ The path to the data directory.
is_train (bool) โ Whether to load the training data or validation data. Default:
True
.drop_last (bool) โ If the number of samples is not divisible by the batch size, whether to drop the last batch or pad the last batch with zeros. Default:
True
.shuffle (bool) โ Whether to shuffle the dataset. Default:
True
.datadir โ The path to the data directory.
is_train โ Whether to load the training data or validation data. Default:
True
.drop_last โ If the number of samples is not divisible by the batch size, whether to drop the last batch or pad the last batch with zeros. Default:
True
.shuffle โ Whether to shuffle the dataset. Default:
True
.webdataset_cache_dir (str) โ WebDataset cache directory.
webdataset_cache_verbose (str) โ WebDataset cache verbosity.
remote (str) โ S3 bucket or root directory where dataset is stored. Default:
's3://mosaicml-internal-dataset-ade20k'
name (str) โ Key used to determine where dataset is cached on local filesystem. Default:
'ade20k'
split (str) โ the dataset split to use either โtrainโ, โvalโ, or โtestโ. Default:
'train'
.base_size (int) โ initial size of the image and target before other augmentations. Default:
512
.min_resize_scale (float) โ the minimum value the samples can be rescaled. Default:
0.5
.max_resize_scale (float) โ the maximum value the samples can be rescaled. Default:
2.0
.final_size (int) โ the final size of the image and target. Default:
512
.ignore_background (bool) โ if true, ignore the background class when calculating the training loss. Default:
True
.
- initialize_object(batch_size, dataloader_hparams)[source]#
Creates a
DataLoader
orDataSpec
for this dataset.- Parameters
batch_size (int) โ The size of the batch the dataloader should yield. This batch size is device-specific and already incorporates the world size.
dataloader_hparams (DataLoaderHparams) โ The dataset-independent hparams for the dataloader.
- Returns
DataLoader or DataSpec โ The
DataLoader
, or if the dataloader yields batches of custom types, aDataSpec
.
- class composer.datasets.ade20k.PadToSize(size, fill=0)[source]#
Bases:
torch.nn.modules.module.Module
Pad an image to a specified size.
- class composer.datasets.ade20k.PhotometricDistoration(brightness, contrast, saturation, hue)[source]#
Bases:
torch.nn.modules.module.Module
Applies a combination of brightness, contrast, saturation, and hue jitters with random intensity.
This is a less severe form of PyTorchโs ColorJitter used by the mmsegmentation library here: https://github.com/open-mmlab/mmsegmentation/blob/master/mmseg/datasets/pipelines/transforms.py#L837
- class composer.datasets.ade20k.RandomCropPair(crop_size, class_max_percent=1.0, num_retry=1)[source]#
Bases:
torch.nn.modules.module.Module
Crop the image and target at a randomly sampled position.
- Parameters
crop_size (Tuple[int, int]) โ the size (height x width) of the crop.
class_max_percent (float) โ the maximum percent of the image area a single class should occupy. Default is 1.0.
num_retry (int) โ the number of times to resample the crop if
class_max_percent
threshold is not reached. Default is 1.
- class composer.datasets.ade20k.RandomHFlipPair(probability=0.5)[source]#
Bases:
torch.nn.modules.module.Module
Flip the image and target horizontally with a specified probability.
- Parameters
probability (float) โ the probability of flipping the image and target. Default:
0.5
.
- class composer.datasets.ade20k.RandomResizePair(min_scale, max_scale, base_size=None)[source]#
Bases:
torch.nn.modules.module.Module
Resize the image and target to
base_size
scaled by a randomly sampled value.- Parameters
min_scale (float) โ the minimum value the samples can be rescaled.
max_scale (float) โ the maximum value the samples can be rescaled.
base_size (Tuple[int, int]) โ a specified base size (height x width) to scale to get the resized dimensions. When this is None, use the input image size. Default:
None
.