StreamingImageNet1k

class composer.datasets.StreamingImageNet1k(remote, local, split, shuffle, resize_size=- 1, crop_size=224, batch_size=None)

Implementation of the ImageNet1k dataset using StreamingDataset.

Parameters

• remote (str) – Remote directory (S3 or local filesystem) where dataset is stored.

• local (str) – Local filesystem directory where dataset is cached during operation.

• split (str) – The dataset split to use, either ‘train’ or ‘val’.

• shuffle (bool) – Whether to shuffle the samples in this dataset.

• resize_size (int, optional) – The resize size to use. Use -1 to not resize. Default: -1.

• size (crop) – The crop size to use. Default: 224.

• batch_size (Optional[int]) – Hint batch_size that will be used on each device’s DataLoader. Default: None.

decode_class(data)

Decode the sample class.

Parameters

data (bytes) – The raw bytes.

Returns

np.int64 – The class encoded by the bytes.

decode_image(data)

Decode the sample image.

Parameters

data (bytes) – The raw bytes.

Returns

Image – PIL image encoded by the bytes.