StreamingCIFAR10#

class composer.datasets.StreamingCIFAR10(remote, local, split, shuffle, batch_size=None)[source]#

Implementation of the CIFAR10 dataset using StreamingDataset.

Parameters

  • remote (str) โ€“ Remote directory (S3 or local filesystem) where dataset is stored.

  • local (str) โ€“ Local filesystem directory where dataset is cached during operation.

  • split (str) โ€“ The dataset split to use, either โ€˜trainโ€™ or โ€˜valโ€™.

  • shuffle (bool) โ€“ Whether to shuffle the samples in this dataset.

  • batch_size (Optional[int]) โ€“ Hint batch_size that will be used on each deviceโ€™s DataLoader. Default: None.

decode_class(data)[source]#

Decode the sample class.

Parameters

data (bytes) โ€“ The raw bytes.

Returns

np.int64 โ€“ The class encoded by the bytes.

decode_image(data)[source]#

Decode the sample image.

Parameters

data (bytes) โ€“ The raw bytes.

Returns

Image โ€“ PIL image encoded by the bytes.