composer.datasets.webdataset#

composer.datasets.webdataset

Functions

`WebDataset`	Return a pipeline for WebDataset-style data files.
`create_webdataset`	Write an entire WebDataset to a local directory, given an iterable of samples.
`create_webdatasets_from_image_folder`	Given a directory tree of classified images, create a WebDataset per dataset split.
`load_webdataset`	Load WebDataset from remote, optionally caching, with the given preprocessing and batching.
`pipes`	Capture C-level stdout/stderr in a context manager.

Classes

`ShardWriter`	Like TarWriter but splits into multiple shards.
`tqdm`	Decorate an iterable object, returning an iterator which acts exactly like the original iterable, but prints a dynamically updating progressbar every time a value is requested.

Attributes

Any
Dict
Iterable
List
Optional
TYPE_CHECKING
Tuple
Union
log
webdataset_installed

composer.datasets.webdataset.create_webdataset(samples, dataset_dir, split, n_samples, n_shards, use_tqdm=True)[source]#

Write an entire WebDataset to a local directory, given an iterable of samples.

Parameters

samples (iterable of dict) – Each dataset sample.
dataset_dir (str) – Output dataset directory.
split (str) – Dataset split.
n_samples (int) – Number of samples in dataset.
n_shards (int) – Number of full shards to write (may write a leftovers shard).
use_tqdm (bool) – Whether to show progress with tqdm.

composer.datasets.webdataset.create_webdatasets_from_image_folder(in_root, out_root, n_shards, use_tqdm=True)[source]#

Given a directory tree of classified images, create a WebDataset per dataset split.

Directory tree format: (path to dataset)/(split name)/(class name)/(image file).

Parameters

in_root (str) – Input dataset root.
out_root (str) – Output WebDataset root.
n_shards (int) – Number of full shards to write (may write a leftovers shard).
use_tqdm (bool) – Whether to show progress with tqdm.

composer.datasets.webdataset.load_webdataset(remote, name, split, cache_dir, cache_verbose, shuffle, shuffle_buffer, preprocess, n_devices, workers_per_device, batch_size, drop_last)[source]#

Load WebDataset from remote, optionally caching, with the given preprocessing and batching.

Parameters

remote (str) – Remote path (either an s3:// url or a directory on local filesystem).
name (str) – Name of this dataset, used to locate dataset in local cache.
cache_dir (str, optional) – Root directory of local filesystem cache.
cache_verbose (bool) – WebDataset caching verbosity.
shuffle (bool) – Whether to shuffle samples.
shuffle_buffer (int) – How many samples to buffer when shuffling.
preprocess (Callable) – What transformations to apply to the samples, as WebDataset iterator(s).
n_devices (int) – Number of devices.
workers_per_device (int) – Number of workers per device.
batch_size (int) – Batch size.
drop_last (bool) – Whether to drop partial last batches.