StreamingC4

class composer.datasets.StreamingC4(remote, local, split, shuffle, tokenizer_name, max_seq_len, group_method='truncate', max_retries=2, timeout=120, batch_size=None)

Implementation of the C4 (Colossal Cleaned Common Crawl) dataset using StreamingDataset.

Parameters

• remote (str) – Remote directory (S3 or local filesystem) where dataset is stored.

• local (str) – Local filesystem directory where dataset is cached during operation.

• split (str) – The dataset split to use, either ‘train’ or ‘val’.

• shuffle (bool) – Whether to shuffle the samples in this dataset.

• tokenizer_name (str) – The name of the HuggingFace tokenizer to use to tokenize samples.

• max_seq_len (int) – The max sequence length of each token sample.

• group_method (str) – How to group text samples into token samples. Currently only supporting 'truncate'.

• max_retries (int) – Number of download re-attempts before giving up. Default: 2.

• timeout (float) – How long to wait for shard to download before raising an exception. Default: 120 sec.

• batch_size (Optional[int]) – Hint batch_size that will be used on each device’s DataLoader. Default: None.