download#
Download handling for StreamingDataset
.
Functions
Downloads a file from remote to local, or waits for it to be downloaded. |
- composer.datasets.streaming.download.dispatch_download(remote, local)[source]#
Use the correct download handler to download the file
- composer.datasets.streaming.download.download_from_http(remote, local)[source]#
Download a file from a http/https remote to local.
- composer.datasets.streaming.download.download_from_local(remote, local)[source]#
Download a file from remote to local.
- composer.datasets.streaming.download.download_or_wait(remote, local, wait=False, max_retries=2, timeout=60)[source]#
Downloads a file from remote to local, or waits for it to be downloaded.
Does not do any thread safety checks, so we assume the calling function is using
wait
correctly.- Parameters
remote (Optional[str]) โ Remote path (S3, SFTP, or local filesystem).
local (str) โ Local path (local filesystem).
wait (bool, default False) โ If
true
, then do not actively download the file, but instead wait (up totimeout
seconds) for the file to arrive.max_retries (int, default 2) โ Number of download re-attempts before giving up.
timeout (float, default 60) โ How long to wait for file to download before raising an exception.