composer.datasets.streaming.download#
Download handling for StreamingDataset
.
Functions
Downloads a file from remote to local, or waits for it to be downloaded. |
- composer.datasets.streaming.download.dispatch_download(remote, local, timeout)[source]#
Use the correct download handler to download the file
- composer.datasets.streaming.download.download_from_local(remote, local)[source]#
Download a file from remote to local.
- composer.datasets.streaming.download.download_from_s3(remote, local, timeout)[source]#
Download a file from remote to local.
- composer.datasets.streaming.download.download_from_sftp(remote, local)[source]#
Download a file from remote SFTP server to local filepath.
Authentication must be provided via username/password in the
remote
URI, or a valid SSH config, or a default key discoverable in~/.ssh/
.
- composer.datasets.streaming.download.download_or_wait(remote, local, wait=False, max_retries=2, timeout=60)[source]#
Downloads a file from remote to local, or waits for it to be downloaded.
Does not do any thread safety checks, so we assume the calling function is using
wait
correctly.- Parameters
remote (Optional[str]) โ Remote path (S3, SFTP, or local filesystem).
local (str) โ Local path (local filesystem).
wait (bool, default False) โ If
true
, then do not actively download the file, but instead wait (up totimeout
seconds) for the file to arrive.max_retries (int, default 2) โ Number of download re-attempts before giving up.
timeout (float, default 60) โ How long to wait for file to download before raising an exception.