composer.utils.file_retriever

Utility for retrieving files.

Functions

get_file

Get a file from a local folder, URL, or object store.

Exceptions

GetFileNotFoundException

Exception if get_file() failed due to a not found error.

exception composer.utils.file_retriever.GetFileNotFoundException

Bases: RuntimeError

Exception if get_file() failed due to a not found error.

composer.utils.file_retriever.get_file(path, destination, object_store=None, chunk_size=1048576, progress_bar=True)

Get a file from a local folder, URL, or object store.

Parameters

• path (str) –

  The path to the file to retreive.

  • If object_store is specified, then the path should be the object name for the file to get. Do not include the the cloud provider or bucket name.

  • If object_store is not specified but the path begins with http:// or https://, the object at this URL will be downloaded.

  • Otherwise, path is presumed to be a local filepath.

• destination (str) –

  The destination filepath.

  If path is a local filepath, then a symlink to path at destination will be created. Otherwise, path will be downloaded to a file at destination.

• object_store (ObjectStoreProvider, optional) –

  An ObjectStoreProvider, if path is located inside an object store (i.e. AWS S3 or Google Cloud Storage). (default: None)

  This ObjectStoreProvider instance will be used to retreive the file. The path parameter should be set to the object name within the object store.

  Set this parameter to None (the default) if path is a URL or a local file.

• chunk_size (int, optional) – Chunk size (in bytes). Ignored if path is a local file. (default: 1MB)

• progress_bar (bool, optional) – Whether to show a progress bar. Ignored if path is a local file. (default: True)

Raises

GetFileNotFoundException – If the path does not exist, a GetFileNotFoundException exception will be raised.