composer.utils.object_store.libcloud_object_store

Utility for uploading to and downloading from cloud object stores.

Classes

LibcloudObjectStore

Utility for uploading to and downloading from object (blob) stores, such as Amazon S3.

class composer.utils.object_store.libcloud_object_store.LibcloudObjectStore(provider, container, chunk_size=1048576, provider_kwargs=None)

Bases: composer.utils.object_store.object_store.ObjectStore

Utility for uploading to and downloading from object (blob) stores, such as Amazon S3.

Example

Here’s an example for an Amazon S3 bucket named MY_CONTAINER:

>>> from composer.utils import LibcloudObjectStore
>>> object_store = LibcloudObjectStore(
...     provider="s3",
...     container="MY_CONTAINER",
...     provider_kwargs={
...         "key": "AKIA...",
...         "secret": "*********",
...     }
... )
>>> object_store
<composer.utils.object_store.libcloud_object_store.LibcloudObjectStore object at ...>

Parameters

• provider (str) –

  Cloud provider to use. Valid options are:

  • atmos

  • auroraobjects

  • azure_blobs

  • backblaze_b2

  • cloudfiles

  • digitalocean_spaces

  • google_storage

  • ktucloud

  • local

  • minio

  • nimbus

  • ninefold

  • oss

  • rgw

  • s3

  See also

  Full list of libcloud providers

• container (str) – The name of the container (i.e. bucket) to use.

• provider_kwargs (Dict[str, Any], optional) –

  Keyword arguments to pass into the constructor for the specified provider. These arguments would usually include the cloud region and credentials.

  Common keys are:

  • key (str): API key or username to be used (required).

  • secret (str): Secret password to be used (required).

  • secure (bool): Whether to use HTTPS or HTTP. Note: Some providers only support HTTPS, and it is on by default.

  • host (str): Override hostname used for connections.

  • port (int): Override port used for connections.

  • api_version (str): Optional API version. Only used by drivers which support multiple API versions.

  • region (str): Optional driver region. Only used by drivers which support multiple regions.

  See also

  libcloud.storage.base.StorageDriver