object_store_hparams#

Hyperparameters and registry for the ObjectStore implementations.

Hparams

These classes are used with yahp for YAML-based configuration.

LibcloudObjectStoreHparams

LibcloudObjectStore hyperparameters.

ObjectStoreHparams

Base class for ObjectStore hyperparameters.

S3ObjectStoreHparams

S3ObjectStore hyperparameters.

SFTPObjectStoreHparams

SFTPObjectStore hyperparameters.

Attributes

  • object_store_registry

class composer.utils.object_store.object_store_hparams.LibcloudObjectStoreHparams(provider, container, key_environ=None, secret_environ=None, region=None, host=None, port=None, extra_init_kwargs=<factory>)[source]#

Bases: composer.utils.object_store.object_store_hparams.ObjectStoreHparams

LibcloudObjectStore hyperparameters.

Example

Hereโ€™s an example on how to connect to an Amazon S3 bucket. This example assumes:

  • The container is named named MY_CONTAINER.

  • The AWS Access Key ID is stored in an environment variable named AWS_ACCESS_KEY_ID.

  • The Secret Access Key is in an environmental variable named AWS_SECRET_ACCESS_KEY.

>>> from composer.utils.object_store.object_store_hparams import LibcloudObjectStoreHparams
>>> provider_hparams = LibcloudObjectStoreHparams(
...     provider="s3",
...     container="MY_CONTAINER",
...     key_environ="AWS_ACCESS_KEY_ID",
...     secret_environ="AWS_SECRET_ACCESS_KEY",
... )
>>> provider = provider_hparams.initialize_object()
>>> provider
<composer.utils.object_store.libcloud_object_store.LibcloudObjectStore object at ...>
Parameters

  • provider (str) โ€“

    Cloud provider to use.

    See LibcloudObjectStore for documentation.

  • container (str) โ€“ The name of the container (i.e. bucket) to use.

  • key_environ (str, optional) โ€“

    The name of an environment variable containing the API key or username to use to connect to the provider. If no key is required, then set this field to None. (default: None)

    For security reasons, composer requires that the key be specified via an environment variable. For example, if your key is an environment variable called OBJECT_STORE_KEY that is set to MY_KEY, then you should set this parameter equal to OBJECT_STORE_KEY. Composer will read the key like this:

    >>> import os
>>> params = LibcloudObjectStoreHparams(key_environ="OBJECT_STORE_KEY")
>>> key = os.environ[params.key_environ]
>>> key
'MY_KEY'

  • secret_environ (str, optional) โ€“

    The name of an environment variable containing the API secret or password to use for the provider. If no secret is required, then set this field to None. (default: None)

    For security reasons, composer requires that the secret be specified via an environment variable. For example, if your secret is an environment variable called OBJECT_STORE_SECRET that is set to MY_SECRET, then you should set this parameter equal to OBJECT_STORE_SECRET. Composer will read the secret like this:

    >>> import os
>>> params = LibcloudObjectStoreHparams(secret_environ="OBJECT_STORE_SECRET")
>>> secret = os.environ[params.secret_environ]
>>> secret
'MY_SECRET'

  • region (str, optional) โ€“ Cloud region to use for the cloud provider. Most providers do not require the region to be specified. (default: None)

  • host (str, optional) โ€“ Override the hostname for the cloud provider. (default: None)

  • port (int, optional) โ€“ Override the port for the cloud provider. (default: None)

  • extra_init_kwargs (Dict[str, Any], optional) โ€“

    Extra keyword arguments to pass into the constructor for the specified provider. (default: None, which is equivalent to an empty dictionary)

    See also

    libcloud.storage.base.StorageDriver

class composer.utils.object_store.object_store_hparams.ObjectStoreHparams[source]#

Bases: yahp.hparams.Hparams, abc.ABC

Base class for ObjectStore hyperparameters.

abstract get_kwargs()[source]#

Returns the kwargs to construct the object store returned by get_object_store_class().

Returns

Dict[str, Any] โ€“ The kwargs.

abstract get_object_store_cls()[source]#

Returns the type of ObjectStore.

class composer.utils.object_store.object_store_hparams.S3ObjectStoreHparams(bucket, prefix='', region_name=None, endpoint_url=None, client_config=None, transfer_config=None)[source]#

Bases: composer.utils.object_store.object_store_hparams.ObjectStoreHparams

S3ObjectStore hyperparameters.

The S3ObjectStore uses boto3 to handle uploading files to and downloading files from S3-Compatible object stores.

Note

To follow best security practices, credentials cannot be specified as part of the hyperparameters. Instead, please ensure that credentials are in the environment, which will be read automatically.

See guide to credentials for more information.

Parameters
class composer.utils.object_store.object_store_hparams.SFTPObjectStoreHparams(host, port=22, username=None, known_hosts_filename=None, known_hosts_filename_environ='COMPOSER_SFTP_KNOWN_HOSTS_FILE', key_filename=None, key_filename_environ='COMPOSER_SFTP_KEY_FILE', missing_host_key_policy='RejectPolicy', cwd='', connect_kwargs=None)[source]#

Bases: composer.utils.object_store.object_store_hparams.ObjectStoreHparams

SFTPObjectStore hyperparameters.

The SFTPObjectStore uses paramiko to upload and download files from SSH servers

Note

To follow best security practices, credentials shouldnโ€™t be specified as part of the hyperparameters. Instead, please ensure that credentials are in the environment, which will be read automatically.

Parameters