composer.utils.object_store.object_store_hparams#
Hyperparameters and registry for the ObjectStore
implementations.
Hparams
These classes are used with yahp
for YAML
-based configuration.
|
|
Base class for |
|
|
|
|
Attributes
object_store_registry
- class composer.utils.object_store.object_store_hparams.LibcloudObjectStoreHparams(provider, container, key_environ=None, secret_environ=None, region=None, host=None, port=None, extra_init_kwargs=<factory>)[source]#
Bases:
composer.utils.object_store.object_store_hparams.ObjectStoreHparams
LibcloudObjectStore
hyperparameters.Example
Hereโs an example on how to connect to an Amazon S3 bucket. This example assumes:
The container is named named
MY_CONTAINER
.The AWS Access Key ID is stored in an environment variable named
AWS_ACCESS_KEY_ID
.The Secret Access Key is in an environmental variable named
AWS_SECRET_ACCESS_KEY
.
>>> from composer.utils.object_store.object_store_hparams import LibcloudObjectStoreHparams >>> provider_hparams = LibcloudObjectStoreHparams( ... provider="s3", ... container="MY_CONTAINER", ... key_environ="AWS_ACCESS_KEY_ID", ... secret_environ="AWS_SECRET_ACCESS_KEY", ... ) >>> provider = provider_hparams.initialize_object() >>> provider <composer.utils.object_store.libcloud_object_store.LibcloudObjectStore object at ...>
- Parameters
provider (str) โ
Cloud provider to use.
See
LibcloudObjectStore
for documentation.container (str) โ The name of the container (i.e. bucket) to use.
key_environ (str, optional) โ
The name of an environment variable containing the API key or username to use to connect to the provider. If no key is required, then set this field to
None
. (default:None
)For security reasons, composer requires that the key be specified via an environment variable. For example, if your key is an environment variable called
OBJECT_STORE_KEY
that is set toMY_KEY
, then you should set this parameter equal toOBJECT_STORE_KEY
. Composer will read the key like this:>>> import os >>> params = LibcloudObjectStoreHparams(key_environ="OBJECT_STORE_KEY") >>> key = os.environ[params.key_environ] >>> key 'MY_KEY'
secret_environ (str, optional) โ
The name of an environment variable containing the API secret or password to use for the provider. If no secret is required, then set this field to
None
. (default:None
)For security reasons, composer requires that the secret be specified via an environment variable. For example, if your secret is an environment variable called
OBJECT_STORE_SECRET
that is set toMY_SECRET
, then you should set this parameter equal toOBJECT_STORE_SECRET
. Composer will read the secret like this:>>> import os >>> params = LibcloudObjectStoreHparams(secret_environ="OBJECT_STORE_SECRET") >>> secret = os.environ[params.secret_environ] >>> secret 'MY_SECRET'
region (str, optional) โ Cloud region to use for the cloud provider. Most providers do not require the region to be specified. (default:
None
)host (str, optional) โ Override the hostname for the cloud provider. (default:
None
)port (int, optional) โ Override the port for the cloud provider. (default:
None
)extra_init_kwargs (Dict[str, Any], optional) โ
Extra keyword arguments to pass into the constructor for the specified provider. (default:
None
, which is equivalent to an empty dictionary)See also
- class composer.utils.object_store.object_store_hparams.ObjectStoreHparams[source]#
Bases:
yahp.hparams.Hparams
,abc.ABC
Base class for
ObjectStore
hyperparameters.- abstract get_kwargs()[source]#
Returns the kwargs to construct the object store returned by
get_object_store_class()
.- Returns
Dict[str, Any] โ The kwargs.
- abstract get_object_store_cls()[source]#
Returns the type of
ObjectStore
.
- class composer.utils.object_store.object_store_hparams.S3ObjectStoreHparams(bucket, prefix='', region_name=None, endpoint_url=None, client_config=None, transfer_config=None)[source]#
Bases:
composer.utils.object_store.object_store_hparams.ObjectStoreHparams
S3ObjectStore
hyperparameters.The
S3ObjectStore
usesboto3
to handle uploading files to and downloading files from S3-Compatible object stores.Note
To follow best security practices, credentials cannot be specified as part of the hyperparameters. Instead, please ensure that credentials are in the environment, which will be read automatically.
See guide to credentials for more information.
- Parameters
bucket (str) โ See
S3ObjectStore
.prefix (str) โ See
S3ObjectStore
.region_name (str, optional) โ See
S3ObjectStore
.endpoint_url (str, optional) โ See
S3ObjectStore
.client_config (dict, optional) โ See
S3ObjectStore
.transfer_config (dict, optional) โ See
S3ObjectStore
.
- class composer.utils.object_store.object_store_hparams.SFTPObjectStoreHparams(host, port=22, username=None, known_hosts_filename=None, known_hosts_filename_environ='COMPOSER_SFTP_KNOWN_HOSTS_FILE', key_filename=None, key_filename_environ='COMPOSER_SFTP_KEY_FILE', missing_host_key_policy='RejectPolicy', cwd='', connect_kwargs=None)[source]#
Bases:
composer.utils.object_store.object_store_hparams.ObjectStoreHparams
SFTPObjectStore
hyperparameters.The
SFTPObjectStore
usesparamiko
to upload and download files from SSH serversNote
To follow best security practices, credentials shouldnโt be specified as part of the hyperparameters. Instead, please ensure that credentials are in the environment, which will be read automatically.
- Parameters
host (str) โ See
SFTPObjectStore
.port (int, optional) โ See
SFTPObjectStore
.username (str, optional) โ See
SFTPObjectStore
.known_hosts_filename (str, optional) โ See
SFTPObjectStore
.key_filename (str, optional) โ See
SFTPObjectStore
.cwd (str, optional) โ See
SFTPObjectStore
.connect_kwargs (Dict[str, Any], optional) โ See
SFTPObjectStore
.