composer.utils.reproducibility#
Helper utilities for configuring deterministic training to ensure reproducibility.
Note
For deterministic model initialization, seed_all()
and/or
configure_deterministic_mode()
should be
invoked before creating and initializing a model, before creating the Trainer
.
For example:
>>> import torch.nn
>>> from composer.utils import reproducibility
>>> reproducibility.configure_deterministic_mode()
>>> reproducibility.seed_all(42)
>>> model = MyModel()
>>> def init_weights(m):
... if isinstance(m, torch.nn.Linear):
... torch.nn.init.xavier_uniform(m.weight)
>>> # model will now be deterministically initialized, since the seed is set.
>>> init_weights(model)
>>> trainer = Trainer(model=model, seed=42)
Note that the seed must also be passed to the Trainer, otherwise the Trainer
would generate a random seed based on the timestamp (see get_random_seed()
).
- composer.utils.reproducibility.MAX_SEED[source]#
The maximum allowed seed, which is \(2^{32} - 1\).
- Type
Functions
Configure PyTorch deterministic mode. |
|
Get a randomly created seed to use for seeding rng objects. |
|
The state of the RNG objects. |
|
Restore the RNG state. |
|
Seed all rng objects. |
Attributes
- composer.utils.reproducibility.configure_deterministic_mode()[source]#
Configure PyTorch deterministic mode.
Note
When using the
Trainer
, you can use thedeterministic_mode
flag instead of invoking this function directly. For example:>>> trainer = Trainer(deterministic_mode=True)
However, to configure deterministic mode for operations before the trainer is initialized, manually invoke this function at the beginning of your training script.
Note
When training on a GPU, this function must be invoked before any CUDA operations.
Note
Deterministic mode degrades performance. Do not use outside of testing and debugging.
- composer.utils.reproducibility.get_random_seed()[source]#
Get a randomly created seed to use for seeding rng objects.
Warning
This random seed is NOT cryptographically secure.
- Returns
int โ A random seed.
- composer.utils.reproducibility.get_rng_state()[source]#
The state of the RNG objects.
- Returns
List[Dict[str, Any]] โ A list of RNG State Dicts, indexed by global rank.
- composer.utils.reproducibility.load_rng_state(rng_state_dicts)[source]#
Restore the RNG state.
- Parameters
rng_state_dicts (List[Dict[str, Any]]) โ The list of RNG state dicts to restore, as returned by
get_rng_state()
.
- composer.utils.reproducibility.seed_all(seed)[source]#
Seed all rng objects.
Note
When using the
Trainer
, you can use theseed
parameter instead of invoking this function directly. For example:>>> trainer = Trainer(seed=42)
However, to configure the random seed for operations before the trainer is initialized, manually invoke this function at the beginning of your training script.
- Parameters
seed (int) โ The random seed