composer.Algorithm
Algorithms are implemented in both a standalone functional form (see composer.functional) and as subclasses of Algorithm
for integration in the MosaicML Trainer
. This section describes the latter form.
For ease of composability, algorithms in our Trainer are based on the two-way callbacks concept from Howard et al., 2020. Each algorithm implements two methods:
Algorithm.match()
: returnsTrue
if the algorithm should be run given the currentState
andEvent
.Algorithm.apply()
: performs an in-place modification of the givenState
For example, a simple algorithm that shortens training:
from composer import Algorithm, State, Event, Logger
class ShortenTraining(Algorithm):
def match(self, state: State, event: Event, logger: Logger) -> bool:
return event == Event.TRAINING_START
def apply(self, state: State, event: Event, logger: Logger):
state.max_epochs /= 2 # cut training time in half
For a complete list of algorithms, see composer.algorithms.
For reference, available events include:
Name |
Description |
---|---|
|
Immediately after |
|
Start of training. For multi-GPU training, runs after the DDP process fork. |
|
Start and end of an Epoch. |
|
Start and end of a batch, inclusive of the optimizer step and any gradient scaling. |
|
Immediately after the dataloader is called. Typically used for on-GPU dataloader transforms. |
|
Before and after the forward-loss-backward computation for a training batch. When using gradient_accumulation, these are still called only once. |
|
Before and after the call to |
|
Before and after the loss computation. |
|
Before and after the backward pass. |
|
End of training. |
|
Start and end of evaluation through the validation dataset. |
|
Before and after the call to |
|
Before and after the call to |
For more information about events, see composer.Event.