passes

Algorithm Passes reorder or modify the execution of algorithms by the Engine.

The order in which algorithms are run matters significantly during composition. For example, the SelectiveBackprop algorithm runs on the Event.AFTER_DATALOADER event and must run before any data augmentations. Engine runs re-ordering passes to resolve such ordering issues or conflicts.

These modifications are represented as algorithm passes, which are functions that modify a list of algorithms.

For example, an algorithm pass that ensures a certain algorithm runs last, would be implemented as:

def run_last(algorithms: Sequence[Algorithm], event: Event) -> Sequence[Algorithm]:
   algorithms = sorted(algorithms, key=lambda x: isinstance(x, MyAlgorithm))

The passes in this module are registered by default into Engine.

Functions

set_filo_order	Establish a FILO order of algorithms before_ and after_ events.
sort_fused_layernorm_last	FusedLayerNorm should run after other algorithms that add LayerNorms (e.g.
sort_selective_backprop_first	Selective Backprop should run before any algorithms modify the loss.
sort_to_back	Helper function to sort instances of a provided class to the back.
sort_to_front	Helper function to sort instances of a provided class to the front.
warn_if_multiple_loss_interpolation	Multiple algorithms that interpolate the loss may have unexpected behavior.

Classes

Algorithm	Base class for algorithms.
Event	Enum to represent training loop events.
TypeVar	Type variable.

Attributes

• AlgorithmPass

• Any

• Callable

• Sequence

• T

composer.core.passes.set_filo_order(algorithms, event)

Establish a FILO order of algorithms before_ and after_ events.

For the events that follow the before_* and after_* pattern (e.g., Event.BEFORE_LOSS and Event.AFTER_LOSS), the ordering of algorithms is reversed for the ``after_*` events. For example, four given algorithms A, B, C, and D will run in ABCD ordering on the before_* event while DCBA ordering on the after_* event.

This allows algorithms to “clean up” their changes. For example, LabelSmoothing will smooth the labels upon the Event.BEFORE_LOSS event and then restore the original unsmoothed labels on the Event.AFTER_LOSS event.

Events with the pattern _start or _end will not be affected.

composer.core.passes.sort_fused_layernorm_last(algorithms, event)

FusedLayerNorm should run after other algorithms that add LayerNorms (e.g. GatedLinearUnits).

This ensures that all LayerNorms are converted to optimized fused versions.

composer.core.passes.sort_selective_backprop_first(algorithms, event)

Selective Backprop should run before any algorithms modify the loss.

SelectiveBackprop runs after the dataloader returns the batch and executes an extra forward pass to rank and prune the examples in the batch by loss. To ensure a clean estimate of loss, SelectiveBackprop should run before any other data augmentations (e.g., MixUp) on the Event.AFTER_DATALOADER event.

composer.core.passes.sort_to_back(list_to_sort, cls)

Helper function to sort instances of a provided class to the back.

Example

>>> sort_to_back([1, 'b', 3], str)
[1, 3, 'b']

Parameters

• list_to_sort – list of objects to sort

• cls – sorts all objects of this class to the back

Returns

sorted_list – Sorted List

composer.core.passes.sort_to_front(list_to_sort, cls)

Helper function to sort instances of a provided class to the front.

Example

>>> sort_to_front([1, 'b', 3], str)
['b', 1, 3]

Parameters

• list_to_sort – list of objects to sort

• cls – sorts all objects of this class to the front

Returns

sorted_list – Sorted List

composer.core.passes.warn_if_multiple_loss_interpolation(algorithms, event)

Multiple algorithms that interpolate the loss may have unexpected behavior.