fused_layernorm

Modules

composer.algorithms.fused_layernorm.fused_layernorm

Module fused_layernorm.

Replaces all instances of torch.nn.LayerNorm with a apex.normalization.fused_layer_norm.FusedLayerNorm.

By fusing multiple kernel launches into one, this usually improves GPU utilization.

See the Method Card for more details.

Functions

apply_fused_layernorm

Replaces all instances of torch.nn.LayerNorm with a apex.normalization.fused_layer_norm.FusedLayerNorm.

Classes

FusedLayerNorm

Replaces all instances of torch.nn.LayerNorm with a apex.normalization.fused_layer_norm.FusedLayerNorm.