composer.functional#

Functional API for applying algorithms in your own training loop.

from composer import functional as cf
from torchvision import models

model = models.resnet50()

# replace some layers with blurpool
cf.apply_blurpool(model)
# replace some layers with squeeze-excite
cf.apply_squeeze_excite(model, latent_channels=64, min_channels=128)

Functions

apply_alibi

Removes position embeddings and replaces the attention function and attention mask as per Alibi.

apply_blurpool

Add anti-aliasing filters to strided torch.nn.Conv2d and/or torch.nn.MaxPool2d modules.

apply_channels_last

Changes the memory format of the model to torch.channels_last.

apply_factorization

Replaces torch.nn.Linear and torch.nn.Conv2d modules with FactorizedLinear and FactorizedConv2d modules.

apply_fused_layernorm

Replaces all instances of torch.nn.LayerNorm with a apex.normalization.fused_layer_norm.FusedLayerNorm.

apply_gated_linear_units

Replaces the Linear layers in the feed-forward network with Gated Linear Units.

apply_ghost_batchnorm

Replace batch normalization modules with ghost batch normalization modules.

apply_gradient_clipping

Clips all gradients in model based on specified clipping_type.

apply_gyro_dropout

Replaces all instances of torch.nn.Dropout with a GyroDropout.

apply_low_precision_layernorm

apply_squeeze_excite

Adds Squeeze-and-Excitation blocks (Hu et al, 2019) after torch.nn.Conv2d layers.

apply_stochastic_depth

Applies Stochastic Depth (Huang et al, 2016) to the specified model.

apply_weight_standardization

Weight Standardization standardizes convolutional weights in a model.

augmix_image

Applies the AugMix (Hendrycks et al, 2020) data augmentation.

colout_batch

Applies ColOut augmentation to a batch of images and (optionally) targets, dropping the same random rows and columns from all images and targets in a batch.

compute_ema

Updates the weights of ema_model to be closer to the weights of model according to an exponential weighted average.

cutmix_batch

Create new samples using combinations of pairs of samples.

cutout_batch

See CutOut.

freeze_layers

Progressively freeze the layers of the network in-place during training, starting with the earlier layers.

mixup_batch

Create new samples using convex combinations of pairs of samples.

randaugment_image

Randomly applies a sequence of image data augmentations to an image or batch of images.

resize_batch

Resize inputs and optionally outputs by cropping or interpolating.

select_using_loss

Prunes minibatches as a subroutine of SelectiveBackprop.

set_batch_sequence_length

Set the sequence length of a batch.

should_selective_backprop

Decides if selective backprop should be run based on time in training.

smooth_labels

Shrink targets towards a uniform distribution as in Szegedy et al.