๐Ÿƒ Methods Overview#

card

shadow

Alibi

NLP

Replace attention with AliBi

AugMix

CV

Image-preserving data augmentations

Apply blur before pooling

Use channels last memory format (NHWC)

ColOut

CV

Remove columns and rows from images for augmentation and efficiency

CutMix

CV

Combine pairs of examples in non-overlapping regions and mix labels

CutOut

CV

Randomly erase rectangular blocks from the image

Factorize

CV NLP

Factorize GEMMs into smaller GEMMs

Use smaller samples to compute batchnorm

Smooth labels with a uniform prior

Progressively freeze layers during training.

MixUp

CV

Blend pairs of examples and labels

Increase the input image size during training

Apply a series of random augmentations

SAM

CV

Measure sharpness of optimization space

Scale the learning rate schedule by a factor

Drop examples with small loss contributions.

Progressively increase sequence length

Replace eligible layers with Squeeze-Excite layers

Replace a specified layer with a stochastic verion that randomly drops the layer or samples during training

SWA

CV NLP

Compute running average of model weights