composer.algorithms.hparams

composer.algorithms.hparams

Functions

asdict	Return the fields of a dataclass instance as a new dictionary mapping field names to field values.
dataclass	Returns the same class as was passed in, with dunder methods added based on the fields defined in the class.

Classes

Alibi	ALiBi (Attention with Linear Biases; Press et al, 2021) dispenses with position embeddings and instead directly biases attention matrices such that nearby tokens attend to one another more strongly.
AugMix	AugMix (Hendrycks et al, 2020) creates width sequences of depth image augmentations, applies each sequence with random intensity, and returns a convex combination of the width augmented images and the original image.
BlurPool	BlurPool adds anti-aliasing filters to convolutional layers to increase accuracy and invariance to small shifts in the input.
ChannelsLast	Changes the memory format of the model to torch.channels_last.
ColOut	Drops a fraction of the rows and columns of an input image.
CutMix	CutMix trains the network on non-overlapping combinations of pairs of examples and interpolated targets rather than individual examples and targets.
CutOut	Cutout is a data augmentation technique that works by masking out one or more square regions of an input image.
Factorize	Decomposes linear operators into pairs of smaller linear operators.
GhostBatchNorm	Replaces batch normalization modules with Ghost Batch Normalization modules that simulate the effect of using a smaller batch size.
LabelSmoothing	Shrink targets towards a uniform distribution as in Szegedy et al.
LayerFreezing	Progressively freeze the layers of the network during training, starting with the earlier layers.
MixUp	MixUp trains the network on convex combinations of pairs of examples and targets rather than individual examples and targets.
NoOpModel	composer.algorithms.no_op_model.no_op_model.NoOpModel
ProgressiveResizing	Apply Fastai's progressive resizing data augmentation to speed up training.
RandAugment	Randomly applies a sequence of image data augmentations (Cubuk et al, 2019) to an image.
SAM	Adds sharpness-aware minimization (Foret et al, 2020) by wrapping an existing optimizer with a SAMOptimizer.
SWA	Apply Stochastic Weight Averaging (Izmailov et al, 2018)
ScaleSchedule	Deprecated - do not use.
SelectiveBackprop	Selectively backpropagate gradients from a subset of each batch (Jiang et al, 2019).
SeqLengthWarmup	Progressively increases the sequence length during training.
SqueezeExcite	Adds Squeeze-and-Excitation blocks (Hu et al, 2019) after the Conv2d modules in a neural network.
StochasticDepth	Applies Stochastic Depth (Huang et al, 2016) to the specified model.

Hparams

These classes are used with yahp for YAML-based configuration.

AlgorithmHparams	Hyperparameters for algorithms.
AlibiHparams	See Alibi
AugMixHparams	See AugMix
BlurPoolHparams	See BlurPool
ChannelsLastHparams	ChannelsLast has no hyperparameters, so this class has no member variables.
ColOutHparams	See ColOut
CutMixHparams	See CutMix
CutOutHparams	See CutOut
FactorizeHparams	See Factorize
GhostBatchNormHparams	See GhostBatchNorm
LabelSmoothingHparams	See LabelSmoothing
LayerFreezingHparams	See LayerFreezing
MixUpHparams	See MixUp
NoOpModelHparams	composer.algorithms.hparams.NoOpModelHparams
ProgressiveResizingHparams	See ProgressiveResizing
RandAugmentHparams	See RandAugment
SAMHparams	See SAM
SWAHparams	See SWA
ScaleScheduleHparams	See ScaleSchedule
SelectiveBackpropHparams	See SelectiveBackprop
SeqLengthWarmupHparams	composer.algorithms.hparams.SeqLengthWarmupHparams
SqueezeExciteHparams	See SqueezeExcite
StochasticDepthHparams	See StochasticDepth

Attributes

• Optional

class composer.algorithms.hparams.AlibiHparams(position_embedding_attribute, attention_module_name, attr_to_replace, alibi_attention, mask_replacement_function=None, heads_per_layer=None, max_sequence_length=8192, train_sequence_length_scaling=0.25)

Bases: composer.algorithms.algorithm_hparams.AlgorithmHparams

See Alibi

class composer.algorithms.hparams.AugMixHparams(severity=3, depth=- 1, width=3, alpha=1.0, augmentation_set='all')

Bases: composer.algorithms.algorithm_hparams.AlgorithmHparams

See AugMix

class composer.algorithms.hparams.BlurPoolHparams(replace_convs=True, replace_maxpools=True, blur_first=True)

Bases: composer.algorithms.algorithm_hparams.AlgorithmHparams

See BlurPool

class composer.algorithms.hparams.ChannelsLastHparams

Bases: composer.algorithms.algorithm_hparams.AlgorithmHparams

ChannelsLast has no hyperparameters, so this class has no member variables.

class composer.algorithms.hparams.ColOutHparams(p_row=0.15, p_col=0.15, batch=True)

Bases: composer.algorithms.algorithm_hparams.AlgorithmHparams

See ColOut

class composer.algorithms.hparams.CutMixHparams(num_classes, alpha=1.0, uniform_sampling=False)

Bases: composer.algorithms.algorithm_hparams.AlgorithmHparams

See CutMix

class composer.algorithms.hparams.CutOutHparams(num_holes=1, length=0.5, uniform_sampling=False)

Bases: composer.algorithms.algorithm_hparams.AlgorithmHparams

See CutOut

class composer.algorithms.hparams.FactorizeHparams(factorize_convs=True, factorize_linears=True, min_channels=512, latent_channels=0.25, min_features=512, latent_features=0.25)

Bases: composer.algorithms.algorithm_hparams.AlgorithmHparams

See Factorize

class composer.algorithms.hparams.GhostBatchNormHparams(ghost_batch_size=32)

Bases: composer.algorithms.algorithm_hparams.AlgorithmHparams

See GhostBatchNorm

class composer.algorithms.hparams.LabelSmoothingHparams(smoothing=0.1)

Bases: composer.algorithms.algorithm_hparams.AlgorithmHparams

See LabelSmoothing

class composer.algorithms.hparams.LayerFreezingHparams(freeze_start=0.5, freeze_level=1.0)

Bases: composer.algorithms.algorithm_hparams.AlgorithmHparams

See LayerFreezing

class composer.algorithms.hparams.MixUpHparams(alpha=0.2, interpolate_loss=False)

Bases: composer.algorithms.algorithm_hparams.AlgorithmHparams

See MixUp

class composer.algorithms.hparams.NoOpModelHparams

Bases: composer.algorithms.algorithm_hparams.AlgorithmHparams

composer.algorithms.hparams.NoOpModelHparams

class composer.algorithms.hparams.ProgressiveResizingHparams(mode='resize', initial_scale=0.5, finetune_fraction=0.2, resize_targets=False)

Bases: composer.algorithms.algorithm_hparams.AlgorithmHparams

See ProgressiveResizing

class composer.algorithms.hparams.RandAugmentHparams(severity=9, depth=2, augmentation_set='all')

Bases: composer.algorithms.algorithm_hparams.AlgorithmHparams

See RandAugment

class composer.algorithms.hparams.SAMHparams(rho=0.05, epsilon=1e-12, interval=1)

Bases: composer.algorithms.algorithm_hparams.AlgorithmHparams

See SAM

class composer.algorithms.hparams.SWAHparams(swa_start='0.7dur', swa_end='0.97dur', update_interval='1ep', schedule_swa_lr=False, anneal_strategy='linear', anneal_steps=10, swa_lr=None)

Bases: composer.algorithms.algorithm_hparams.AlgorithmHparams

See SWA

class composer.algorithms.hparams.ScaleScheduleHparams(ratio=1.0)

Bases: composer.algorithms.algorithm_hparams.AlgorithmHparams

See ScaleSchedule

class composer.algorithms.hparams.SelectiveBackpropHparams(start=0.5, end=0.9, keep=0.5, scale_factor=0.5, interrupt=2)

Bases: composer.algorithms.algorithm_hparams.AlgorithmHparams

See SelectiveBackprop

class composer.algorithms.hparams.SeqLengthWarmupHparams(duration: float = 0.3, min_seq_length: int = 8, max_seq_length: int = 1024, step_size: int = 8, truncate: bool = True)

Bases: composer.algorithms.algorithm_hparams.AlgorithmHparams

composer.algorithms.hparams.SeqLengthWarmupHparams

class composer.algorithms.hparams.SqueezeExciteHparams(latent_channels=64, min_channels=128)

Bases: composer.algorithms.algorithm_hparams.AlgorithmHparams

See SqueezeExcite

class composer.algorithms.hparams.StochasticDepthHparams(target_layer_name, stochastic_method='block', drop_rate=0.2, drop_distribution='linear', use_same_gpu_seed=True, drop_warmup='0dur')

Bases: composer.algorithms.algorithm_hparams.AlgorithmHparams

See StochasticDepth