composer.models.transformer_hparams#

General YAHP interface for ComposerTransformers.

Hparams

These classes are used with yahp for YAML-based configuration.

TransformerHparams

Defines the necessary hyparameters for a Transformer base module.
class composer.models.transformer_hparams.TransformerHparams(initializers=<factory>, num_classes=None, tokenizer_name=None, pretrained_model_name=None, model_config=<factory>, use_pretrained=False, gradient_checkpointing=False)[source]#

Bases: composer.models.model_hparams.ModelHparams, abc.ABC

Defines the necessary hyparameters for a Transformer base module.

Parameters

  • pretrained_model_name (Optional[str]) โ€“ โ€œPretrained model name to pull from Huggingface Model Hub.โ€

  • model_config (Dict[str, JSON]) โ€“ A dictionary providing a HuggingFace model configuration.

  • tokenizer_name (str) โ€“ The tokenizer used for this model, necessary to assert required model inputs.

  • use_pretrained (bool, optional) โ€“ Whether to initialize the model with the pretrained weights. Default: False

  • gradient_checkpointing (bool, optional) โ€“ Use gradient checkpointing. Default: False.