composer.models.gpt2.model#

GPT-2 model based on Hugging Face GPT-2.

Implemented as a wrapper using ComposerTrainer.

Classes

GPT2Model

Implements ComposerTransformer to wrap Hugging Face GPT-2 transformers.
class composer.models.gpt2.model.GPT2Model(module, config, tokenizer=None, gradient_checkpointing=False)[source]#

Bases: composer.models.transformer_shared.ComposerTransformer

Implements ComposerTransformer to wrap Hugging Face GPT-2 transformers. Logs training and validation perplexity.

From Language Models are Unsupervised Multitask Learners (Radford et al, 2018).

Parameters

  • module (GPT2Model) โ€“ The model to wrap with this module.

  • config (GPT2Config) โ€“ The config for the model.

  • tokenizer (GPT2Tokenizer) โ€“ The tokenizer used for this model. Necessary to process model inputs.

  • gradient_checkpointing (bool, optional) โ€“ Use gradient checkpointing. default: False.

To create a GPT-2 model for language modeling pretraining:

from composer.models import GPT2Model
import transformers

config = transformers.GPT2Config()
hf_model = transformers.GPT2LMHeadModel(config=config) # gpt2-small model from huggingface
tokenizer = transformers.GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2Model(module=hf_model, config=config, tokenizer=tokenizer)