composer.models.bert.model#
Implements a BERT wrapper around a ComposerTransformer
.
Classes
BERT model based on ๐ค Transformers. |
- class composer.models.bert.model.BERTModel(module, config, tokenizer=None)[source]#
Bases:
composer.models.transformer_shared.ComposerTransformer
BERT model based on ๐ค Transformers.
For more information, see Transformers.
- Parameters
module (transformers.BertModel) โ An instance of BertModel that contains the forward pass function.
config (transformers.BertConfig) โ The BertConfig object that stores information about the model hyperparameters.
tokenizer (transformers.BertTokenizer) โ An instance of BertTokenizer. Necessary to process model inputs.
To create a BERT model for Language Model pretraining:
from composer.models import BERTModel import transformers config = transformers.BertConfig() hf_model = transformers.BertLMHeadModel(config=config) tokenizer = transformers.BertTokenizer.from_pretrained("bert-base-uncased") model = BERTModel(module=hf_model, config=config, tokenizer=tokenizer)
- validate(batch)[source]#
Runs the validation step.
- Parameters
batch (Dict) โ a dictionary of Dict[str, Tensor] of inputs that the model expects, as found in
ComposerTransformer.get_model_inputs()
.- Returns
tuple (Tensor, Tensor) โ with the output from the forward pass and the correct labels. This is fed into directly into the output of
ComposerModel.metrics()
.