composer.core.evaluator#

A wrapper for a dataloader to include metrics that apply to a specific dataset.

Classes

Evaluator

A wrapper for a dataloader to include metrics that apply to a specific dataset.
class composer.core.evaluator.Evaluator(*, label, dataloader, metrics)[source]#

A wrapper for a dataloader to include metrics that apply to a specific dataset.

For example, CrossEntropyLoss metric for NLP models.

>>> from torchmetrics.classification.accuracy import Accuracy
>>> eval_evaluator = Evaluator(label="myEvaluator", dataloader=eval_dataloader, metrics=Accuracy())
>>> trainer = Trainer(
...     model=model,
...     train_dataloader=train_dataloader,
...     eval_dataloader=eval_evaluator,
...     optimizers=optimizer,
...     max_duration="1ep",
... )
Parameters

  • label (str) โ€“ Name of the Evaluator

  • dataloader (Union[DataSpec, DataLoader]) โ€“ DataLoader/DataSpec for evaluation data

  • metrics (Metric | MetricCollection) โ€“ torchmetrics.Metric to log. metrics will be deep-copied to ensure that each evaluator updates only its metrics.