InContextLearningMultipleChoiceAccuracy

class composer.metrics.InContextLearningMultipleChoiceAccuracy(dist_sync_on_step=False)

Computes accuracy for In-context learning (ICL) multiple choice (MC) tasks.

ICL MC tasks consists of a series of questions with some number of possible choices (only one of which can be correct). At inference time each possible choice is given to the model as a separate input and the one for which the model assigns the lowest perplexity to the choice is considered the model’s choice. The model is correct if it “chooses” the right answer.

Context: The dog is->fuzzynthe water is->hotnthe tree is-> Continuation: green

Adds metric state variables:

correct (float): The number of instances where the prediction masked the target. total (float): The number of total instances that were predicted.

Parameters

dist_sync_on_step (bool, optional) – Synchronize metric state across processes at each forward() before returning the value at the step. Default: False.