scorer

scorer

Abstract base classes for scoring.

Classes

Scorer

Bases: ABC

Base class for all scorers.

Methods:

score abstractmethod

score(
    record: EvalRecord, model_answer: str
) -> Tuple[Optional[bool], Dict[str, Any]]

Score a model answer against the reference.

Returns (is_correct, metadata) where is_correct may be None if scoring could not be determined.

Source code in src/diapason/evals/core/scorer.py

@abstractmethod
def score(
    self,
    record: EvalRecord,
    model_answer: str,
) -> Tuple[Optional[bool], Dict[str, Any]]:
    """Score a model answer against the reference.

    Returns (is_correct, metadata) where is_correct may be None
    if scoring could not be determined.
    """

LLMJudgeScorer

LLMJudgeScorer(
    judge_backend: InferenceBackend, judge_model: str
)

Bases: Scorer

Base for scorers that need an LLM to judge answers.

Source code in src/diapason/evals/core/scorer.py

def __init__(self, judge_backend: InferenceBackend, judge_model: str) -> None:
    self._judge_backend = judge_backend
    self._judge_model = judge_model