Skip to content

simpleqa_judge

diapason.evals.scorers.simpleqa_judge

SimpleQA scorer -- normalized exact match with LLM fallback.

Evaluates short factual answers using exact string matching (with normalization) and falls back to an LLM judge for semantic comparison.

SimpleQAScorer

Bases: LLMJudgeScorer

SimpleQA evaluation: exact match with normalization + LLM fallback.

exact_match(model_answer, ground_truth)

Exact-match scorer with normalization for numbers and strings.