Skip to content

mmlu_pro_mcq

diapason.evals.scorers.mmlu_pro_mcq

MMLU-Pro MCQ scorer — LLM-based letter extraction + exact match.

Adapted from IPW's mcq.py evaluation handler.

MMLUProScorer

Bases: LLMJudgeScorer

Score MMLU-Pro responses by extracting answer letter via LLM.