Skip to content

reasoning_judge

diapason.evals.scorers.reasoning_judge

Reasoning judge scorer -- LLM-as-judge for math and reasoning tasks.

Attempts normalized exact match first, then falls back to an LLM judge for semantic comparison. Adapted from IPW's reasoning evaluation handlers.

ReasoningJudgeScorer

Bases: LLMJudgeScorer

LLM-as-judge evaluation for reasoning tasks.

Fast path: normalized exact match (no API call). Slow path: LLM judge for semantic equivalence.

reasoning_exact_match(model_answer, ground_truth)

Normalized exact match for reasoning answers.

Handles numbers, LaTeX boxed answers, and plain strings.