Skip to content

pinchbench

diapason.evals.scorers.pinchbench

PinchBench grading helpers and scorer.

Provides transcript translation (Diapason events → PinchBench format), automated grading (exec of embedded Python), LLM judge grading, and hybrid combination. Used by PinchBenchTaskEnv.run_tests() and the standalone PinchBenchScorer.

Reference: https://github.com/pinchbench/skill

PinchBenchScorer

Bases: LLMJudgeScorer

PinchBench scorer for the non-agentic EvalRunner path.

events_to_transcript(events)

Build PinchBench-format transcript from raw EventRecorder events.

Pairs TOOL_CALL_START/END events to extract tool name, arguments, and results. Called by run_tests() before QueryTrace exists.

grade_pinchbench_task(*, record, transcript, workspace_path, judge_backend=None, judge_model='anthropic/claude-opus-4-5')

Top-level grading entry point. Routes by grading_type.

Returns {"score": float, "breakdown": dict, "notes": str}.