pinchbench
diapason.evals.scorers.pinchbench
¶
PinchBench grading helpers and scorer.
Provides transcript translation (Diapason events → PinchBench format), automated grading (exec of embedded Python), LLM judge grading, and hybrid combination. Used by PinchBenchTaskEnv.run_tests() and the standalone PinchBenchScorer.
Reference: https://github.com/pinchbench/skill
PinchBenchScorer
¶
events_to_transcript(events)
¶
Build PinchBench-format transcript from raw EventRecorder events.
Pairs TOOL_CALL_START/END events to extract tool name, arguments, and results. Called by run_tests() before QueryTrace exists.
grade_pinchbench_task(*, record, transcript, workspace_path, judge_backend=None, judge_model='anthropic/claude-opus-4-5')
¶
Top-level grading entry point. Routes by grading_type.
Returns {"score": float, "breakdown": dict, "notes": str}.