pinchbench
diapason.evals.datasets.pinchbench
¶
PinchBench dataset provider — real-world agent task benchmark.
Clones the pinchbench/skill repo at runtime and parses task markdown files into EvalRecords for use with AgenticRunner.
Reference: https://github.com/pinchbench/skill
PinchBenchDataset
¶
Bases: DatasetProvider
PinchBench real-world agent benchmark.
Clones pinchbench/skill from GitHub (or uses a local path) and parses task markdown files into EvalRecords.
set_judge(judge_backend, judge_model)
¶
Set the judge backend/model for LLM-judge and hybrid grading.