Skip to content

pinchbench

diapason.evals.datasets.pinchbench

PinchBench dataset provider — real-world agent task benchmark.

Clones the pinchbench/skill repo at runtime and parses task markdown files into EvalRecords for use with AgenticRunner.

Reference: https://github.com/pinchbench/skill

PinchBenchDataset

Bases: DatasetProvider

PinchBench real-world agent benchmark.

Clones pinchbench/skill from GitHub (or uses a local path) and parses task markdown files into EvalRecords.

set_judge(judge_backend, judge_model)

Set the judge backend/model for LLM-judge and hybrid grading.