Skip to content

skill_benchmark

diapason.evals.skill_benchmark

SkillBenchmarkRunner — orchestrate the 4-condition × N-seed × M-task PinchBench sweep that measures whether skills + DSPy/GEPA optimization improves agent performance.

Plan 2B implementation. See: docs/superpowers/specs/2026-04-08-skills-benchmark-evaluation-design.md

ConditionComparison dataclass

Top-level result of a SkillBenchmarkRunner sweep.

ConditionResult dataclass

Aggregated result for a single condition across all seeds.

SkillBenchmarkConfig dataclass

Configuration for a single SkillBenchmarkRunner sweep.

SkillBenchmarkRunner

Orchestrates the 4-condition × N-seed × M-task PinchBench sweep.

Each condition is a different SystemBuilder configuration: - no_skills: cfg.skills.enabled = False - skills_on: enabled, overlay_dir = empty (no overlays load) - skills_optimized_dspy: enabled, overlay_dir = config.overlay_dir_dspy - skills_optimized_gepa: enabled, overlay_dir = config.overlay_dir_gepa

Per-seed runs share the same backend instantiation but pass a fresh seed to the EvalRunner.

run_all_conditions()

Run all 4 conditions × all seeds.

Returns a ConditionComparison with per-condition results and the computed deltas (skills_on - no_skills, etc.).

run_condition(condition)

Run the benchmark for condition across all configured seeds.

Returns a ConditionResult with mean ± stddev pass rate and the per-task / per-skill aggregations.

write_report(comparison)

Write a markdown report to output_dir.

Filename: pinchbench-skills-eval-{YYYY-MM-DD}.md