skill_benchmark
diapason.evals.skill_benchmark
¶
SkillBenchmarkRunner — orchestrate the 4-condition × N-seed × M-task PinchBench sweep that measures whether skills + DSPy/GEPA optimization improves agent performance.
Plan 2B implementation. See: docs/superpowers/specs/2026-04-08-skills-benchmark-evaluation-design.md
ConditionComparison
dataclass
¶
Top-level result of a SkillBenchmarkRunner sweep.
ConditionResult
dataclass
¶
Aggregated result for a single condition across all seeds.
SkillBenchmarkConfig
dataclass
¶
Configuration for a single SkillBenchmarkRunner sweep.
SkillBenchmarkRunner
¶
Orchestrates the 4-condition × N-seed × M-task PinchBench sweep.
Each condition is a different SystemBuilder configuration: - no_skills: cfg.skills.enabled = False - skills_on: enabled, overlay_dir = empty (no overlays load) - skills_optimized_dspy: enabled, overlay_dir = config.overlay_dir_dspy - skills_optimized_gepa: enabled, overlay_dir = config.overlay_dir_gepa
Per-seed runs share the same backend instantiation but pass a fresh seed to the EvalRunner.
run_all_conditions()
¶
Run all 4 conditions × all seeds.
Returns a ConditionComparison with per-condition results and the computed deltas (skills_on - no_skills, etc.).
run_condition(condition)
¶
Run the benchmark for condition across all configured seeds.
Returns a ConditionResult with mean ± stddev pass rate and the per-task / per-skill aggregations.
write_report(comparison)
¶
Write a markdown report to output_dir.
Filename: pinchbench-skills-eval-{YYYY-MM-DD}.md