benchmark_gate
diapason.learning.spec_search.gate.benchmark_gate
¶
BenchmarkGate: accept or reject edits based on benchmark performance.
Runs the personal benchmark via a provided scorer callable, compares before/after snapshots, and decides accept/reject based on thresholds.
See spec §7.3.
BenchmarkGate
¶
Runs the personal benchmark and decides accept/reject.
Parameters¶
scorer :
Callable that runs the benchmark and returns a BenchmarkSnapshot.
Signature: (benchmark_version, subsample_size, seed) -> BenchmarkSnapshot.
benchmark_version :
Which benchmark version to score against (locked per session).
min_improvement :
Minimum overall score improvement to accept (default 0.0).
max_regression :
Maximum per-cluster score drop before rejecting (default 0.05).
subsample_size :
Number of tasks to score per gate run (default 50).
evaluate(*, before, session_seed)
¶
GateResult
dataclass
¶
Result of a benchmark gate evaluation.