Skip to content

benchmark_gate

diapason.learning.spec_search.gate.benchmark_gate

BenchmarkGate: accept or reject edits based on benchmark performance.

Runs the personal benchmark via a provided scorer callable, compares before/after snapshots, and decides accept/reject based on thresholds.

See spec §7.3.

BenchmarkGate

Runs the personal benchmark and decides accept/reject.

Parameters

scorer : Callable that runs the benchmark and returns a BenchmarkSnapshot. Signature: (benchmark_version, subsample_size, seed) -> BenchmarkSnapshot. benchmark_version : Which benchmark version to score against (locked per session). min_improvement : Minimum overall score improvement to accept (default 0.0). max_regression : Maximum per-cluster score drop before rejecting (default 0.05). subsample_size : Number of tasks to score per gate run (default 50).

evaluate(*, before, session_seed)

Run the benchmark and compare against the before snapshot.

Parameters

before : Snapshot captured before the edit was applied. session_seed : Deterministic seed for subsampling (same across all gate runs in one session).

Returns

GateResult accepted is True if the edit should be committed.

GateResult dataclass

Result of a benchmark gate evaluation.