coding_assistant
diapason.evals.scorers.coding_assistant
¶
coding_assistant scorer — test-based evaluation of bug fixes.
Extracts fixed code from model output, runs the test suite, and computes: - fix_rate: fraction of originally-failing tests now passing - regressions: count of originally-passing tests that broke