livecodebench
diapason.evals.scorers.livecodebench
¶
LiveCodeBench scorer — sandboxed code execution with test cases.
Extracts code from model output, runs it against test cases in a sandboxed subprocess with timeout and resource limits, and scores based on pass/fail of each test case.
Reference: https://livecodebench.github.io/