Skip to content

terminalbench_native_structural

diapason.evals.scorers.terminalbench_native_structural

TerminalBench Native scorer — test-result-based evaluation.

Reads is_resolved and test_results from the record's metadata (populated by the native terminal-bench harness) and returns a deterministic pass/fail without any LLM judging.

TerminalBenchNativeScorer

Bases: Scorer

Test-result-based scorer for TerminalBench Native tasks.

The native terminal-bench package produces is_resolved and test_results fields after executing a task. This scorer reads those fields from record.metadata and translates them into the standard (is_correct, meta) tuple.