terminalbench_native
diapason.evals.backends.terminalbench_native
¶
Native TerminalBench V2.1 backend.
Uses Harness for Docker-based execution and scoring.
TerminalBenchNativeBackend
¶
Bases: InferenceBackend
Runs terminal-bench tasks natively via Harness with Docker execution.
Uses terminal-bench's own agent + LiteLLM to call the model, Docker containers for task execution, and built-in test scripts for scoring. This gives real agentic evaluation, not text-only.
__init__(model='openai/default', api_base='http://localhost:8000/v1', temperature=0.2, agent_name='naive', output_dir='results/terminalbench/', max_samples=None, dataset_name='terminal-bench-core', dataset_version='0.1.1', system_prompt='', max_tokens=16384, n_concurrent=4, global_agent_timeout_sec=1800.0, global_timeout_multiplier=None)
¶
Args of note:
Hard wall-clock bound for each trial's
agent phase. terminal-bench runs installed-agent SETUP inside
this same budget with an infinite tmux timeout, so this bounds
SETUP+RUN together (a setup-only timeout needs an upstream
terminal-bench change). When set, it REPLACES each task's own
max_agent_timeout_sec. Set None or 0 to fall back
to per-task budgets.
global_timeout_multiplier: Scales per-task budgets when
global_agent_timeout_sec is not set. None keeps
terminal-bench's default (1.0).
run_harness(run_id)
¶
Run the full terminal-bench harness and return results.
summarize_benchmark_results(results, *, model, benchmark='terminalbench-native')
¶
Convert terminal-bench BenchmarkResults into a RunSummary.
Trials are classified into three buckets:
- resolved:
is_resolvedis True -> counted correct. - model miss: unresolved, but the model was actually contacted -> counted in the accuracy denominator.
- harness/infra failure: excluded from the accuracy denominator and
reported in
RunSummary.errorsplus the returned failure list.
Zero-model-contact signal choice: terminal-bench 0.2.18 leaves
failure_mode UNSET both on clean success and on genuine unresolved
misses, so failure_mode cannot distinguish "the model tried and failed"
from "the agent never called the model". Token usage can: this backend
always runs terminus-2, which reports real LiteLLM usage, so an
unresolved trial with zero/missing input+output tokens means no model
request ever completed — an infrastructure failure (in-container setup
hang/death, tmux failure), not a model miss. CAVEAT: terminal-bench
"installed agents" (openhands, claude-code, ...) hardcode 0 tokens even
on success; if this backend ever honors agent_name for installed
agents, this heuristic must be gated on the agent type.