Skip to content

terminalbench_v2_1

diapason.evals.scorers.terminalbench_v2_1

TerminalBench V2.1 scorer.

Two modes:

  1. Agentic mode (preferred) — the dataset's create_task_env spun up a container, the agent interacted with it through docker_shell_exec, and on context exit the env ran tests/test.sh and wrote tbv21_reward into record.metadata. The scorer just reads that value.

  2. One-shot mode (fallback) — no env was attached. The model answer is treated as a bash script (extracted from a bash ... fence if present). The scorer runs the script in the task container and then the tests, same as before.

This means the same scorer supports both backend = "diapason-direct" and backend = "diapason-agent" TB v2.1 configs.

TerminalBenchV21Scorer

Bases: Scorer

Reward = 1 if the task's tests pass, 0 otherwise.