terminalbench_v2_1
diapason.evals.datasets.terminalbench_v2_1
¶
TerminalBench V2.1 dataset provider.
Loads tasks from the terminal-bench-2.1 repo layout (ekellbuch/terminal-bench-2, branch terminal-bench-2.1). Each task lives in a top-level directory containing:
<task_name>/
task.toml # metadata + docker image + timeouts
instruction.md # the agent prompt
environment/ # Dockerfile + supporting files (pre-built into task.toml's docker_image)
solution/ # oracle solve.sh (not used by eval)
tests/ # test.sh + test_outputs.py (pytest) used by the verifier
Reference: https://github.com/ekellbuch/terminal-bench-2/tree/terminal-bench-2.1
TerminalBenchV21Dataset
¶
Bases: DatasetProvider
TerminalBench V2.1 dataset (89 Docker-based terminal tasks).