Skip to content

lifelong_agent_env

diapason.evals.environments.lifelong_agent_env

Multi-turn task environments for LifelongAgentBench.

Implements faithful reproductions of the original's interaction protocols: - DB: MySQL Docker container (SQLite fallback with degraded-mode warning) - KG: Variable-store API simulation (SPARQL endpoint if configured) - OS: Docker container with correct image

Reference: https://github.com/caixd-220529/LifelongAgentBench

DBEnvironment

Bases: TaskEnvironment

Multi-turn DB environment matching the original's interaction protocol.

The original uses MySQL Docker containers. We try MySQL first and fall back to SQLite with a clear degraded-mode warning.

KGEnvironment

Bases: TaskEnvironment

KG API simulation environment.

The original uses a Freebase SPARQL endpoint. When no endpoint is configured, we simulate API calls using the gold action_list from the dataset as oracle responses and clearly warn about degraded mode.

Metrics match the original: exact-set match + F1 on answer entities.

OSEnvironment

Bases: TaskEnvironment

Docker-based OS interaction environment.

Matches the original's three-phase protocol: 1. Start container, run initialization_command_item 2. Agent sends bash commands, gets output 3. Run evaluation_command_item — pass iff exit_code == 0

TaskEnvironment

Bases: ABC

Environment for multi-turn interactive evaluation.

Subclasses implement the reset/step/evaluate lifecycle: 1. reset(record) — initialize the environment, return initial observation 2. step(agent_response) — parse agent action, execute, return feedback 3. evaluate() — assess final state, return (is_correct, metadata) 4. close() — release resources

max_turns property

Maximum interaction turns for this environment.

Subclasses should override to match the original benchmark's per-task-type turn limits. Default: 15.

close()

Release resources (Docker containers, DB connections, etc.).

evaluate() abstractmethod

Evaluate the final state after interaction completes.

Returns:

Name Type Description
is_correct Optional[bool]

True/False/None (None = not scorable).

metadata Dict[str, Any]

Scoring details dict.

reset(record) abstractmethod

Initialize environment for a record.

Returns the initial observation/context for the agent (e.g. schema description, entity list, task instructions).

run_agent_loop(generate_fn, record)

Run the full reset → [generate → step] × N → evaluate cycle.

Called by AgenticRunner for environments that use the reset/step/evaluate protocol instead of one-shot generation.

Maintains full conversation history across turns so the agent retains context. Strips <think> tags from agent responses before passing to step().

After completion, sets on self: - last_eval_result: (is_correct, metadata) - all_responses: list of raw agent responses per turn - turn_wall_clocks: list of per-turn wall clock seconds - interaction_history: message list (for lifelong injection)

step(agent_response) abstractmethod

Process an agent response.

Returns:

Name Type Description
observation str

Feedback text to show the agent (e.g. SQL result, API return value, bash output).

is_done bool

True if the agent signaled completion (e.g. Action: Answer, Final Answer:, Act: finish).

create_task_environment(record, *, sparql_endpoint=None, os_image=None, os_timeout=120)

Factory: create the right environment for a LifelongAgentBench record.