Skip to content

Index

diapason.evals.environments

Task environments for interactive multi-turn evaluation.

TaskEnvironment

Bases: ABC

Environment for multi-turn interactive evaluation.

Subclasses implement the reset/step/evaluate lifecycle: 1. reset(record) — initialize the environment, return initial observation 2. step(agent_response) — parse agent action, execute, return feedback 3. evaluate() — assess final state, return (is_correct, metadata) 4. close() — release resources

max_turns property

Maximum interaction turns for this environment.

Subclasses should override to match the original benchmark's per-task-type turn limits. Default: 15.

close()

Release resources (Docker containers, DB connections, etc.).

evaluate() abstractmethod

Evaluate the final state after interaction completes.

Returns:

Name Type Description
is_correct Optional[bool]

True/False/None (None = not scorable).

metadata Dict[str, Any]

Scoring details dict.

reset(record) abstractmethod

Initialize environment for a record.

Returns the initial observation/context for the agent (e.g. schema description, entity list, task instructions).

run_agent_loop(generate_fn, record)

Run the full reset → [generate → step] × N → evaluate cycle.

Called by AgenticRunner for environments that use the reset/step/evaluate protocol instead of one-shot generation.

Maintains full conversation history across turns so the agent retains context. Strips <think> tags from agent responses before passing to step().

After completion, sets on self: - last_eval_result: (is_correct, metadata) - all_responses: list of raw agent responses per turn - turn_wall_clocks: list of per-turn wall clock seconds - interaction_history: message list (for lifelong injection)

step(agent_response) abstractmethod

Process an agent response.

Returns:

Name Type Description
observation str

Feedback text to show the agent (e.g. SQL result, API return value, bash output).

is_done bool

True if the agent signaled completion (e.g. Action: Answer, Final Answer:, Act: finish).