Index
diapason.evals.environments
¶
Task environments for interactive multi-turn evaluation.
TaskEnvironment
¶
Bases: ABC
Environment for multi-turn interactive evaluation.
Subclasses implement the reset/step/evaluate lifecycle:
1. reset(record) — initialize the environment, return initial observation
2. step(agent_response) — parse agent action, execute, return feedback
3. evaluate() — assess final state, return (is_correct, metadata)
4. close() — release resources
max_turns
property
¶
Maximum interaction turns for this environment.
Subclasses should override to match the original benchmark's per-task-type turn limits. Default: 15.
close()
¶
Release resources (Docker containers, DB connections, etc.).
evaluate()
abstractmethod
¶
Evaluate the final state after interaction completes.
Returns:
| Name | Type | Description |
|---|---|---|
is_correct |
Optional[bool]
|
True/False/None (None = not scorable). |
metadata |
Dict[str, Any]
|
Scoring details dict. |
reset(record)
abstractmethod
¶
Initialize environment for a record.
Returns the initial observation/context for the agent (e.g. schema description, entity list, task instructions).
run_agent_loop(generate_fn, record)
¶
Run the full reset → [generate → step] × N → evaluate cycle.
Called by AgenticRunner for environments that use the reset/step/evaluate protocol instead of one-shot generation.
Maintains full conversation history across turns so the agent
retains context. Strips <think> tags from agent responses
before passing to step().
After completion, sets on self:
- last_eval_result: (is_correct, metadata)
- all_responses: list of raw agent responses per turn
- turn_wall_clocks: list of per-turn wall clock seconds
- interaction_history: message list (for lifelong injection)
step(agent_response)
abstractmethod
¶
Process an agent response.
Returns:
| Name | Type | Description |
|---|---|---|
observation |
str
|
Feedback text to show the agent (e.g. SQL result, API return value, bash output). |
is_done |
bool
|
True if the agent signaled completion (e.g. |