Skip to content

environment

diapason.evals.core.environment

Environment provider ABC for benchmarks requiring external environments.

EnvironmentProvider

Bases: ABC

Manages an external environment for evaluation benchmarks.

Provides lifecycle management (setup/reset/teardown) and environment-state validation for benchmarks that need Docker containers, ServiceNow instances, or other live systems.

reset() abstractmethod

Reset environment state between tasks.

Called between records within an episode to restore the environment to a known state.

setup() abstractmethod

Start the environment and return connection info.

Returns a dict with environment-specific connection details (e.g., URLs, ports, credentials).

teardown() abstractmethod

Stop the environment and release resources.

validate(record) abstractmethod

Check environment state against expected outcome.

Parameters:

Name Type Description Default
record EvalRecord

The eval record containing the expected state in metadata.

required

Returns:

Type Description
bool

(is_correct, metadata) where is_correct indicates whether

Dict[str, Any]

the environment is in the expected state.

TaskEnvironmentError

Bases: RuntimeError

A task execution environment failed to start or operate.

Raised when infrastructure backing a task (Docker container, docker compose project, tmux session, recording binaries, ...) breaks. This is a harness/environment failure, not a model failure: runners record it distinctly (QueryTrace.error_kind == "harness_error") so scoring can exclude the sample from resolve-rate instead of silently counting it as a model miss.