Skip to content

runner

diapason.evals.core.runner

EvalRunner — parallel execution of evaluation samples.

Supports two modes: - Parallel mode (default): Samples processed concurrently via ThreadPoolExecutor. - Episode mode (episode_mode=True): Samples processed sequentially within episodes, with in-context example injection from prior successful completions. Required for lifelong-learning benchmarks like LifelongAgentBench.

When a dataset provides create_task_env() returning a TaskEnvironment, samples are evaluated via multi-turn interactive loops instead of single-shot generation — matching benchmarks that require agent-environment interaction.

EvalRunner

Runs an evaluation benchmark with parallel sample execution.

results property

Return a copy of collected evaluation results.

run(progress_callback=None)

Execute the evaluation and return a summary.

Parameters:

Name Type Description Default
progress_callback Optional[Callable[[int, int], None]]

Optional (completed, total) callback invoked after each sample completes, useful for driving progress bars.

None