runner
diapason.evals.core.runner
¶
EvalRunner — parallel execution of evaluation samples.
Supports two modes:
- Parallel mode (default): Samples processed concurrently via ThreadPoolExecutor.
- Episode mode (episode_mode=True): Samples processed sequentially within
episodes, with in-context example injection from prior successful completions.
Required for lifelong-learning benchmarks like LifelongAgentBench.
When a dataset provides create_task_env() returning a TaskEnvironment,
samples are evaluated via multi-turn interactive loops instead of single-shot
generation — matching benchmarks that require agent-environment interaction.
EvalRunner
¶
Runs an evaluation benchmark with parallel sample execution.
results
property
¶
Return a copy of collected evaluation results.
run(progress_callback=None)
¶
Execute the evaluation and return a summary.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
progress_callback
|
Optional[Callable[[int, int], None]]
|
Optional |
None
|