Skip to content

dataset

diapason.evals.core.dataset

Abstract base class for dataset providers.

DatasetProvider

Bases: ABC

Base class for all evaluation dataset providers.

create_task_env(record)

Return a task environment context manager, or None.

iter_episodes()

Iterate over episodes (groups of sequential records).

Default: each record is its own single-record episode. Override for benchmarks requiring sequential processing with shared agent state within an episode.

iter_records() abstractmethod

Iterate over loaded records.

load(*, max_samples=None, split=None, seed=None) abstractmethod

Load the dataset (possibly downloading from HuggingFace).

size() abstractmethod

Return the number of loaded records.

verify_requirements()

Return list of unsatisfied requirements, or empty list.