dataset
diapason.evals.core.dataset
¶
Abstract base class for dataset providers.
DatasetProvider
¶
Bases: ABC
Base class for all evaluation dataset providers.
create_task_env(record)
¶
Return a task environment context manager, or None.
iter_episodes()
¶
Iterate over episodes (groups of sequential records).
Default: each record is its own single-record episode. Override for benchmarks requiring sequential processing with shared agent state within an episode.
iter_records()
abstractmethod
¶
Iterate over loaded records.
load(*, max_samples=None, split=None, seed=None)
abstractmethod
¶
Load the dataset (possibly downloading from HuggingFace).
size()
abstractmethod
¶
Return the number of loaded records.
verify_requirements()
¶
Return list of unsatisfied requirements, or empty list.