Skip to content

config

diapason.evals.core.config

TOML config loader and matrix expansion for eval suites.

EvalConfigError

Bases: Exception

Raised for structural issues in eval config files.

expand_suite(suite)

Expand an EvalSuiteConfig into a list of RunConfigs (models x benchmarks).

Merge precedence (highest wins): benchmark-level > model-level > [defaults] > built-in defaults

Parameters:

Name Type Description Default
suite EvalSuiteConfig

The parsed eval suite config.

required

Returns:

Type Description
List[RunConfig]

List of RunConfig, one per model-benchmark pair.

load_eval_config(path)

Load and validate an eval suite config from a TOML file.

Parameters:

Name Type Description Default
path str | Path

Path to the TOML config file.

required

Returns:

Type Description
EvalSuiteConfig

Validated EvalSuiteConfig.

Raises:

Type Description
EvalConfigError

On structural validation failures.

FileNotFoundError

If the config file doesn't exist.