config
diapason.evals.core.config
¶
TOML config loader and matrix expansion for eval suites.
EvalConfigError
¶
Bases: Exception
Raised for structural issues in eval config files.
expand_suite(suite)
¶
Expand an EvalSuiteConfig into a list of RunConfigs (models x benchmarks).
Merge precedence (highest wins): benchmark-level > model-level > [defaults] > built-in defaults
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
suite
|
EvalSuiteConfig
|
The parsed eval suite config. |
required |
Returns:
| Type | Description |
|---|---|
List[RunConfig]
|
List of RunConfig, one per model-benchmark pair. |
load_eval_config(path)
¶
Load and validate an eval suite config from a TOML file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str | Path
|
Path to the TOML config file. |
required |
Returns:
| Type | Description |
|---|---|
EvalSuiteConfig
|
Validated EvalSuiteConfig. |
Raises:
| Type | Description |
|---|---|
EvalConfigError
|
On structural validation failures. |
FileNotFoundError
|
If the config file doesn't exist. |