Skip to content

make_configs

diapason.evals.comparison.make_configs

CLI for generating framework-comparison eval configs from _template.toml.

Usage

python -m diapason.evals.comparison.make_configs \ --framework hermes --model qwen-9b --benchmark gaia

main(framework, model, benchmark, output_dir, all_tier1)

Generate eval config TOMLs for the framework-comparison experiment.

materialize_config(framework, model, benchmark, output_dir=None, judge_model=DEFAULT_JUDGE_MODEL)

Materialize one TOML config from the template.