Skip to content

Index

diapason.learning.optimize

Optimization framework for Diapason configuration tuning.

BenchmarkScore dataclass

Per-benchmark metrics from a multi-benchmark evaluation trial.

BenchmarkSpec dataclass

Specification for one benchmark in a multi-benchmark optimization.

LLMOptimizer

Uses a cloud LLM to propose optimal Diapason configs.

Inspired by DSPy's GEPA: uses textual feedback from execution traces rather than just scalar rewards.

analyze_trial(trial, summary, traces=None, sample_scores=None, per_benchmark=None)

Ask the LLM to analyze a completed trial. Returns structured feedback.

propose_initial()

Propose a reasonable starting config from the search space.

propose_merge(candidates, history, frontier_ids=None)

Combine best aspects of frontier members into one config.

propose_next(history, traces=None, frontier_ids=None)

Ask the LLM to propose the next config to evaluate.

propose_targeted(history, base_config, target_primitive, frontier_ids=None)

Propose a config that only changes one primitive.

MultiBenchTrialRunner

Evaluates a proposed config across multiple benchmarks.

Delegates to :class:TrialRunner per benchmark, then aggregates results into a single composite :class:TrialResult with weighted metrics and per-benchmark breakdowns.

run_trial(trial)

Run trial against all benchmarks and return a composite result.

ObjectiveSpec dataclass

A single optimization objective.

OptimizationEngine

Orchestrates the optimize loop: propose -> evaluate -> analyze -> repeat.

export_best_recipe(run, path)

Export the best trial's config as a TOML recipe file.

Parameters:

Name Type Description Default
run OptimizationRun

A completed :class:OptimizationRun.

required
path Path

Destination path for the TOML file.

required

Returns:

Type Description
Path

The path written to.

Raises:

Type Description
ValueError

If there is no best trial in the run.

run(progress_callback=None)

Execute the full optimization loop.

  1. Generate a run_id via uuid.
  2. llm_optimizer.propose_initial() -> first config.
  3. Loop up to max_trials: a. trial_runner.run_trial(config) -> TrialResult b. llm_optimizer.analyze_trial(config, summary, traces) c. Update TrialResult with analysis text d. Append to history e. If store, store.save_trial(result) f. Update best_trial if accuracy improved g. Check early stopping (no improvement for patience trials) h. If not stopped, llm_optimizer.propose_next(history)
  4. Set run status to "completed".
  5. If store, store.save_run(optimization_run).
  6. Return the :class:OptimizationRun.

Parameters:

Name Type Description Default
progress_callback Optional[Callable[[int, int], None]]

Optional (trial_num, max_trials) -> None called after each trial completes.

None

OptimizationRun dataclass

Complete optimization session.

OptimizationStore

SQLite-backed storage for optimization runs and trials.

close()

Close the underlying SQLite connection.

get_run(run_id)

Retrieve an optimization run by id, or None.

get_trials(run_id)

Retrieve all trial results for a given run.

list_runs(limit=50)

Return summary dicts of recent optimization runs.

save_run(run)

Persist an optimization run (insert or update).

save_trial(run_id, trial)

Persist a single trial result.

SampleScore dataclass

Per-sample metrics from an evaluation trial.

SearchDimension dataclass

One tunable dimension in the config space.

SearchSpace dataclass

The full space of configs the optimizer can propose.

to_prompt_description()

Render search space as structured text for the LLM optimizer.

TrialConfig dataclass

A single candidate configuration proposed by the optimizer.

to_recipe()

Map params back to Recipe fields.

TrialFeedback dataclass

Structured feedback from trial analysis.

TrialResult dataclass

Result of evaluating a trial, with both scalar and textual feedback.

TrialRunner

Evaluates a proposed config against a benchmark.

Bridges the optimization types (:class:TrialConfig) to the eval framework (:class:EvalRunner) so the optimizer can score candidate configurations end-to-end.

run_trial(trial)

Run trial against the configured benchmark and return a result.

Steps: 1. Convert trial to a :class:Recipe and extract params. 2. Build a :class:RunConfig from recipe + benchmark settings. 3. Lazily import eval-framework registries to resolve the benchmark -> dataset + scorer, and build the backend. 4. Execute via EvalRunner.run() -> :class:RunSummary. 5. Map the summary into a :class:TrialResult.

build_search_space(config)

Build a SearchSpace from a TOML-style config dict.

Expected format::

{
    "optimize": {
        "search": [
            {
                "name": "agent.type",
                "type": "categorical",
                "values": ["orchestrator", "native_react"],
                "description": "Agent architecture",
            },
            {
                "name": "intelligence.temperature",
                "type": "continuous",
                "low": 0.0,
                "high": 1.0,
                "description": "Generation temperature",
            },
        ],
        "fixed": {"engine": "ollama", "model": "qwen3:8b"},
        "constraints": {
            "rules": ["SimpleAgent should only have max_turns = 1"],
        },
    }
}

compute_pareto_frontier(trials, objectives)

Compute the Pareto frontier: trials not dominated by any other.

A trial A dominates trial B if A is >= B on all objectives and > B on at least one (direction-aware: maximize flips the comparison).

load_benchmark_specs(data)

Extract benchmark specs from a loaded optimization config.

Supports two formats: - Multi-benchmark: [[optimize.benchmarks]] array of tables - Single-benchmark fallback: optimize.benchmark string

Returns a list of :class:BenchmarkSpec (from trial_runner). Returns an empty list if no benchmarks are configured (caller should fall back to CLI --benchmark).

load_objectives(data)

Extract objectives from a loaded optimization config.

Reads optimize.objectives (a list of tables) and returns a list of :class:ObjectiveSpec. Falls back to :data:DEFAULT_OBJECTIVES if the key is absent.

load_optimize_config(path)

Load an optimization config TOML file.

Returns the raw dict with keys such as optimize.max_trials, optimize.benchmark, optimize.search, optimize.fixed, optimize.constraints, etc.

Raises:

Type Description
FileNotFoundError

If path does not exist.