Index
diapason.learning.optimize
¶
Optimization framework for Diapason configuration tuning.
BenchmarkScore
dataclass
¶
Per-benchmark metrics from a multi-benchmark evaluation trial.
BenchmarkSpec
dataclass
¶
Specification for one benchmark in a multi-benchmark optimization.
LLMOptimizer
¶
Uses a cloud LLM to propose optimal Diapason configs.
Inspired by DSPy's GEPA: uses textual feedback from execution traces rather than just scalar rewards.
analyze_trial(trial, summary, traces=None, sample_scores=None, per_benchmark=None)
¶
Ask the LLM to analyze a completed trial. Returns structured feedback.
propose_initial()
¶
Propose a reasonable starting config from the search space.
propose_merge(candidates, history, frontier_ids=None)
¶
Combine best aspects of frontier members into one config.
propose_next(history, traces=None, frontier_ids=None)
¶
Ask the LLM to propose the next config to evaluate.
propose_targeted(history, base_config, target_primitive, frontier_ids=None)
¶
Propose a config that only changes one primitive.
MultiBenchTrialRunner
¶
Evaluates a proposed config across multiple benchmarks.
Delegates to :class:TrialRunner per benchmark, then aggregates
results into a single composite :class:TrialResult with weighted
metrics and per-benchmark breakdowns.
run_trial(trial)
¶
Run trial against all benchmarks and return a composite result.
ObjectiveSpec
dataclass
¶
A single optimization objective.
OptimizationEngine
¶
Orchestrates the optimize loop: propose -> evaluate -> analyze -> repeat.
export_best_recipe(run, path)
¶
Export the best trial's config as a TOML recipe file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
run
|
OptimizationRun
|
A completed :class: |
required |
path
|
Path
|
Destination path for the TOML file. |
required |
Returns:
| Type | Description |
|---|---|
Path
|
The path written to. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If there is no best trial in the run. |
run(progress_callback=None)
¶
Execute the full optimization loop.
- Generate a run_id via uuid.
llm_optimizer.propose_initial()-> first config.- Loop up to
max_trials: a.trial_runner.run_trial(config)-> TrialResult b.llm_optimizer.analyze_trial(config, summary, traces)c. Update TrialResult with analysis text d. Append to history e. If store,store.save_trial(result)f. Update best_trial if accuracy improved g. Check early stopping (no improvement for patience trials) h. If not stopped,llm_optimizer.propose_next(history) - Set run status to
"completed". - If store,
store.save_run(optimization_run). - Return the :class:
OptimizationRun.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
progress_callback
|
Optional[Callable[[int, int], None]]
|
Optional |
None
|
OptimizationRun
dataclass
¶
Complete optimization session.
OptimizationStore
¶
SQLite-backed storage for optimization runs and trials.
close()
¶
Close the underlying SQLite connection.
get_run(run_id)
¶
Retrieve an optimization run by id, or None.
get_trials(run_id)
¶
Retrieve all trial results for a given run.
list_runs(limit=50)
¶
Return summary dicts of recent optimization runs.
save_run(run)
¶
Persist an optimization run (insert or update).
save_trial(run_id, trial)
¶
Persist a single trial result.
SampleScore
dataclass
¶
Per-sample metrics from an evaluation trial.
SearchDimension
dataclass
¶
One tunable dimension in the config space.
SearchSpace
dataclass
¶
The full space of configs the optimizer can propose.
to_prompt_description()
¶
Render search space as structured text for the LLM optimizer.
TrialConfig
dataclass
¶
A single candidate configuration proposed by the optimizer.
to_recipe()
¶
Map params back to Recipe fields.
TrialFeedback
dataclass
¶
Structured feedback from trial analysis.
TrialResult
dataclass
¶
Result of evaluating a trial, with both scalar and textual feedback.
TrialRunner
¶
Evaluates a proposed config against a benchmark.
Bridges the optimization types (:class:TrialConfig) to the eval
framework (:class:EvalRunner) so the optimizer can score candidate
configurations end-to-end.
run_trial(trial)
¶
Run trial against the configured benchmark and return a result.
Steps:
1. Convert trial to a :class:Recipe and extract params.
2. Build a :class:RunConfig from recipe + benchmark settings.
3. Lazily import eval-framework registries to resolve the
benchmark -> dataset + scorer, and build the backend.
4. Execute via EvalRunner.run() -> :class:RunSummary.
5. Map the summary into a :class:TrialResult.
build_search_space(config)
¶
Build a SearchSpace from a TOML-style config dict.
Expected format::
{
"optimize": {
"search": [
{
"name": "agent.type",
"type": "categorical",
"values": ["orchestrator", "native_react"],
"description": "Agent architecture",
},
{
"name": "intelligence.temperature",
"type": "continuous",
"low": 0.0,
"high": 1.0,
"description": "Generation temperature",
},
],
"fixed": {"engine": "ollama", "model": "qwen3:8b"},
"constraints": {
"rules": ["SimpleAgent should only have max_turns = 1"],
},
}
}
compute_pareto_frontier(trials, objectives)
¶
Compute the Pareto frontier: trials not dominated by any other.
A trial A dominates trial B if A is >= B on all objectives and > B on at least one (direction-aware: maximize flips the comparison).
load_benchmark_specs(data)
¶
Extract benchmark specs from a loaded optimization config.
Supports two formats:
- Multi-benchmark: [[optimize.benchmarks]] array of tables
- Single-benchmark fallback: optimize.benchmark string
Returns a list of :class:BenchmarkSpec (from trial_runner).
Returns an empty list if no benchmarks are configured (caller
should fall back to CLI --benchmark).
load_objectives(data)
¶
Extract objectives from a loaded optimization config.
Reads optimize.objectives (a list of tables) and returns
a list of :class:ObjectiveSpec. Falls back to
:data:DEFAULT_OBJECTIVES if the key is absent.
load_optimize_config(path)
¶
Load an optimization config TOML file.
Returns the raw dict with keys such as optimize.max_trials,
optimize.benchmark, optimize.search, optimize.fixed,
optimize.constraints, etc.
Raises:
| Type | Description |
|---|---|
FileNotFoundError
|
If path does not exist. |