Skip to content

Index

diapason.learning

Learning primitive -- router policies, reward functions, learning.

AgentConfigEvolver

Analyze traces to evolve agent TOML configs with versioning.

Parameters

trace_store: A :class:TraceStore used to fetch historical traces. config_dir: Directory where agent TOML configs are written. min_quality: Minimum average feedback score for a recommendation to be emitted.

analyze()

Analyze traces, return recommendations per query class.

Returns a list of dicts, each containing: - query_class: the classified query category - recommended_tools: list of tool names sorted by frequency - recommended_agent: the best-performing agent for this class - recommended_max_turns: suggested max_turns value - sample_count: number of traces analyzed for this class

list_versions(agent_name)

List all versions (including current) for agent_name.

Returns a list of dicts with version, path, and modified. Versions are numbered starting from 1 (oldest archived) through to the current (highest version number).

rollback(agent_name, version)

Rollback to a specific version.

Raises :class:ValueError if the requested version does not exist.

write_config(agent_name, *, tools, max_turns=10, temperature=0.3, system_prompt='')

Write agent TOML config, archiving previous version first.

Returns the :class:Path to the written config file.

ComplexityQueryAnalyzer

Bases: QueryAnalyzer

Query analyzer that produces a complexity-aware RoutingContext.

Drop-in replacement for DefaultQueryAnalyzer — adds complexity_score, has_reasoning, and suggested_max_tokens to the returned RoutingContext.

HeuristicRewardFunction

Bases: RewardFunction

Computes a scalar reward based on latency, cost, and token efficiency.

Each component is normalised to [0, 1] and combined via a weighted sum.

HeuristicRouter

Bases: RouterPolicy

Rule-based model router.

Rules (applied in order): 1. Code detected → prefer model with "code"/"coder" in name 2. Math detected → prefer larger model 3. Low complexity (score < 0.20) → prefer smaller/faster model 4. High complexity (score >= 0.55 OR reasoning keywords) → prefer larger model 5. High urgency (>0.8) → override to smaller model 6. Default fallback → default_model → fallback_model → first available

LLMOptimizer

Uses a cloud LLM to propose optimal Diapason configs.

Inspired by DSPy's GEPA: uses textual feedback from execution traces rather than just scalar rewards.

analyze_trial(trial, summary, traces=None, sample_scores=None, per_benchmark=None)

Ask the LLM to analyze a completed trial. Returns structured feedback.

propose_initial()

Propose a reasonable starting config from the search space.

propose_merge(candidates, history, frontier_ids=None)

Combine best aspects of frontier members into one config.

propose_next(history, traces=None, frontier_ids=None)

Ask the LLM to propose the next config to evaluate.

propose_targeted(history, base_config, target_primitive, frontier_ids=None)

Propose a config that only changes one primitive.

LearningOrchestrator

Orchestrate a single trace->learn->eval cycle.

Parameters

trace_store: Object with list_traces(limit=...) returning List[Trace] (typically a :class:TraceStore). config_dir: Directory where agent TOML configs are written / evolved. eval_fn: Optional callable returning a float score (higher = better). Called before and after learning to gate acceptance. min_improvement: Minimum improvement in eval score required to accept the update. min_sft_pairs: Minimum number of SFT pairs required to trigger LoRA training. min_quality: Minimum feedback quality threshold for :class:TrainingDataMiner. lora_config: Optional :class:LoRATrainingConfig. When provided (and enough SFT pairs exist and torch is available), LoRA training runs. model_name: Model name for LoRA training (passed to :class:LoRATrainer).

run(*, agent_id=None)

Execute one learning cycle.

Parameters

agent_id: When provided, only traces from this agent are considered.

Returns

Dict[str, Any] A result with at least timestamp and status keys.

Steps
  1. Mine traces: extract sft_pairs, routing_pairs, agent_pairs
  2. If no data: return skipped
  3. Run baseline eval (if eval_fn provided)
  4. Update routing recommendations
  5. Evolve agent configs
  6. Run LoRA training (if lora_config provided AND enough pairs AND torch available)
  7. Run post-learning eval (if eval_fn provided)
  8. Accept/reject based on improvement threshold

LoRATrainer

Fine-tune a local causal LM with LoRA (or QLoRA) adapters.

Parameters

config: LoRA training configuration. model_name: HuggingFace model identifier or local path. device: PyTorch device string. None auto-detects (cuda > mps > cpu).

Raises

ImportError If torch is not installed.

prepare_dataset(pairs)

Convert SFT pairs to tokenized examples.

Each returned dict contains input_ids, attention_mask, and text (the raw formatted string before tokenization).

Parameters

pairs: List of dicts with at least input and output keys, as produced by :class:TrainingDataMiner.extract_sft_pairs.

train(pairs)

Run LoRA fine-tuning on the given SFT pairs.

Parameters

pairs: List of dicts with at least input and output keys.

Returns

dict Training summary with keys: status, epochs, total_steps, avg_loss, adapter_path, training_samples.

LoRATrainingConfig dataclass

Configuration for LoRA / QLoRA fine-tuning.

OptimizationEngine

Orchestrates the optimize loop: propose -> evaluate -> analyze -> repeat.

export_best_recipe(run, path)

Export the best trial's config as a TOML recipe file.

Parameters:

Name Type Description Default
run OptimizationRun

A completed :class:OptimizationRun.

required
path Path

Destination path for the TOML file.

required

Returns:

Type Description
Path

The path written to.

Raises:

Type Description
ValueError

If there is no best trial in the run.

run(progress_callback=None)

Execute the full optimization loop.

  1. Generate a run_id via uuid.
  2. llm_optimizer.propose_initial() -> first config.
  3. Loop up to max_trials: a. trial_runner.run_trial(config) -> TrialResult b. llm_optimizer.analyze_trial(config, summary, traces) c. Update TrialResult with analysis text d. Append to history e. If store, store.save_trial(result) f. Update best_trial if accuracy improved g. Check early stopping (no improvement for patience trials) h. If not stopped, llm_optimizer.propose_next(history)
  4. Set run status to "completed".
  5. If store, store.save_run(optimization_run).
  6. Return the :class:OptimizationRun.

Parameters:

Name Type Description Default
progress_callback Optional[Callable[[int, int], None]]

Optional (trial_num, max_trials) -> None called after each trial completes.

None

OptimizationStore

SQLite-backed storage for optimization runs and trials.

close()

Close the underlying SQLite connection.

get_run(run_id)

Retrieve an optimization run by id, or None.

get_trials(run_id)

Retrieve all trial results for a given run.

list_runs(limit=50)

Return summary dicts of recent optimization runs.

save_run(run)

Persist an optimization run (insert or update).

save_trial(run_id, trial)

Persist a single trial result.

QueryAnalyzer

Bases: ABC

Query analysis for routing contexts.

analyze(query, **kwargs) abstractmethod

Analyze a query and return a RoutingContext.

RewardFunction

Bases: ABC

Compute a scalar reward for a routing decision.

compute(context, model_key, response, **kwargs) abstractmethod

Return reward in [0, 1].

RouterPolicy

Bases: ABC

Model selection policy (used by the learning system).

select_model(context) abstractmethod

Select the best model key for the given routing context.

RoutingContext dataclass

Context describing a query for model routing decisions.

TrainingDataMiner

Extract supervised training pairs from stored traces.

Parameters

trace_store: Any object with a list_traces(limit=...) method returning List[Trace] (typically a :class:TraceStore). min_quality: Minimum feedback score for a trace to be included. min_samples_per_class: Minimum number of samples a query class must have to appear in routing/agent-config results.

extract_agent_config_pairs(*, agent=None)

Return per-query-class agent and tool recommendations.

Returns a dict mapping query class to:

  • best_agent — agent with the highest average feedback.
  • best_tools — most frequently used tools by the best agent.
  • avg_feedback — average feedback across all agents for the class.
  • sample_count — total number of qualifying traces in the class.

extract_routing_pairs(*, agent=None)

Return per-query-class routing recommendations.

Returns a dict mapping query class to:

  • best_model — model with highest average feedback for the class.
  • avg_feedback — average feedback across all models for the class.
  • sample_count — total number of qualifying traces in the class.
  • all_models — dict of {model: {"avg_feedback": float, "count": int}}.

extract_sft_pairs(*, agent=None)

Return SFT training pairs from high-quality traces.

Each entry is a dict with keys: input, output, query_class, model, feedback.

Duplicate (input, output) pairs are collapsed; the first occurrence is kept.

build_routing_context(query, *, urgency=0.5, model=None)

Populate a RoutingContext from a raw query string.

When model is provided, the suggested token budget is adjusted for thinking models that need extra headroom.

ensure_registered()

Ensure all learning policies are registered in RouterPolicyRegistry.

score_complexity(query)

Score a query's complexity from 0.0 (trivial) to 1.0 (very complex).

The score is a weighted combination of independent signals, each contributing a fraction between 0 and 1. Weights reflect how much each signal correlates with the amount of reasoning and output tokens a model will need.