Index
diapason.learning
¶
Learning primitive -- router policies, reward functions, learning.
AgentConfigEvolver
¶
Analyze traces to evolve agent TOML configs with versioning.
Parameters¶
trace_store:
A :class:TraceStore used to fetch historical traces.
config_dir:
Directory where agent TOML configs are written.
min_quality:
Minimum average feedback score for a recommendation to be emitted.
analyze()
¶
Analyze traces, return recommendations per query class.
Returns a list of dicts, each containing:
- query_class: the classified query category
- recommended_tools: list of tool names sorted by frequency
- recommended_agent: the best-performing agent for this class
- recommended_max_turns: suggested max_turns value
- sample_count: number of traces analyzed for this class
list_versions(agent_name)
¶
List all versions (including current) for agent_name.
Returns a list of dicts with version, path, and modified.
Versions are numbered starting from 1 (oldest archived) through to
the current (highest version number).
rollback(agent_name, version)
¶
Rollback to a specific version.
Raises :class:ValueError if the requested version does not exist.
write_config(agent_name, *, tools, max_turns=10, temperature=0.3, system_prompt='')
¶
Write agent TOML config, archiving previous version first.
Returns the :class:Path to the written config file.
ComplexityQueryAnalyzer
¶
Bases: QueryAnalyzer
Query analyzer that produces a complexity-aware RoutingContext.
Drop-in replacement for DefaultQueryAnalyzer — adds
complexity_score, has_reasoning, and suggested_max_tokens
to the returned RoutingContext.
HeuristicRewardFunction
¶
Bases: RewardFunction
Computes a scalar reward based on latency, cost, and token efficiency.
Each component is normalised to [0, 1] and combined via a weighted sum.
HeuristicRouter
¶
Bases: RouterPolicy
Rule-based model router.
Rules (applied in order): 1. Code detected → prefer model with "code"/"coder" in name 2. Math detected → prefer larger model 3. Low complexity (score < 0.20) → prefer smaller/faster model 4. High complexity (score >= 0.55 OR reasoning keywords) → prefer larger model 5. High urgency (>0.8) → override to smaller model 6. Default fallback → default_model → fallback_model → first available
LLMOptimizer
¶
Uses a cloud LLM to propose optimal Diapason configs.
Inspired by DSPy's GEPA: uses textual feedback from execution traces rather than just scalar rewards.
analyze_trial(trial, summary, traces=None, sample_scores=None, per_benchmark=None)
¶
Ask the LLM to analyze a completed trial. Returns structured feedback.
propose_initial()
¶
Propose a reasonable starting config from the search space.
propose_merge(candidates, history, frontier_ids=None)
¶
Combine best aspects of frontier members into one config.
propose_next(history, traces=None, frontier_ids=None)
¶
Ask the LLM to propose the next config to evaluate.
propose_targeted(history, base_config, target_primitive, frontier_ids=None)
¶
Propose a config that only changes one primitive.
LearningOrchestrator
¶
Orchestrate a single trace->learn->eval cycle.
Parameters¶
trace_store:
Object with list_traces(limit=...) returning List[Trace]
(typically a :class:TraceStore).
config_dir:
Directory where agent TOML configs are written / evolved.
eval_fn:
Optional callable returning a float score (higher = better).
Called before and after learning to gate acceptance.
min_improvement:
Minimum improvement in eval score required to accept the update.
min_sft_pairs:
Minimum number of SFT pairs required to trigger LoRA training.
min_quality:
Minimum feedback quality threshold for :class:TrainingDataMiner.
lora_config:
Optional :class:LoRATrainingConfig. When provided (and enough
SFT pairs exist and torch is available), LoRA training runs.
model_name:
Model name for LoRA training (passed to :class:LoRATrainer).
run(*, agent_id=None)
¶
Execute one learning cycle.
Parameters¶
agent_id: When provided, only traces from this agent are considered.
Returns¶
Dict[str, Any]
A result with at least timestamp and status keys.
Steps¶
- Mine traces: extract sft_pairs, routing_pairs, agent_pairs
- If no data: return skipped
- Run baseline eval (if eval_fn provided)
- Update routing recommendations
- Evolve agent configs
- Run LoRA training (if lora_config provided AND enough pairs AND torch available)
- Run post-learning eval (if eval_fn provided)
- Accept/reject based on improvement threshold
LoRATrainer
¶
Fine-tune a local causal LM with LoRA (or QLoRA) adapters.
Parameters¶
config:
LoRA training configuration.
model_name:
HuggingFace model identifier or local path.
device:
PyTorch device string. None auto-detects (cuda > mps > cpu).
Raises¶
ImportError
If torch is not installed.
prepare_dataset(pairs)
¶
Convert SFT pairs to tokenized examples.
Each returned dict contains input_ids, attention_mask,
and text (the raw formatted string before tokenization).
Parameters¶
pairs:
List of dicts with at least input and output keys,
as produced by :class:TrainingDataMiner.extract_sft_pairs.
LoRATrainingConfig
dataclass
¶
Configuration for LoRA / QLoRA fine-tuning.
OptimizationEngine
¶
Orchestrates the optimize loop: propose -> evaluate -> analyze -> repeat.
export_best_recipe(run, path)
¶
Export the best trial's config as a TOML recipe file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
run
|
OptimizationRun
|
A completed :class: |
required |
path
|
Path
|
Destination path for the TOML file. |
required |
Returns:
| Type | Description |
|---|---|
Path
|
The path written to. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If there is no best trial in the run. |
run(progress_callback=None)
¶
Execute the full optimization loop.
- Generate a run_id via uuid.
llm_optimizer.propose_initial()-> first config.- Loop up to
max_trials: a.trial_runner.run_trial(config)-> TrialResult b.llm_optimizer.analyze_trial(config, summary, traces)c. Update TrialResult with analysis text d. Append to history e. If store,store.save_trial(result)f. Update best_trial if accuracy improved g. Check early stopping (no improvement for patience trials) h. If not stopped,llm_optimizer.propose_next(history) - Set run status to
"completed". - If store,
store.save_run(optimization_run). - Return the :class:
OptimizationRun.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
progress_callback
|
Optional[Callable[[int, int], None]]
|
Optional |
None
|
OptimizationStore
¶
SQLite-backed storage for optimization runs and trials.
close()
¶
Close the underlying SQLite connection.
get_run(run_id)
¶
Retrieve an optimization run by id, or None.
get_trials(run_id)
¶
Retrieve all trial results for a given run.
list_runs(limit=50)
¶
Return summary dicts of recent optimization runs.
save_run(run)
¶
Persist an optimization run (insert or update).
save_trial(run_id, trial)
¶
Persist a single trial result.
QueryAnalyzer
¶
Bases: ABC
Query analysis for routing contexts.
analyze(query, **kwargs)
abstractmethod
¶
Analyze a query and return a RoutingContext.
RewardFunction
¶
Bases: ABC
Compute a scalar reward for a routing decision.
compute(context, model_key, response, **kwargs)
abstractmethod
¶
Return reward in [0, 1].
RouterPolicy
¶
Bases: ABC
Model selection policy (used by the learning system).
select_model(context)
abstractmethod
¶
Select the best model key for the given routing context.
RoutingContext
dataclass
¶
Context describing a query for model routing decisions.
TrainingDataMiner
¶
Extract supervised training pairs from stored traces.
Parameters¶
trace_store:
Any object with a list_traces(limit=...) method returning
List[Trace] (typically a :class:TraceStore).
min_quality:
Minimum feedback score for a trace to be included.
min_samples_per_class:
Minimum number of samples a query class must have to appear in
routing/agent-config results.
extract_agent_config_pairs(*, agent=None)
¶
Return per-query-class agent and tool recommendations.
Returns a dict mapping query class to:
best_agent— agent with the highest average feedback.best_tools— most frequently used tools by the best agent.avg_feedback— average feedback across all agents for the class.sample_count— total number of qualifying traces in the class.
extract_routing_pairs(*, agent=None)
¶
Return per-query-class routing recommendations.
Returns a dict mapping query class to:
best_model— model with highest average feedback for the class.avg_feedback— average feedback across all models for the class.sample_count— total number of qualifying traces in the class.all_models— dict of{model: {"avg_feedback": float, "count": int}}.
extract_sft_pairs(*, agent=None)
¶
Return SFT training pairs from high-quality traces.
Each entry is a dict with keys: input, output,
query_class, model, feedback.
Duplicate (input, output) pairs are collapsed; the first
occurrence is kept.
build_routing_context(query, *, urgency=0.5, model=None)
¶
Populate a RoutingContext from a raw query string.
When model is provided, the suggested token budget is adjusted for thinking models that need extra headroom.
ensure_registered()
¶
Ensure all learning policies are registered in RouterPolicyRegistry.
score_complexity(query)
¶
Score a query's complexity from 0.0 (trivial) to 1.0 (very complex).
The score is a weighted combination of independent signals, each contributing a fraction between 0 and 1. Weights reflect how much each signal correlates with the amount of reasoning and output tokens a model will need.