Index
diapason.learning.intelligence.orchestrator
¶
Orchestrator training infrastructure — SFT and GRPO pipelines.
Provides structured-mode training for the OrchestratorAgent with:
- Episode types: Action, Observation, Episode, EpisodeState
- Reward: Multi-objective reward balancing accuracy, cost, energy, latency, power
- Prompt registry: Canonical system prompts for structured mode
- Policy model: HuggingFace LM wrapper for action prediction
- Environment: RL environment using Diapason ToolExecutor
- SFT trainer: Supervised fine-tuning on successful trajectories
- GRPO trainer: Group Relative Policy Optimization
Importing this module triggers registration of orchestrator_sft and
orchestrator_grpo in :class:~diapason.core.registry.LearningRegistry.
AdaptiveRewardWeights
¶
Adaptive reward weights that shift during training.
Early training focuses on accuracy (higher alpha). Late training optimises efficiency (higher cost/energy/power weights).
get_weights(current_step)
¶
Get weights for current_step via linear interpolation.
Episode
dataclass
¶
Complete RL episode with aggregate metrics.
correct = False
class-attribute
instance-attribute
¶
Whether final answer matches ground truth.
final_answer = ''
class-attribute
instance-attribute
¶
Final answer produced by orchestrator.
ground_truth = ''
class-attribute
instance-attribute
¶
Ground truth answer.
initial_prompt
instance-attribute
¶
Initial question/task.
steps = field(default_factory=list)
class-attribute
instance-attribute
¶
Sequence of (action, observation) pairs.
task_id
instance-attribute
¶
Unique task identifier.
add_step(action, observation)
¶
Add a step to the episode and update aggregate metrics.
compute_ipj()
¶
Compute Intelligence Per Joule (IPJ).
Returns:
| Type | Description |
|---|---|
float
|
IPJ score (higher is better). 0.0 if energy is zero or |
float
|
the answer is incorrect. |
num_turns()
¶
Return number of turns in episode.
to_dict()
¶
Convert episode to dictionary for serialization.
EpisodeState
dataclass
¶
Mutable state during episode execution.
final_answer = None
class-attribute
instance-attribute
¶
Final answer (set when is_final_answer action is taken).
history = field(default_factory=list)
class-attribute
instance-attribute
¶
History of (action, observation) pairs.
initial_prompt
instance-attribute
¶
Initial task/question.
add_turn(action, observation)
¶
Add a turn to the episode history.
num_turns()
¶
Return number of turns so far.
to_episode(task_id, ground_truth, correct)
¶
Convert state to Episode for reward computation.
EpisodeStep
dataclass
¶
MultiObjectiveReward
¶
Multi-objective reward combining accuracy, cost, energy, latency, power.
Formula::
reward = alpha * accuracy
- beta_cost * (cost / cost_scale)
- beta_energy * (energy / energy_scale)
- gamma_latency * (latency / latency_scale)
- gamma_power * (power / power_scale)
Normalizers
dataclass
¶
Normalization constants for reward scaling.
These are typical values used to scale metrics to similar ranges. Tune based on your specific tools and tasks.
OrchestratorAction
dataclass
¶
Orchestrator action: thought + tool selection + tool input.
is_final_answer = False
class-attribute
instance-attribute
¶
Whether this action provides the final answer.
thought
instance-attribute
¶
Reasoning about what to do next.
tool_input
instance-attribute
¶
Input/prompt to send to the tool.
tool_name
instance-attribute
¶
Selected tool name (e.g., 'calculator', 'think').
OrchestratorEnvironment
¶
RL environment that executes tools via Diapason ToolExecutor.
Parameters¶
tools:
List of :class:BaseTool instances available to the agent.
max_turns:
Maximum number of turns per episode.
get_available_tools()
¶
Return names of all available tools.
is_done(state)
¶
Check if the episode is complete.
reset(task)
¶
Reset the environment for a new episode.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
task
|
str
|
The initial task/question. |
required |
Returns:
| Type | Description |
|---|---|
EpisodeState
|
A fresh :class: |
step(state, action)
¶
Execute one step: dispatch the tool and observe the result.
Raises:
| Type | Description |
|---|---|
ValueError
|
If the tool is not available or max turns exceeded. |
OrchestratorGRPOConfig
dataclass
¶
Configuration for orchestrator GRPO training.
OrchestratorGRPOTrainer
¶
GRPO trainer for orchestrator policy.
torch must be installed to call :meth:train.
train()
¶
Run the GRPO training loop.
OrchestratorObservation
dataclass
¶
Result from executing an action, with flat telemetry fields.
content
instance-attribute
¶
Tool response content.
cost_usd = 0.0
class-attribute
instance-attribute
¶
Cost in USD.
energy_joules = 0.0
class-attribute
instance-attribute
¶
Energy consumed in joules.
latency_seconds = 0.0
class-attribute
instance-attribute
¶
Latency in seconds.
power_watts = 0.0
class-attribute
instance-attribute
¶
Power usage in watts.
tokens = 0
class-attribute
instance-attribute
¶
Tokens consumed.
OrchestratorPolicyModel
¶
Wrapper around a causal LM for orchestrator policy prediction.
Input format (prompt)::
Task: {initial_prompt}
Available tools: calculator, think, ...
History:
Turn 1:
Thought: ...
Tool: ...
Observation: ...
What should you do next?
Format your response as:
THOUGHT: [your reasoning]
TOOL: [tool_name]
INPUT: [input for tool]
Output format (from model)::
THOUGHT: [reasoning]
TOOL: [tool_name]
INPUT: [input]
--- or ---
FINAL_ANSWER: [answer]
from_checkpoint(checkpoint_path, **kwargs)
classmethod
¶
Load from a previously saved checkpoint directory.
from_pretrained(model_name='Qwen/Qwen3-1.7B', gradient_checkpointing=False, load_in_8bit=False, device=None, **kwargs)
classmethod
¶
Load model from a HuggingFace checkpoint.
Raises ImportError if transformers is not installed.
predict_action(state, available_tools)
¶
Predict the next action given current state.
save(path)
¶
Save model and tokenizer to path.
OrchestratorSFTConfig
dataclass
¶
Configuration for orchestrator SFT training.
OrchestratorSFTDataset
¶
Dataset for SFT training from generated trace JSONL files.
OrchestratorSFTTrainer
¶
SFT trainer for orchestrator policy.
Performs standard next-token cross-entropy loss on successful
trajectories. torch must be installed to call :meth:train.
train()
¶
Run the SFT training loop.
PolicyOutput
dataclass
¶
Output from policy model prediction.
confidence = 1.0
class-attribute
instance-attribute
¶
Confidence score (if available).
is_final_answer = False
class-attribute
instance-attribute
¶
Whether this provides the final answer.
raw_text = ''
class-attribute
instance-attribute
¶
Raw model output.
thought
instance-attribute
¶
Reasoning about what to do.
tool_input
instance-attribute
¶
Input for the tool.
tool_name
instance-attribute
¶
Selected tool.
RewardWeights
dataclass
¶
Weights for multi-objective reward function.
Each metric has its own coefficient: - alpha: Accuracy (correctness of answer) - beta_cost: API/cloud cost in USD - beta_energy: Energy consumption in joules - gamma_latency: Response time in seconds - gamma_power: Peak power usage in watts
__post_init__()
¶
Validate weights sum to 1.0.
build_system_prompt(tool_names=None, *, tools=None)
¶
Build the complete system prompt for the given tools.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
tool_names
|
Optional[List[str]]
|
Tool names to include. If |
None
|
tools
|
Optional[List['BaseTool']]
|
Optional list of |
None
|
Returns:
| Type | Description |
|---|---|
str
|
Complete system prompt string. |
extract_answer(text)
¶
Extract the core answer from a potentially verbose response.
Handles patterns like: - "The answer is 4" - "Result: 4.0" - "4" (unchanged) - "Therefore, the answer is approximately 4"
grade_answer(predicted, expected, tolerance=1e-06)
¶
Grade an answer against expected, with smart matching.
Handles: - Exact string match (case-insensitive) - Numeric comparison with tolerance - Answer extraction from verbose responses
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
predicted
|
str
|
The model's answer. |
required |
expected
|
str
|
Ground truth answer. |
required |
tolerance
|
float
|
Tolerance for numeric comparisons. |
1e-06
|
Returns:
| Type | Description |
|---|---|
bool
|
True if answer is correct. |
normalize_number(s)
¶
Try to parse a string as a number.
Returns None if not a valid number.