Skip to content

Index

diapason.learning.intelligence.orchestrator

Orchestrator training infrastructure — SFT and GRPO pipelines.

Provides structured-mode training for the OrchestratorAgent with:

  • Episode types: Action, Observation, Episode, EpisodeState
  • Reward: Multi-objective reward balancing accuracy, cost, energy, latency, power
  • Prompt registry: Canonical system prompts for structured mode
  • Policy model: HuggingFace LM wrapper for action prediction
  • Environment: RL environment using Diapason ToolExecutor
  • SFT trainer: Supervised fine-tuning on successful trajectories
  • GRPO trainer: Group Relative Policy Optimization

Importing this module triggers registration of orchestrator_sft and orchestrator_grpo in :class:~diapason.core.registry.LearningRegistry.

AdaptiveRewardWeights

Adaptive reward weights that shift during training.

Early training focuses on accuracy (higher alpha). Late training optimises efficiency (higher cost/energy/power weights).

get_weights(current_step)

Get weights for current_step via linear interpolation.

Episode dataclass

Complete RL episode with aggregate metrics.

correct = False class-attribute instance-attribute

Whether final answer matches ground truth.

final_answer = '' class-attribute instance-attribute

Final answer produced by orchestrator.

ground_truth = '' class-attribute instance-attribute

Ground truth answer.

initial_prompt instance-attribute

Initial question/task.

steps = field(default_factory=list) class-attribute instance-attribute

Sequence of (action, observation) pairs.

task_id instance-attribute

Unique task identifier.

add_step(action, observation)

Add a step to the episode and update aggregate metrics.

compute_ipj()

Compute Intelligence Per Joule (IPJ).

Returns:

Type Description
float

IPJ score (higher is better). 0.0 if energy is zero or

float

the answer is incorrect.

num_turns()

Return number of turns in episode.

to_dict()

Convert episode to dictionary for serialization.

EpisodeState dataclass

Mutable state during episode execution.

final_answer = None class-attribute instance-attribute

Final answer (set when is_final_answer action is taken).

history = field(default_factory=list) class-attribute instance-attribute

History of (action, observation) pairs.

initial_prompt instance-attribute

Initial task/question.

add_turn(action, observation)

Add a turn to the episode history.

num_turns()

Return number of turns so far.

to_episode(task_id, ground_truth, correct)

Convert state to Episode for reward computation.

EpisodeStep dataclass

Single step in an episode.

action instance-attribute

Action taken.

observation instance-attribute

Result of action.

turn instance-attribute

Step number (0-indexed).

MultiObjectiveReward

Multi-objective reward combining accuracy, cost, energy, latency, power.

Formula::

reward = alpha * accuracy
         - beta_cost  * (cost   / cost_scale)
         - beta_energy * (energy / energy_scale)
         - gamma_latency * (latency / latency_scale)
         - gamma_power   * (power   / power_scale)

compute(episode)

Compute scalar reward for an episode.

compute_batch(episodes)

Compute rewards for a batch of episodes.

compute_with_breakdown(episode)

Compute reward with detailed per-component breakdown.

Normalizers dataclass

Normalization constants for reward scaling.

These are typical values used to scale metrics to similar ranges. Tune based on your specific tools and tasks.

OrchestratorAction dataclass

Orchestrator action: thought + tool selection + tool input.

is_final_answer = False class-attribute instance-attribute

Whether this action provides the final answer.

thought instance-attribute

Reasoning about what to do next.

tool_input instance-attribute

Input/prompt to send to the tool.

tool_name instance-attribute

Selected tool name (e.g., 'calculator', 'think').

OrchestratorEnvironment

RL environment that executes tools via Diapason ToolExecutor.

Parameters

tools: List of :class:BaseTool instances available to the agent. max_turns: Maximum number of turns per episode.

get_available_tools()

Return names of all available tools.

is_done(state)

Check if the episode is complete.

reset(task)

Reset the environment for a new episode.

Parameters:

Name Type Description Default
task str

The initial task/question.

required

Returns:

Type Description
EpisodeState

A fresh :class:EpisodeState.

step(state, action)

Execute one step: dispatch the tool and observe the result.

Raises:

Type Description
ValueError

If the tool is not available or max turns exceeded.

OrchestratorGRPOConfig dataclass

Configuration for orchestrator GRPO training.

OrchestratorGRPOTrainer

GRPO trainer for orchestrator policy.

torch must be installed to call :meth:train.

train()

Run the GRPO training loop.

OrchestratorObservation dataclass

Result from executing an action, with flat telemetry fields.

content instance-attribute

Tool response content.

cost_usd = 0.0 class-attribute instance-attribute

Cost in USD.

energy_joules = 0.0 class-attribute instance-attribute

Energy consumed in joules.

latency_seconds = 0.0 class-attribute instance-attribute

Latency in seconds.

power_watts = 0.0 class-attribute instance-attribute

Power usage in watts.

tokens = 0 class-attribute instance-attribute

Tokens consumed.

OrchestratorPolicyModel

Wrapper around a causal LM for orchestrator policy prediction.

Input format (prompt)::

Task: {initial_prompt}

Available tools: calculator, think, ...

History:
Turn 1:
  Thought: ...
  Tool: ...
  Observation: ...

What should you do next?
Format your response as:
THOUGHT: [your reasoning]
TOOL: [tool_name]
INPUT: [input for tool]

Output format (from model)::

THOUGHT: [reasoning]
TOOL: [tool_name]
INPUT: [input]
--- or ---
FINAL_ANSWER: [answer]

from_checkpoint(checkpoint_path, **kwargs) classmethod

Load from a previously saved checkpoint directory.

from_pretrained(model_name='Qwen/Qwen3-1.7B', gradient_checkpointing=False, load_in_8bit=False, device=None, **kwargs) classmethod

Load model from a HuggingFace checkpoint.

Raises ImportError if transformers is not installed.

predict_action(state, available_tools)

Predict the next action given current state.

save(path)

Save model and tokenizer to path.

OrchestratorSFTConfig dataclass

Configuration for orchestrator SFT training.

OrchestratorSFTDataset

Dataset for SFT training from generated trace JSONL files.

OrchestratorSFTTrainer

SFT trainer for orchestrator policy.

Performs standard next-token cross-entropy loss on successful trajectories. torch must be installed to call :meth:train.

train()

Run the SFT training loop.

PolicyOutput dataclass

Output from policy model prediction.

confidence = 1.0 class-attribute instance-attribute

Confidence score (if available).

is_final_answer = False class-attribute instance-attribute

Whether this provides the final answer.

raw_text = '' class-attribute instance-attribute

Raw model output.

thought instance-attribute

Reasoning about what to do.

tool_input instance-attribute

Input for the tool.

tool_name instance-attribute

Selected tool.

RewardWeights dataclass

Weights for multi-objective reward function.

Each metric has its own coefficient: - alpha: Accuracy (correctness of answer) - beta_cost: API/cloud cost in USD - beta_energy: Energy consumption in joules - gamma_latency: Response time in seconds - gamma_power: Peak power usage in watts

__post_init__()

Validate weights sum to 1.0.

build_system_prompt(tool_names=None, *, tools=None)

Build the complete system prompt for the given tools.

Parameters:

Name Type Description Default
tool_names Optional[List[str]]

Tool names to include. If None, uses all tools from :data:TOOL_DESCRIPTIONS. This path is kept for backward compatibility with training pipelines.

None
tools Optional[List['BaseTool']]

Optional list of BaseTool instances. When provided, rich descriptions are auto-generated from ToolSpec, replacing the hardcoded :data:TOOL_DESCRIPTIONS lookup. Unknown / MCP tools get full descriptions instead of "Tool: {name}".

None

Returns:

Type Description
str

Complete system prompt string.

extract_answer(text)

Extract the core answer from a potentially verbose response.

Handles patterns like: - "The answer is 4" - "Result: 4.0" - "4" (unchanged) - "Therefore, the answer is approximately 4"

grade_answer(predicted, expected, tolerance=1e-06)

Grade an answer against expected, with smart matching.

Handles: - Exact string match (case-insensitive) - Numeric comparison with tolerance - Answer extraction from verbose responses

Parameters:

Name Type Description Default
predicted str

The model's answer.

required
expected str

Ground truth answer.

required
tolerance float

Tolerance for numeric comparisons.

1e-06

Returns:

Type Description
bool

True if answer is correct.

normalize_number(s)

Try to parse a string as a number.

Returns None if not a valid number.