Skip to content

Index

diapason.learning.optimize.feedback

Feedback subsystem: LLM-as-judge scoring and signal aggregation.

FeedbackCollector

Collects feedback signals: explicit user scores + LLM judge evaluations.

Signals are stored in-memory as dicts with at least trace_id, score, source, and timestamp keys.

evaluate_traces(traces, judge)

Score traces via the LLM judge and record the results.

Returns the list of newly created feedback records.

get_records(trace_id=None)

Return stored records, optionally filtered by trace_id.

record_explicit(trace_id, score, source='api')

Record an explicit numeric score (0-1) for a trace.

record_thumbs(trace_id, thumbs_up)

Record a thumbs-up / thumbs-down signal (converted to 1.0/0.0).

stats()

Return aggregate statistics over all recorded feedback.

Returns a dict with count, mean_score, and a simple distribution bucket (low / medium / high).

TraceJudge

LLM-as-judge for scoring traces when no ground truth exists.

Given a :class:Trace, the judge constructs a prompt showing the query, agent steps, and final result, then asks an LLM to rate the quality on a 0-1 scale.

batch_evaluate(traces)

Evaluate multiple traces sequentially.

Returns a list of (score, feedback) tuples, one per trace.

score_trace(trace)

Score a single trace.

Returns:

Type Description
float

(score, feedback) where score is in [0, 1] and

str

feedback is the judge's textual reasoning.