Index
diapason.learning.optimize.feedback
¶
Feedback subsystem: LLM-as-judge scoring and signal aggregation.
FeedbackCollector
¶
Collects feedback signals: explicit user scores + LLM judge evaluations.
Signals are stored in-memory as dicts with at least trace_id,
score, source, and timestamp keys.
evaluate_traces(traces, judge)
¶
Score traces via the LLM judge and record the results.
Returns the list of newly created feedback records.
get_records(trace_id=None)
¶
Return stored records, optionally filtered by trace_id.
record_explicit(trace_id, score, source='api')
¶
Record an explicit numeric score (0-1) for a trace.
record_thumbs(trace_id, thumbs_up)
¶
Record a thumbs-up / thumbs-down signal (converted to 1.0/0.0).
stats()
¶
Return aggregate statistics over all recorded feedback.
Returns a dict with count, mean_score, and a simple
distribution bucket (low / medium / high).
TraceJudge
¶
LLM-as-judge for scoring traces when no ground truth exists.
Given a :class:Trace, the judge constructs a prompt showing the
query, agent steps, and final result, then asks an LLM to rate the
quality on a 0-1 scale.