Skip to content

judge

diapason.learning.optimize.feedback.judge

TraceJudge -- LLM-as-judge scoring for agent traces.

TraceJudge

LLM-as-judge for scoring traces when no ground truth exists.

Given a :class:Trace, the judge constructs a prompt showing the query, agent steps, and final result, then asks an LLM to rate the quality on a 0-1 scale.

batch_evaluate(traces)

Evaluate multiple traces sequentially.

Returns a list of (score, feedback) tuples, one per trace.

score_trace(trace)

Score a single trace.

Returns:

Type Description
float

(score, feedback) where score is in [0, 1] and

str

feedback is the judge's textual reasoning.