Index
diapason.learning.optimize.personal
¶
Personal benchmark system -- synthesize benchmarks from interaction traces.
PersonalBenchmark
dataclass
¶
A synthesized benchmark from user interaction traces.
PersonalBenchmarkDataset
¶
Bases: DatasetProvider
Wraps a PersonalBenchmark as a DatasetProvider for EvalRunner.
PersonalBenchmarkSample
dataclass
¶
A single sample in a personal benchmark.
PersonalBenchmarkScorer
¶
Bases: LLMJudgeScorer
Judges a candidate response against the best-known response from traces.
score(record, model_answer)
¶
Compare model_answer against record.reference using the judge LLM.
Returns (is_correct, metadata) where is_correct indicates whether
the candidate answer is at least as good as the reference.
PersonalBenchmarkSynthesizer
¶
Mines interaction traces into a reusable personal benchmark.
synthesize(workflow_id='default', min_feedback=0.7, max_samples=100)
¶
Build a personal benchmark from high-quality traces.
- Query traces that have feedback >= min_feedback.
- Group by query class (agent + first 50 chars of query).
- For each class, pick the trace with the highest feedback as reference.
- Return a :class:
PersonalBenchmarkcapped at max_samples.