Skip to content

Index

diapason.learning.optimize.personal

Personal benchmark system -- synthesize benchmarks from interaction traces.

PersonalBenchmark dataclass

A synthesized benchmark from user interaction traces.

PersonalBenchmarkDataset

Bases: DatasetProvider

Wraps a PersonalBenchmark as a DatasetProvider for EvalRunner.

iter_records()

Iterate over loaded records.

load(*, max_samples=None, split=None, seed=None)

Convert :class:PersonalBenchmarkSample instances to :class:EvalRecord.

size()

Return the number of loaded records.

PersonalBenchmarkSample dataclass

A single sample in a personal benchmark.

PersonalBenchmarkScorer

Bases: LLMJudgeScorer

Judges a candidate response against the best-known response from traces.

score(record, model_answer)

Compare model_answer against record.reference using the judge LLM.

Returns (is_correct, metadata) where is_correct indicates whether the candidate answer is at least as good as the reference.

PersonalBenchmarkSynthesizer

Mines interaction traces into a reusable personal benchmark.

synthesize(workflow_id='default', min_feedback=0.7, max_samples=100)

Build a personal benchmark from high-quality traces.

  1. Query traces that have feedback >= min_feedback.
  2. Group by query class (agent + first 50 chars of query).
  3. For each class, pick the trace with the highest feedback as reference.
  4. Return a :class:PersonalBenchmark capped at max_samples.