types

types

Core data types for the evaluation framework.

Classes

EvalRecord dataclass

EvalRecord(
    record_id: str,
    problem: str,
    reference: str,
    category: str,
    subject: str = "",
    metadata: Dict[str, Any] = dict(),
)

A single evaluation sample.

EvalResult dataclass

EvalResult(
    record_id: str,
    model_answer: str,
    is_correct: Optional[bool] = None,
    score: Optional[float] = None,
    latency_seconds: float = 0.0,
    prompt_tokens: int = 0,
    completion_tokens: int = 0,
    cost_usd: float = 0.0,
    error: Optional[str] = None,
    scoring_metadata: Dict[str, Any] = dict(),
    ttft: float = 0.0,
    energy_joules: float = 0.0,
    power_watts: float = 0.0,
    gpu_utilization_pct: float = 0.0,
    throughput_tok_per_sec: float = 0.0,
    mfu_pct: float = 0.0,
    mbu_pct: float = 0.0,
    ipw: float = 0.0,
    ipj: float = 0.0,
    energy_per_output_token_joules: float = 0.0,
    throughput_per_watt: float = 0.0,
    mean_itl_ms: float = 0.0,
    estimated_flops: float = 0.0,
    trace_steps: int = 0,
    trace_energy_joules: float = 0.0,
    trace_data: Optional[Dict[str, Any]] = None,
    framework: str = "diapason",
    framework_commit: str = "",
    tool_calls: int = 0,
    turn_count: int = 0,
)

Result of evaluating a single sample.

RunConfig dataclass

RunConfig(
    benchmark: str,
    backend: str,
    model: str,
    max_samples: Optional[int] = None,
    max_workers: int = 4,
    temperature: float = 0.0,
    max_tokens: int = 2048,
    judge_model: str = "gpt-5-mini-2025-08-07",
    judge_engine: str = "cloud",
    engine_key: Optional[str] = None,
    agent_name: Optional[str] = None,
    tools: List[str] = list(),
    output_path: Optional[str] = None,
    seed: int = 42,
    dataset_split: Optional[str] = None,
    telemetry: bool = False,
    gpu_metrics: bool = False,
    metadata: Dict[str, Any] = dict(),
    warmup_samples: int = 0,
    wandb_project: str = "",
    wandb_entity: str = "",
    wandb_tags: str = "",
    wandb_group: str = "",
    sheets_spreadsheet_id: str = "",
    sheets_worksheet: str = "Results",
    sheets_credentials_path: str = "",
    system_prompt: str = "",
    episode_mode: bool = False,
    dataset_subset: Optional[str] = None,
    max_turns: Optional[int] = None,
    base_url: Optional[str] = None,
    api_key: Optional[str] = None,
    record_ids: Optional[List[str]] = None,
    global_agent_timeout_sec: Optional[float] = None,
    global_timeout_multiplier: Optional[float] = None,
)

Configuration for an evaluation run.

MetricStats dataclass

MetricStats(
    mean: float = 0.0,
    median: float = 0.0,
    min: float = 0.0,
    max: float = 0.0,
    std: float = 0.0,
    p90: float = 0.0,
    p95: float = 0.0,
    p99: float = 0.0,
)

Descriptive statistics for a single metric across samples.

RunSummary dataclass

RunSummary(
    benchmark: str,
    category: str,
    backend: str,
    model: str,
    total_samples: int,
    scored_samples: int,
    correct: int,
    accuracy: float,
    errors: int,
    mean_latency_seconds: float,
    total_cost_usd: float,
    per_subject: Dict[str, Dict[str, float]] = dict(),
    started_at: float = 0.0,
    ended_at: float = 0.0,
    accuracy_stats: Optional[MetricStats] = None,
    latency_stats: Optional[MetricStats] = None,
    ttft_stats: Optional[MetricStats] = None,
    energy_stats: Optional[MetricStats] = None,
    power_stats: Optional[MetricStats] = None,
    gpu_utilization_stats: Optional[MetricStats] = None,
    throughput_stats: Optional[MetricStats] = None,
    mfu_stats: Optional[MetricStats] = None,
    mbu_stats: Optional[MetricStats] = None,
    ipw_stats: Optional[MetricStats] = None,
    ipj_stats: Optional[MetricStats] = None,
    energy_per_output_token_stats: Optional[
        MetricStats
    ] = None,
    throughput_per_watt_stats: Optional[MetricStats] = None,
    itl_stats: Optional[MetricStats] = None,
    input_token_stats: Optional[MetricStats] = None,
    output_token_stats: Optional[MetricStats] = None,
    total_energy_joules: float = 0.0,
    total_estimated_flops: float = 0.0,
    flops_stats: Optional[MetricStats] = None,
    warmup_samples_excluded: int = 0,
    steady_state_reached: bool = False,
    energy_method: str = "",
    avg_power_watts: float = 0.0,
    total_input_tokens: int = 0,
    total_output_tokens: int = 0,
    trace_step_type_stats: Dict[
        str, Dict[str, float]
    ] = dict(),
    efficiency: Optional[Dict[str, Any]] = None,
    normalized_statistics: Optional[Dict[str, Any]] = None,
    normalized_efficiency: Optional[Dict[str, Any]] = None,
    mean_continuous_score: Optional[float] = None,
    median_continuous_score: Optional[float] = None,
    pct_above_0_5: Optional[float] = None,
    pct_above_0_7: Optional[float] = None,
    pct_above_0_8: Optional[float] = None,
    pct_above_0_9: Optional[float] = None,
    _output_path: Optional[Path] = None,
    _traces_dir: Optional[Path] = None,
)

Summary statistics for a completed evaluation run.

MetaConfig dataclass

MetaConfig(name: str = '', description: str = '')

Suite-level metadata.

DefaultsConfig dataclass

DefaultsConfig(
    temperature: float = 0.0, max_tokens: int = 2048
)

Default generation parameters applied to all runs.

JudgeConfig dataclass

JudgeConfig(
    model: str = "gpt-5-mini-2025-08-07",
    engine: Optional[str] = None,
    provider: Optional[str] = None,
    temperature: float = 0.0,
    max_tokens: int = 1024,
)

Configuration for the LLM judge.

ExecutionConfig dataclass

ExecutionConfig(
    max_workers: int = 4,
    output_dir: str = "results/",
    seed: int = 42,
    telemetry: bool = False,
    gpu_metrics: bool = False,
    warmup_samples: int = 0,
    energy_vendor: str = "",
    wandb_project: str = "",
    wandb_entity: str = "",
    wandb_tags: str = "",
    wandb_group: str = "",
    sheets_spreadsheet_id: str = "",
    sheets_worksheet: str = "Results",
    sheets_credentials_path: str = "",
    max_turns: Optional[int] = None,
    global_agent_timeout_sec: Optional[float] = None,
    global_timeout_multiplier: Optional[float] = None,
)

Execution-level settings for the eval run.

ModelConfig dataclass

ModelConfig(
    name: str = "",
    engine: Optional[str] = None,
    provider: Optional[str] = None,
    temperature: Optional[float] = None,
    max_tokens: Optional[int] = None,
    param_count_b: float = 0.0,
    active_params_b: Optional[float] = None,
    gpu_peak_tflops: float = 0.0,
    gpu_peak_bandwidth_gb_s: float = 0.0,
    num_gpus: int = 1,
)

Configuration for a single model in the eval suite.

BenchmarkConfig dataclass

BenchmarkConfig(
    name: str = "",
    backend: str = "diapason-direct",
    max_samples: Optional[int] = None,
    split: Optional[str] = None,
    agent: Optional[str] = None,
    tools: List[str] = list(),
    judge_model: Optional[str] = None,
    temperature: Optional[float] = None,
    max_tokens: Optional[int] = None,
    subset: Optional[str] = None,
    record_ids: Optional[List[str]] = None,
    global_agent_timeout_sec: Optional[float] = None,
    global_timeout_multiplier: Optional[float] = None,
)

Configuration for a single benchmark in the eval suite.

EvalSuiteConfig dataclass

EvalSuiteConfig(
    meta: MetaConfig = MetaConfig(),
    defaults: DefaultsConfig = DefaultsConfig(),
    judge: JudgeConfig = JudgeConfig(),
    run: ExecutionConfig = ExecutionConfig(),
    models: List[ModelConfig] = list(),
    benchmarks: List[BenchmarkConfig] = list(),
    backend_external_base_url: Optional[str] = None,
    backend_external_api_key: Optional[str] = None,
)

Top-level configuration for an eval suite (models x benchmarks).