Skip to content

test_runner

diapason.evals.tests.test_runner

Tests for the EvalRunner.

TestEvalRunner

test_ipw_ipj_computation(tmp_path)

IPW and IPJ should be computed for correct samples.

test_ipw_ipj_zero_for_incorrect(tmp_path)

IPW and IPJ should be 0 for incorrect samples.

test_mfu_mbu_with_metadata(tmp_path)

MFU/MBU should be computed when model metadata is provided.

test_summary_json_includes_metric_stats(tmp_path)

Summary JSON file should serialize MetricStats fields.

test_summary_metric_stats(tmp_path)

Summary should include MetricStats for telemetry fields.

test_telemetry_fields_in_jsonl(tmp_path)

Verify telemetry fields are written to JSONL output.

TestRunnerTokenStats

test_summary_has_avg_power(tmp_path)

RunSummary should include avg_power_watts.

test_summary_has_total_input_output_tokens(tmp_path)

RunSummary should include total token counts.