test_runner
diapason.evals.tests.test_runner
¶
Tests for the EvalRunner.
TestEvalRunner
¶
test_ipw_ipj_computation(tmp_path)
¶
IPW and IPJ should be computed for correct samples.
test_ipw_ipj_zero_for_incorrect(tmp_path)
¶
IPW and IPJ should be 0 for incorrect samples.
test_mfu_mbu_with_metadata(tmp_path)
¶
MFU/MBU should be computed when model metadata is provided.
test_summary_json_includes_metric_stats(tmp_path)
¶
Summary JSON file should serialize MetricStats fields.
test_summary_metric_stats(tmp_path)
¶
Summary should include MetricStats for telemetry fields.
test_telemetry_fields_in_jsonl(tmp_path)
¶
Verify telemetry fields are written to JSONL output.