Skip to content

cli

diapason.evals.cli

CLI for the Diapason evaluation framework.

list_cmd()

List available benchmarks and backends.

main()

Diapason Evaluation Framework.

reparse_judge(jsonl_path, out_path, in_place, summary_out)

Re-parse stored judge output and recover records that failed.

Reads each record raw_judge_output, runs it through the (fixed) parser, and updates records whose old score was 0/None when a new score is recoverable. Writes a summary.json with the recomputed accuracy + the continuous-score fields, and prints a diff.

run(ctx, config_path, benchmark, backend, base_url, api_key, model, engine_key, agent_name, tools, max_samples, max_workers, judge_model, output_path, seed, dataset_split, temperature, max_tokens, telemetry, gpu_metrics, compact, trace_detail, wandb_project, wandb_entity, wandb_tags, wandb_group, sheets_spreadsheet_id, sheets_worksheet, sheets_credentials_path, model_filter, judge_engine, agentic, episode_mode, concurrency, query_timeout, verbose)

Run a single benchmark evaluation, or a full suite from a TOML config.

run_all(model, engine_key, max_samples, max_workers, judge_model, output_dir, seed, verbose)

Run all benchmarks.

summarize(jsonl_path)

Summarize results from a JSONL output file.