Skip to content

liveresearch

diapason.evals.scorers.liveresearch

DeepResearchBench scorer — LLM-as-judge for deep research quality.

Evaluates research output quality across four dimensions from the DeepResearchBench rubric: comprehensiveness, insight, instruction_following, and readability. Uses LLM-as-judge with per-task criteria when available, falling back to a generic research quality rubric.

Reference: https://github.com/Ayanami0730/deep_research_bench Paper: https://arxiv.org/abs/2510.14240

LiveResearchBenchScorer

Bases: LLMJudgeScorer

LLM-as-judge scorer for DeepResearchBench deep research tasks.

Evaluates research reports across four dimensions: comprehensiveness, insight, instruction_following, readability. Uses task-specific criteria when available from the benchmark data.

rescore_from_metadata(scoring_metadata, dimension_weights=None)

Re-derive (is_correct, updated_metadata) from a stored raw_judge_output.

Returns None if no raw judge output is present or no scores can be parsed. Used by the evals reparse-judge CLI to fix records that failed under the old, stricter parser.