Skip to content

lifelong_agent_scorer

diapason.evals.scorers.lifelong_agent_scorer

Scorer for LifelongAgentBench.

Reproduces the original evaluation methodology from

https://github.com/caixd-220529/LifelongAgentBench

When used with interactive environments (episode_mode=True), the TaskEnvironment handles scoring directly. This scorer serves as:

  1. Fallback for non-interactive (single-shot) evaluation — with loud warnings that results are degraded and not faithful to the original.
  2. Helper library for shared scoring functions used by both the environments and this scorer.

Three subset scoring strategies matching the original:

  1. db_bench — Two modes matching the original Task._complete():
  2. direct (SELECT): Execute agent SQL, compare tuples with numeric tolerance (rel_tol=1e-6).
  3. md5 (INSERT/UPDATE/DELETE): Execute SQL, compare full table state.

  4. knowledge_graph — Exact-set match + F1 score on answer entities, matching the original's calculate_metric().

  5. os_interaction — Docker evaluation with exit_code == 0.

IMPORTANT: Single-shot scoring is DEGRADED and will always emit warnings. The original benchmark is multi-turn interactive. Use episode_mode=True with the diapason-agent backend for faithful evaluation.

LifelongAgentScorer

Bases: Scorer

Scorer for LifelongAgentBench.

When used with interactive environments (episode_mode), the environment handles scoring and this scorer is bypassed. For single-shot mode, this scorer provides degraded fallback scoring with clear warnings.

Constructor accepts (judge_backend, judge_model) for CLI compatibility but does not use them — all scoring is deterministic.

build_db(table_info)

Build an in-memory SQLite DB from table_info.

Note: The original uses MySQL Docker containers. SQLite is used as a portable fallback. MySQL-specific features (e.g. backtick quoting, GROUP_CONCAT, MD5()) will not be available.