webchorearena_scorer
diapason.evals.scorers.webchorearena_scorer
¶
Scorer for WebChoreArena web chore tasks.
Uses the environment-validated scoring pattern (same as WorkArenaScorer):
the WebChoreArenaTaskEnv runs the original WebArena evaluation harness
(StringEvaluator × URLEvaluator × HTMLContentEvaluator, multiplicative)
and populates record.metadata["is_resolved"] and record.metadata["reward"].
This scorer reads those fields.
The evaluation harness inside the task env faithfully mirrors the original: - StringEvaluator: exact_match, must_include (with |OR| support), fuzzy_match (LLM-judged via GPT-4o), ua_match (unachievable task detection) - URLEvaluator: checks browser's current page URL against reference URLs - HTMLContentEvaluator: navigates to URLs, runs JS locators on DOM, checks element content against expected values