webchorearena
diapason.evals.datasets.webchorearena
¶
WebChoreArena: Realistic tedious web browsing tasks.
Evaluates web agents on 532 tasks across Shopping, Shopping Admin, Reddit, GitLab, and Cross-site environments. Tests massive memory, calculation, and long-term memory capabilities.
Requires a running WebArena standalone environment (Shopping/Magento, Reddit/Postmill, GitLab, Shopping Admin). Tasks are per-site JSON configs cloned from the original GitHub repository.
Source: https://github.com/WebChoreArena/WebChoreArena
WebChoreArenaDataset
¶
Bases: DatasetProvider
WebChoreArena benchmark — interactive browser-based web tasks.
Tasks are enumerated from the original GitHub repository's
config_files/ JSON files. Each task requires a live WebArena
standalone environment and Playwright for evaluation.
create_task_env(record)
¶
Return a WebChoreArenaTaskEnv for the given record.