Skip to content

webchorearena_env

diapason.evals.execution.webchorearena_env

WebChoreArena task environment — Playwright-based browser interaction.

Wraps the WebArena browser environment to provide per-task setup, observation access, action stepping, and evaluation using the original WebArena evaluation harness (StringEvaluator, URLEvaluator, HTMLContentEvaluator combined multiplicatively).

Requires
  • playwright (pip install playwright && playwright install)
  • Running WebArena standalone sites (Shopping, Reddit, GitLab, etc.)
  • Environment variables: SHOPPING, SHOPPING_ADMIN, REDDIT, GITLAB, MAP, WIKIPEDIA

WebChoreArenaTaskEnv

Per-task browser environment for WebChoreArena.

Context manager that creates a Playwright browser, navigates to the task's start URL, and exposes observation/action/evaluate methods.

Evaluation uses the original WebArena evaluator harness with multiplicative combination of StringEvaluator, URLEvaluator, and HTMLContentEvaluator.

run_agent_loop(generate_fn, max_steps=None)

Drive the browser env in a step loop using generate_fn for LLM calls.

Returns the agent's final answer text.

run_tests()

Run the WebArena evaluation harness and populate metadata.