workarena
diapason.evals.datasets.workarena
¶
WorkArena++ enterprise workflow benchmark on ServiceNow.
Faithful integration of the original browsergym-workarena package. Tasks are Python classes that run against a live ServiceNow instance via BrowserGym / Playwright — NOT a static JSON dataset.
L1 = 33 atomic tasks (ICML 2024) L2/L3 = 682 composite tasks (NeurIPS 2024)
Source: https://github.com/ServiceNow/WorkArena Requires: pip install browsergym-workarena playwright==1.44.0
WorkArenaDataset
¶
Bases: DatasetProvider
WorkArena++ benchmark using the native browsergym-workarena package.
Tasks are enumerated from the installed browsergym-workarena
package exactly as in the original benchmark. Each task class is
instantiated with a seed by BrowserGym at evaluation time. Scoring
uses the task's native validate() method against the live
ServiceNow instance — no LLM judge or text matching.