WebHarbor: Docking Real Websites for Evolving GUI Agent Environments

01 · 153 live websites

153 websites, each served locally and driven by an agent.

Each environment is a self-contained, locally served copy of a real website, with its own seeded database, task set and deterministic verifiers. It resets to the same state at the start of every episode. Pick a website to step through one of its tasks.

Showing 12 of 153 websites · port = 40000 + registration index

02 · Motivation

Live sites keep changing. Generated ones lack depth and fidelity.

GUI web agents need environments that look and behave like real websites, yet stay stable enough to evaluate and train on. Live websites change their content, block automation, and hide their most important workflows behind sign-in. Generated environments are controllable, but often lack functional depth, image-rich content, or login-based workflows.

38%

of Online-Mind2Web tasks were blocked by bot detection in our experiments.

22%

of WebVoyager tasks were outdated because the websites had changed.

31.8%

of WebHarbor tasks require signing in through the interface, a class of everyday workflows that live-web benchmarks rarely cover.

Environment	Curation	# Envs.	Offline	Images	Login
Evaluation
WebVoyager	Human + LLM	15	✕	✓	✕
WebArena	Human	6	✓	✓	✓
Online-Mind2Web	Human	136	✕	✓	✕
DeepShop	Agent	1	✕	✓	✕
Training
WebArena-Infinity	Agent	13	✓	✕	✕
AutoWebWorld	Agent	29	✓	△	△
CUA-Gym	Agent	98	✓	△	✕
⚓ WebHarbor	Human + Agent	153	✓	✓	✓

Released web-agent environments. Images = image-rich website content; △ = generated images or partial coverage; Login = the agent must authenticate through the interface. # Envs. follows the actual releases.

03 · Our approach

Tasks set the scope. Coding agents build. People audit.

We call it docking: replacing a website's live dependencies with local assets, seeded state, and controlled resets, while keeping its recognizable look and behavior. Environments keep evolving: new tasks expose missing functionality, and the same loop extends the site.

WebHarbor pipeline: live sites face CAPTCHAs, site changes, and access limits; contributors propose target tasks, coding agents implement them, and reviewers run fidelity audits; the resulting environments serve GUI-agent RL with observation, action, state changes, fast reset, and an LLM judge. — (a) Live sites are blocked by CAPTCHAs, drift, and login walls. (b) Contributors propose tasks, coding agents build, reviewers audit. (c) The docked environments evolve and serve RL training with fast resets.

01

Task-driven docking

Cloning Amazon's catalog or YouTube's library is impractical, so each environment is scoped to its target tasks: their entry and terminal pages, the entities they inspect, the browser actions they need, and the state they change. We start from WebVoyager's 15 websites and 643 tasks, then cover every website in Online-Mind2Web with new authenticated, state-changing workflows.

Environments keep near-miss alternatives and natural navigation paths, so there are no trivial shortcuts to the answer.

02

Skill-driven synthesis & evolution

A coding agent explores the real site in a browser and records its page hierarchy, URL patterns, controls and appearance. Guided by reusable construction skills, it builds a self-contained web app backed by SQLite with bundled images. It then runs every target task, fixes the first unsupported page or state transition, resets, and runs the task again.

Missing pages become new routes; incomplete workflows get form handlers and persistent state; thin content gets new records.

03

Human-in-the-loop refinement

Contributors propose websites and functional requirements and steer the coding agent. Reviewers compare each environment with the real site, execute every task end to end, turn defects into concrete requests for the coding agent, and write task-specific verifiers and judge rubrics.

In a pilot of 26 websites, coding agents left 13 with visual or asset defects and 14 with broken or shallow interactions. Each environment gets 2–3 review rounds of 1–2 hours.

04 · The suite

Image-rich, deep, and stateful.

Five months of community collaboration produced 153 executable website environments with 3,518 tasks. More than half of the tasks need reasoning across pages, and over a third change persistent state.

Task coverage 3,518 tasks

Task category mutually exclusive

Task property can overlap

LLM-based classification of all tasks. Hover a bar for counts, shares and website coverage.

Environment complexity vs. synthetic suites

37.3pages per environment

1,320bundled images per environment

104.8MiB of images per environment

6,611lines of code per environment

Train / test split hover a segment

3,518taskson 153 websites

Train tasks2,288
Test tasks (verified)306
Unallocated tasks924
Train-covered websites123
WebVoyager sites, held out15
Random sites, held out15

Outer ring: tasks. Inner ring: websites. Test candidates were run with five models; only tasks where scripted verifiers and an OpenCode agent judge agree on most runs are kept, at most two per website.

Built for RL seconds per operation

End-to-end latency in isolated, deterministically seeded sandboxes (Orchard), including data transfer and sandbox management.

SQLitestate per site, restored from a seed snapshot on reset

Offlineno live-web traffic; every asset ships with the environment

05 · Results

A 4B agent trained in WebHarbor transfers to the live web.

WebHarbor-4B starts from the same OpenWebRL-4B-SFT checkpoint and uses the same multimodal GRPO objective and GUI harness as OpenWebRL-4B, which was trained on 2,102 tasks across 704 live websites. Only the environments and tasks differ. All 15 WebVoyager websites are held out of training, and Online-Mind2Web's own tasks are not in the suite.

75.8live-web average: +5.2 over OpenWebRL-4B and +22.2 over the shared SFT checkpoint

28.8on WebArena, the best 4B model, in environments never seen in training

−2.7drop when WebArena auto-login is disabled, vs −3.6 and −4.9 for the OpenWebRL models

Live web

Invalid trials (e.g. reCAPTCHA) excluded. Numbers other than WebHarbor-4B as reported by OpenWebRL.

Benchmark fidelity hover a model

6 / 6pairwise rankings preserved

1.00Spearman rank correlation

0.80Pearson score correlation

Agents score 4.4–18.8 points higher on WebHarbor than on the live sites, because content drift and access restrictions no longer interfere, while their relative order is unchanged.

WebHarbor test set 306 verified tasks

Frontier models approach saturation, while 4B agents leave plenty of headroom.

WebArena 4B models, 226 tasks

"No auto-login" gives the credentials in the task instead of signing the agent in; the change from the auto-login score is shown in brackets.

Where RL helps success rate by task capability

OpenWebRL-4B-SFTOpenWebRL-4BWebHarbor-4Bgain over OpenWebRL-4B

WebHarbor-4B improves over OpenWebRL-4B on every capability group on both benchmarks (paired tests, p < 0.01), most on authenticated, multi-page and stateful workflows.

06 · Quickstart

One image. 153 websites. One port each.

All 153 environments ship in one image on Docker Hub, wildawm/webharbor. Site i listens on port 40000 + i, and a token-protected control plane on :8101 resets any site to its seeded state.

pull & run

# the control plane requires a token of at least 32 characters
export WEBSYN_CONTROL_TOKEN="$(python3 -c 'import secrets; print(secrets.token_urlsafe(48))')"
# all 153 sites in one image (about 20 GB)
docker run -e WEBSYN_CONTROL_TOKEN -p 8101:8101 \
  -p 40000-40152:40000-40152 wildawm/webharbor:latest

Open http://localhost:40001 for Amazon, :40060 for YouTube, … through :40152 for SEC.gov. The control plane starts once every site is ready, about 2–3 minutes in.
One site per image for Harbor evaluation and RL: wildawm/webharbor:<site>-v1, e.g. wildawm/webharbor:amazon-v1. To build from source instead, run ./scripts/build.sh in the repository.
Each site has sites/<site>/tasks.jsonl; every task carries a judge rubric and a deterministic verifier at sites/<site>/verify/.

reset & run an agent

# restore one site, or all of them in parallel
curl -H "Authorization: Bearer $WEBSYN_CONTROL_TOKEN" \
  -X POST http://localhost:8101/reset/amazon
curl -H "Authorization: Bearer $WEBSYN_CONTROL_TOKEN" \
  -X POST http://localhost:8101/reset-all

# minimal ReAct agent + LLM-as-judge grader
cd agent_demo && uv sync && uv run playwright install chromium
uv run python agent.py --tasks_file ../sites/amazon/tasks.jsonl \
  --task_id "Amazon--0" --out_dir runs/amazon0
uv run python eval_judge.py --run_dir runs/amazon0

07 · Community

Built in the open by 57 contributors.

WebHarbor was initiated by UNC-Chapel Hill and Microsoft and built together with volunteers from the research community, who proposed websites, steered coding agents, and reviewed each other's environments.

348GitHub pull requests, 89.9% merged

192asset pull requests on Hugging Face

33 / 29contributors / reviewers (22 in both roles)

3.3people involved per website, on average

Contributions

Contributions to the current release are closed. Thank you to everyone who built and reviewed a website. The contribution guide and review guide stay online for reference.

Project leads

Zhaoyang Wang¹, Qianhui Wu²

Core contributors

Boyu Gou³, Demi Wang⁴, Jiacheng Zhang⁵, Peichun Hua⁶, Qize Li⁷, Shi Qiu¹, Yifan Jiang⁸, Yifan Zhang⁹, Zexu Jin⁷

Contributors

Baolin Peng², Bolin Shen¹⁰, Chetan Bansal², Dehan Kong⁷, Deren Lei¹¹, Dongshen Peng¹², Fengqing Jiang¹³, Hao Cheng², Haoqing Wang¹⁴, Jiaming Shan¹⁵, Jicheng Wang¹⁶, Jingguang Zhou⁴, Jixuan He¹, Kaide Zeng¹, Kangyu Zhu¹, Lama Moukheiber¹⁷, Mufan Qiu¹, Peng Xia¹, Qihan Sun¹⁴, Qingwei Lin², Qiushi Sun¹⁸, Ruichen Zhang¹, Shuqing Luo¹, Si Qin², Sicheng Fan¹⁹, Tao Xiong²⁰, Wenlin Yao², Xinyu Zhang⁶, Xinyue Zeng²¹, Xuanrui Li¹⁴, Xuchao Zhang², Yian Wang²², Yichao Liu²³, Yiming Liang²¹, Yiyang Zhou¹, Yizhu Jiao²⁴, Yuetai Li¹³, Yurun Chen²⁰, Yuxuan Zhang²⁵, Zheyuan Yang²⁶, Zhiyuan Yao²⁰, Zhongyang Li³, Zihao Li²⁴, Zili Wang¹⁶, Jianfeng Gao², Huaxiu Yao¹

Affiliations

University of North Carolina at Chapel Hill
Microsoft
The Ohio State University
Carnegie Mellon University
University of Michigan
The Chinese University of Hong Kong
WebAgentLab
University of Southern California
University of Delaware
Florida State University
Meta Superintelligence Labs
Stanford University
University of Washington
Queen Mary University of London
University of California, Santa Barbara
University of California, Davis
Georgia Institute of Technology
The University of Hong Kong
Fudan University
Zhejiang University
Virginia Tech
National University of Singapore
Nankai University
University of Illinois Urbana-Champaign
University of British Columbia
Tongji University

Citation

Cite WebHarbor

bibtex

@misc{webharbor2026,
  title        = {WebHarbor: Docking Real Websites for Evolving GUI Agent Environments},
  author       = {{WebHarbor Team and Contributors}},
  year         = {2026},
  url          = {https://aiming-lab.github.io/webharbor.github.io},
  note         = {Project website.}
}