WebRetriever: A Large-Scale Comprehensive Benchmark for Efficient Web Agent Evaluation¶

Conference: ECCV 2026
Paper: Official paper page · PDF
Code: https://github.com/Mininglamp-AI/WebRetriever
Project: WebRetriever
Area: LLM Agent / Web Agent Evaluation
Keywords: online benchmark, network request evidence, LLM judge, operational documentation, end-to-end extraction

TL;DR¶

WebRetriever combines 1,550 tasks across 800 websites, the interaction-aware NavEval judge, and three evaluation protocols, achieving 91.2% human agreement on its benchmark while revealing that six agents average only 11.8% human-assessed end-to-end task success.

Background & Motivation¶

Evaluating a web agent requires more than checking whether it clicks through to a target page: the agent must also apply the requested query conditions and return the correct result. Controlled environments such as WebArena support reproducibility but cover relatively few websites. Online benchmarks such as Online-Mind2Web offer greater realism, yet still struggle to represent specialized knowledge, professional workflows, and diverse user intentions. High scores on a small set of familiar sites provide limited evidence of competence in enterprise analysis or professional information retrieval.

Success assessment is another bottleneck. A final screenshot may resemble the expected results page without proving that the time range, filters, or query fields were correct. Sending the entire screenshot trajectory to a judge adds redundancy and token cost. Even genuinely successful navigation can end in incorrect readings of tables, charts, or documents, so reaching a page and delivering an answer need separate evaluation.

Rather than training a new browser policy, the paper expands task coverage, the evidence available to the judge, and the definition of success together. Core idea: expose real requests and actions through rule-organized evidence, then separately measure navigation without documentation, navigation with documentation, and end-to-end completion that includes information extraction.

Method¶

Overall Architecture¶

The inputs are a user task, a predefined website entry point, and operational documentation when the protocol permits it. The evaluated agent acts on a live website while Playwright records interaction evidence. NavEval gives an LLM the actions, filtered requests, and final screenshot to decide navigation success; the end-to-end protocol additionally requires correct information extraction, which the main experiment evaluates through human judgment.

The contribution is a task-construction and evaluation workflow, not a newly trained policy network. The diagram preserves four components: dataset construction, protocol design, evidence processing, and judgment. Its human end-to-end branch is deliberate: the main results table does not report a NavEval success rate for Protocol III.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    DATA["Cross-Domain<br/>Task Construction"] --> PROTOCOL["Three Protocols and<br/>Operational Documentation"]
    PROTOCOL --> RUN["Agent Execution<br/>Playwright Recording"]
    RUN --> FILTER["Request Evidence Filtering"]
    FILTER --> JUDGE["Multi-Source Joint Judgment"]
    RUN -->|Protocol III| HUMAN["Human Verification of<br/>Navigation and Extraction"]
    JUDGE --> REPORT["Success Rate and<br/>Human Agreement"]
    HUMAN --> REPORT

Key Designs¶

1. Cross-Domain Task Construction: include professional business constraints in browser tasks

Website sampling starts with SimilarWeb traffic data and covers eight core sectors: technology and internet, business and finance, education and research, culture and entertainment and travel, life services, healthcare, industry, and public services. The authors begin with the top 30 sites by traffic in each sector, then add specialized vertical sites from authoritative navigation resources, ultimately covering 800 websites. The purpose is not merely to increase the count: it is to move beyond everyday shopping and booking tasks and test domain knowledge and authentic business logic.

Industry experts, managers, senior data analysts, and university students contribute tasks based on their experience. General or professional intents are combined with domain-specific fact elements and cross-validated for uniqueness, stability, and logical authenticity. For example, the paper argues that low-frequency over-the-counter trading should not generate artificial requests for intraday high-frequency data; quarterly or annual reports are more realistic targets. Validation therefore checks business plausibility, not just whether a page opens. Difficulty is assigned using human action counts: fewer than 6 steps is easy, 6 through 15 is medium, and more than 15 is hard. This measures human interaction length, not necessarily model reasoning difficulty. When website changes invalidate a task, the authors propose replacing it with a task of matching difficulty. That is a live maintenance policy, not a frozen-snapshot reproducibility guarantee.

2. Three Protocols and Operational Documentation: separate exploration, knowledge use, and result delivery

Protocol I supplies only the task objective and tests navigation to a qualifying target page. Protocol II additionally supplies operational documentation and tests whether the agent can use existing procedural knowledge. The documentation is not an arbitrary hint: an agent first explores the task and generates an action trajectory, which NavEval evaluates. Successful trajectories proceed to annotation; failed ones are retried and, after a retry limit, refined by humans. Annotators further verify correctness and remove redundant operations, after which an LLM turns the refined trajectories into manuals. This distinguishes not knowing how a website works from failing to execute even with instructions. The cache does not specify the retry limit or complete manual-generation prompts, so these cannot be supplied as known settings.

Protocol III moves the endpoint to accurate information retrieval from text, documents, and charts. Its tasks require authoritative sources, browser interaction rather than a simple search shortcut, and queries with unique, stable factual answers. Protocols I and II each contain 1,000 tasks, with 550 overlapping tasks that differ only in documentation availability; Protocol III contains 100 tasks. The first two therefore contain 1,450 distinct tasks, and including Protocol III is consistent with the reported total of 1,550. The protocols do not simply add requirements to one identical full task set. Differences between their main-table averages cannot be interpreted as the isolated effect of documentation or extraction; within-protocol condition changes provide more direct evidence.

3. Request Evidence Filtering: reveal query conditions that screenshots cannot establish

Playwright records screenshots, actions, and triggered network requests during execution, and NavEval also uses navigation URLs. Instead of sending all traffic directly to the LLM, it first matches requests to the task entry URL at the subdomain level. It then removes irrelevant or random fields, normalizes structured payloads, and eliminates invalid entries to produce a compact intermediate request sequence. This exposes query and filter semantics: an agent may click the expected button while sending parameters that violate the user's conditions, a discrepancy the final screenshot may not reveal.

The filtering operation can be expressed below using consistent notation, with request sequence \(R\), website URL \(U\), and filtered result \(R'\). Equation (2) is damaged in the cached text extraction; the expression below only normalizes the meaning of the adjacent prose and does not introduce another algorithm.

\[ R'=\mathcal{F}_{\mathrm{rule}}(R,U). \]

Rules organize the evidence rather than enumerate success conditions for every task; the LLM still performs the final semantic assessment. This differs from maintaining brittle, hard-coded success rules for every website. However, subdomain filtering could omit important cross-domain service requests, and field cleaning could remove relevant signals. These are risks inferred from the design, not effects quantified in the main paper. Figure 4 also includes auxiliary HTML snapshots and DOM/XPath parsing, but the prose does not provide enough detail to treat them as independent contribution modules.

4. Multi-Source Joint Judgment: cross-check actions, requests, and the final visual state

NavEval takes task description \(T\), website URL \(U\), request sequence \(R\), action sequence \(A\), and final screenshot \(I\). Filtering occurs inside the evaluator before the judge combines the evidence and produces a binary decision. Success is not inferred merely from the agent claiming completion. Requests expose fine-grained conditions, actions explain the execution process, and the screenshot shows the visible final state. These signals complement each other, but issuing a correct request does not by itself prove that the returned answer is correct.

\[ P=\operatorname{NavEval}(T,U,R,A,I),\qquad P\in\{\mathrm{True},\mathrm{False}\}. \]

The implementation uses Claude-4.5-Sonnet as the judge and evaluates existing agents, without introducing a new training loss or optimization objective. Success Rate, SR, is the fraction of tasks judged successful; Human Agreement Rate, AR, is the fraction on which automatic and human labels match. To avoid ambiguity from the damaged extraction of Equations (3) and (4), their prose definitions are normalized below. Here \(\hat y_t\) is the automatic label and \(y_t\) the human label; tables report percentages. Human-assessed SR replaces \(\hat y_t\) with \(y_t\) in the first expression.

\[ \mathrm{SR}=\frac{1}{|\mathcal T|}\sum_{t\in\mathcal T}\hat y_t, \qquad \mathrm{AR}=\frac{1}{|\mathcal T|}\sum_{t\in\mathcal T}\mathbf 1[\hat y_t=y_t]. \]

AR measures task-level label agreement, not agent success or mere closeness between two average success rates. Similar average SR values can hide errors on different tasks. Conversely, when failures dominate, overall agreement may obscure mistakes on the minority of successful tasks, motivating label-specific error analysis.

Key Experimental Results¶

Main Results¶

Every task begins at a specified entry URL, and search engine access is restricted so that real-time retrieval shortcuts cannot replace website interaction. The following table selects the human-assessed SR columns from Table 2; all values are percentages, not NavEval AR.

Agent	Protocol I: navigation	Protocol II: navigation with documentation	Protocol III: end-to-end
SeeAct	9.2	17.1	6.0
Agent-E	11.6	20.4	9.0
UI-TARS-1.5	16.5	24.8	8.0
Browser-Use	24.0	31.6	11.0
Gemini-2.5-Pro (Computer-Use)	37.1	45.2	21.0
Claude-4.5 (Computer-Use)	28.1	36.3	16.0
Average	21.1	29.2	11.8

The next table selects judge-quality results from Tables 3 and 4. WebRetriever comparisons use the same Claude-4.5-Sonnet backbone, helping distinguish evidence organization from model choice. The external-benchmark rows use different backbones, so their entire gap cannot be attributed to the NavEval design.

Dataset	Automatic evaluator	Judge backbone	Average AR (%)
WebRetriever	Autonomous Eval	Claude-4.5-Sonnet	75.9
WebRetriever	AgentTrek Eval	Claude-4.5-Sonnet	62.7
WebRetriever	WebVoyager	Claude-4.5-Sonnet	77.1
WebRetriever	WebJudge	Claude-4.5-Sonnet	81.0
WebRetriever	NavEval	Claude-4.5-Sonnet	91.2
Online-Mind2Web	WebJudge	WebJudge-7B	87.2
Online-Mind2Web	NavEval	Claude-4.5-Sonnet	97.0

Ablation Study¶

This table combines the condition ablations from Tables 5 and 6, with SR reported in percent. Documentation-ablation baselines match the NavEval columns of Table 2 rather than the human columns above. The full extraction condition matches the human result for Protocol III. These assessment conventions must not be mixed in calculations.

Protocol and condition	Gemini-2.5-Pro (Computer-Use)	Claude-4.5 (Computer-Use)
Protocol I, originally without documentation	40.9	31.3
Protocol I, add documentation	49.2	39.7
Protocol II, originally with documentation	50.1	40.1
Protocol II, remove documentation	41.4	31.9
Protocol III, require extraction	21.0	16.0
Protocol III, remove extraction requirement	43.0	34.0

Key Findings¶

With the same backbone, NavEval exceeds WebJudge by 10.2 percentage points in WebRetriever AR. This supports the value of evidence organization but does not isolate the contribution of rule-based filtering.
Adding documentation within Protocol I improves the two models by 8.3 and 8.4 percentage points; removing it within Protocol II reduces them by 8.7 and 8.2 points. These comparisons support the value of documentation more directly than subtracting cross-protocol averages.
Removing extraction from Protocol III increases success by 22.0 and 18.0 percentage points, directly exposing the gap between navigation and information delivery. These differences are not standalone extraction-module accuracies.
Table 2 reports average NavEval SR values of 23.6% and 32.3%, above the corresponding human values of 21.1% and 29.2%. High agreement does not eliminate success-rate overestimation.

Highlights & Insights¶

Use network requests as auditable evidence. Visible results often hide query parameters; request context allows finer verification of user constraints. This idea transfers to form filling and enterprise retrieval evaluation.
Treat knowledge availability as an experimental variable. Manuals are produced through exploration, screening, human refinement, and generation. This makes inability to operate and inability to use instructions distinct diagnostic problems.
Move the endpoint of success. The large change after removing extraction shows why navigation scores do not guarantee deliverable answers. Evaluations should retain both process and outcome measures.

Limitations & Future Work¶

Live maintenance does not establish version equivalence. Replacing obsolete tasks is the authors' proposed solution, but equal human action counts do not guarantee equal semantic difficulty. Task versions, website states, and paired evaluations would improve longitudinal comparisons.
Coverage breadth differs from sampling depth. There are 800 websites and 1,550 tasks, which is broad coverage but not evidence of exhaustive workflow testing within each website; Protocol III contains only 100 tasks.
Judge errors remain incompletely characterized. The main paper places backbone self-bias and filtering ablations in supplementary material absent from this cache. No filtering-removal AR or absence of self-bias can be asserted; sufficient confusion matrices and confidence intervals are also unavailable here.
Efficiency is not directly quantified. Request compression and automatic judgment are motivated by human and token costs, but the cached main paper provides no latency, token-count, or monetary-cost comparison. No numerical speedup can be reported.
Controlled conditions limit extrapolation. Restricting search engines isolates interaction capability but differs from production assistants with unrestricted search. Cross-domain request filtering and automated end-to-end verification also need finer robustness evaluation.

Compared with WebArena / VisualWebArena: those benchmarks emphasize controlled semi-real environments, whereas this paper emphasizes large-scale live websites and professional intents. They serve reproducible diagnosis and realistic coverage respectively; raw success rates alone do not determine which is better.
Compared with Online-Mind2Web / WebJudge: WebJudge selects key steps and screenshots, whereas NavEval adds cleaned request evidence. Its external 97.0% AR supports transfer, but the comparison is not fully controlled for judge backbone.
Compared with WebVoyager / AgentTrek Eval: full visual or action trajectories supply process context without necessarily exposing whether query parameters satisfy the task. A transferable direction is to evaluate structured tool-call records jointly with visible outputs rather than continually adding screenshots.

Rating¶

Novelty: 4/5. The contribution combines real-world task coverage, request-evidence organization, and deployment-oriented protocols rather than introducing a new foundation model.
Experimental Thoroughness: 4/5. Six agents, multiple judges, an external benchmark, and documentation/extraction ablations are included; filtering, self-bias, and cost evidence remain incomplete in this cache.
Writing Quality: 4/5. The three-protocol logic is clear, but readers must distinguish human from automatic SR and normalize damaged equation extraction.
Value: 4/5. The benchmark provides useful diagnostics for enterprise web agents, particularly the deployment risk of successful navigation with an inadequate final answer.