WebRISE: Requirement-Induced State Evaluation for MLLM-Generated Web Artifacts¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: Multimodal VLM
Keywords: MLLM Web Generation, Interaction Contract Graph, Behavior Evaluation, State Consistency, Browser Agent
TL;DR¶
WebRISE introduces an implementation-agnostic benchmark that reframes MLLM-generated web artifact evaluation as requirement-induced state-transition conformance through Interaction Contract Graphs, contract-guided browser agent execution, and dual-channel DOM/visual oracles.
Background & Motivation¶
Multimodal large language models (MLLMs) are rapidly advancing beyond static UI comprehension and isolated code synthesis toward generating fully executable, rich interactive web artifacts from multimodal specifications. Today, models can synthesize self-contained single-page applications containing complete HTML, CSS, and client-side JavaScript given diverse inputs ranging from textual requirements and structured Markdown wireframes to hand-drawn sketches, high-fidelity mockups, and screen-recorded user interaction videos. However, this architectural leap poses a fundamental benchmark dilemma: when should a model-generated web artifact be deemed truly usable and functional, rather than merely visually plausible?
In authentic web interaction environments, a generated page frequently fails even when all expected visual controls are present and render seamlessly. For instance, triggering a multi-criteria filter dropdown may leave product lists untouched, unchecking items in a shopping cart may fail to cascade recalculations to the subtotal and checkout button states, or navigating between tabs may quietly wipe user input histories. Prevailing evaluation paradigms largely rely on modality-fragmented and local evidence, which falls into four major categories: static appearance scoring, rigid precompiled script assertions, unconstrained web-agent exploration, or single-checkpoint result inspection. These protocols suffer from two critical systematic flaws: (i) event-centric evaluation, which only tests whether an isolated action yields an immediate reaction without defining the complete requirement-induced state space and transition paths; and (ii) a severe state-consistency gap, where local actions pass superficial checks while violating cross-component invariants, cross-view synchronizations, or lifecycle data persistence.
To resolve the inability of current benchmarks to fully enumerate, isolate, and diagnose requirement-level interactive behavior, this paper posits that evaluating interactive web artifacts requires verifying conformance to requirement-derived state transitions rather than checking arbitrary implementation details or surface appearances. Core idea: formalize multimodal web artifact evaluation as requirement-induced observable state-transition conformance by compiling explicit and implicit requirements into an Interaction Contract Graph (ICG) with dual DOM/visual oracles, executing it via an adaptive browser agent to systematically decouple functional affordance delivery from implicit state-consistency constraints.
Method¶
Overall Architecture¶
WebRISE operates over a task requirement set \(R_\tau\) instantiated across five input modalities (Text, Markdown, Sketch, Image, Video). A candidate MLLM \(f_\theta\) consumes a specification and outputs a self-contained executable HTML artifact \(H\) free from external back-end dependencies. WebRISE compiles the task specification into an Interaction Contract Graph (ICG) \(G_\tau = (S_\tau, T_\tau, \Phi_\tau, M_\tau)\) where states denote observable stable UI configurations, transitions encode intent-driven state changes, predicates define dual DOM/visual validation criteria, and mapping functions attribute verdicts back to explicit and implicit requirements. A contract-guided browser agent executes transitions on the live webpage using ephemeral indexed DOM observations, and dual-channel oracles evaluate both process changes and final stable outcomes to produce fine-grained diagnostic scores.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multimodal Task Specifications<br/>Text / MD / Sketch / Image / Video"] --> B["MLLM Generated HTML Artifact"]
C["Task Requirement Set<br/>Explicit Functions + Implicit Constraints"] --> D["Interaction Contract Graph Construction<br/>Stable States / Intent Transitions / Dual Oracles"]
B --> E["Contract-Guided Agent Execution<br/>Dynamic Indexed DOM / Preconditions / Replay"]
D --> E
E --> F["DOM & Visual Dual-Path Verification<br/>Transient Process Evidence + Visual Postconditions"]
F --> G["Fine-Grained Diagnostic Aggregation<br/>S% / T% / Re% / Ri% / R%"]
Key Designs¶
1. Interaction Contract Graph Construction: compiling unstructured product requirements into state-transition networks with decoupled transients Traditional automated testing binds tightly to brittle selectors or full DOM snapshots that break across diverse model-generated codebases. WebRISE addresses this by normalizing domain tasks into explicit requirements (user-facing features such as filtering, searching, or dragging) and implicit requirements (product-level constraints like cross-view synchronization, disabled boundary states, debounce, and stale-data removal). These requirements are compiled into an Interaction Contract Graph \(G_\tau = (S_\tau, T_\tau, \Phi_\tau, M_\tau)\). Crucially, the state set \(S_\tau\) strictly captures stable, replayable, observable UI configurations, whereas transient intermediate phenomenaโsuch as loading spinners, saving toasts, animation frames, and temporary disabled statesโare factored out and attached to transition edges \(T_\tau\) as process-level predicates. This keeps the reachable state space compact (averaging 11.5 states and 12.4 transitions per task) while maintaining granular coverage mappings \(M_\tau\) back to individual requirements.
2. Contract-Guided Agent Execution: navigating dynamic page states with ephemeral DOM indexing and trajectory replay isolation
Generated web artifacts exhibit vast variations in DOM hierarchy, styling classes, and semantic tags, rendering precompiled test scripts brittle. WebRISE deploys an adaptive browser agent that inspects an indexed, accessibility-aware live DOM observation rather than static CSS selectors. Actionable elements (including native inputs, custom ARIA widgets, non-negative tabindex nodes, and interactive event handlers) are dynamically assigned ephemeral indices regenerated after every action. For each transition \(t_j = (s_{\text{from}}^j, s_{\text{to}}^j, g_j, P_j, A_{\text{dom}}^j, A_{\text{vis}}^j)\), the agent executes purposeful interactions (click, hover, type, drag-and-drop, text selection) toward goal \(g_j\). To isolate branched state transitions from cascading failures, the evaluator restores source state \(s_{\text{from}}^j\) by replaying a cached, previously verified trajectory \(\Pi\). Unreachable source states or failed preconditions immediately trigger SKIPPED or BLOCKED outcomes without falsely penalizing downstream test logic.
3. DOM & Visual Dual-Path Verification: decoupling process-level mutation signals from user-visible steady-state assertions
Relying solely on visual oracles suffers from hallucination and misses background logic, while relying solely on DOM inspection misses visual rendering bugs. WebRISE designs complementary verification paths. The DOM assertion path inspects execution event logs and DOM diffs, applying [CHANGE] predicates to verify transient process evidence (e.g., whether a skeleton screen or loading state was active during fetching) and [AFTER] predicates to confirm persistent attributes (e.g., updated ARIA states, element removals, disabled attributes). Concurrently, the visual assertion path compares pre- and post-interaction viewport screenshots using vision models, evaluating behavioral postconditions such as canvas object hiding, grid sorting rearrangements, and price breakdown updates. Both paths enforce a conservative decision rule: only unambiguous confirmations pass as YES, while ambiguous signals return UNCERTAIN and fail the transition check.
4. Fine-Grained Diagnostic Aggregation: projecting execution outcomes into orthogonal interaction and consistency metrics
Rather than collapsing evaluation into a coarse pass/fail binary, WebRISE projects execution results onto distinct diagnostic dimensions. Each transition outcome is categorized into \(\{ \text{PASS}, \text{FAIL}, \text{BLOCKED}, \text{SKIPPED} \}\), requiring reachable source states, fulfilled goals, and passing dual assertions for a PASS. Aggregation yields state reachability \(S\%\), transition validity \(T\%\), and requirement-level coverage \(C(\hat{R})\). By mapping assertions back to requirement subsets, WebRISE isolates explicit functional coverage \(Re\%\) from implicit constraint coverage \(Ri\%\), computing the explicit-implicit gap \(\Delta = Re\% - Ri\%\). This diagnostic projection unmasks whether an artifact fails due to missing entry-point widgets, faulty event wiring, or broken underlying data consistency.
A Worked Example¶
Consider an e-commerce shopping cart interaction task:
1. Source State Restoration & Preconditions: Replaying a verified action sequence brings the generated page to source state \(S_0\): exactly one checked item in the cart (unit price $398, checked subtotal $398, Proceed to Checkout button enabled).
2. Agent Interaction Execution: The agent receives transition goal \(g_j\): "Uncheck the item checkbox in the shopping cart." Parsing the dynamic DOM index, the agent locates the checkbox and fires a toggle action.
3. Dual-Channel Oracle Verification:
- DOM Assertion: Checks [AFTER] The Proceed to Checkout entry is in a disabled state.
- Visual Assertion: Verifies that post-interaction screenshots depict a checked subtotal of zero and updated summary values.
4. Diagnostic Attribution: In a failing artifact generated by a weaker model, the checkbox UI visually toggles, but price calculations remain stale ($398) and the checkout button remains active. The DOM assertion flags NO (checkout button is visible with disabled: false), and visual inspection flags NO (total remains $398). WebRISE tags the transition as FAIL and attributes it specifically to a "State & Logic" consistency error rather than an execution or affordance failure.
Key Experimental Results¶
Main Results¶
WebRISE benchmarks 14 representative proprietary and open-weight models across 442 tasks and 5 input modalities. The table below details transition validity (\(T\)), overall requirement coverage (\(R\)), auxiliary visual quality (\(V\)), and the macro-averaged Overall score:
| Model Group | Model | Text (T / R / V) | MD (T / R / V) | Sketch (T / R / V) | Image (T / R / V) | Video (T / R / V) | Overall |
|---|---|---|---|---|---|---|---|
| Proprietary | GPT-5.5 | 60.3 / 62.3 / 85.6 | 64.4 / 66.1 / 83.3 | 60.6 / 62.9 / 86.1 | 61.8 / 63.4 / 74.1 | 65.6 / 66.3 / 73.9 | 69.1 |
| Proprietary | GPT-5.4 | 59.7 / 61.4 / 78.4 | 60.5 / 62.2 / 79.8 | 57.8 / 60.3 / 86.6 | 60.0 / 62.1 / 71.5 | 63.1 / 64.8 / 73.7 | 66.8 |
| Proprietary | Claude Opus 4.7 | 48.8 / 50.9 / 68.3 | 54.5 / 56.5 / 76.2 | 49.7 / 52.4 / 77.4 | 57.0 / 58.5 / 70.5 | 65.0 / 66.1 / 72.7 | 61.6 |
| Proprietary | Gemini 3.1 Pro | 50.7 / 53.6 / 69.7 | 58.9 / 61.5 / 79.2 | 52.2 / 54.9 / 84.8 | 54.5 / 57.1 / 72.2 | 52.0 / 54.9 / 71.6 | 61.9 |
| Proprietary | Qwen3.6-Plus | 49.3 / 51.9 / 68.2 | 51.7 / 54.6 / 74.5 | 53.8 / 56.4 / 86.3 | 57.5 / 59.4 / 73.8 | 61.7 / 63.4 / 74.8 | 62.5 |
| Proprietary | Gemini 3 Flash | 44.7 / 48.2 / 71.9 | 50.0 / 54.1 / 79.3 | 46.1 / 49.3 / 85.4 | 54.1 / 57.5 / 72.4 | 45.6 / 48.5 / 70.8 | 58.5 |
| Proprietary | Claude Opus 4.6 | 43.3 / 45.5 / 56.6 | 54.3 / 56.3 / 73.9 | 52.3 / 55.0 / 72.2 | 57.7 / 59.5 / 70.2 | 52.6 / 54.9 / 70.7 | 58.3 |
| Open-Weight | Kimi-K2.6 | 44.6 / 47.3 / 83.1 | 51.7 / 54.9 / 87.1 | 47.8 / 51.5 / 86.3 | 58.5 / 60.4 / 73.2 | 63.7 / 65.4 / 73.5 | 63.3 |
| Open-Weight | Qwen3.6-27B | 47.9 / 50.9 / 75.3 | 57.5 / 60.1 / 83.0 | 50.4 / 53.3 / 87.2 | 55.2 / 57.8 / 74.1 | 54.2 / 57.2 / 74.1 | 62.5 |
| Open-Weight | Kimi-K2.5 | 48.5 / 51.9 / 68.9 | 57.0 / 59.6 / 73.8 | 47.8 / 50.4 / 79.9 | 56.9 / 59.1 / 72.6 | 58.6 / 60.3 / 72.9 | 61.2 |
| Open-Weight | Qwen3.5-397B-A17B | 45.7 / 49.2 / 64.8 | 51.1 / 54.5 / 75.7 | 46.8 / 50.5 / 78.9 | 48.4 / 51.4 / 72.8 | 49.3 / 52.8 / 72.1 | 57.6 |
| Open-Weight | Qwen3.5-27B | 36.3 / 40.0 / 59.9 | 41.7 / 45.5 / 72.1 | 38.6 / 42.7 / 76.8 | 42.6 / 46.7 / 70.6 | 43.1 / 46.9 / 71.8 | 51.7 |
| Open-Weight | Qwen3.5-122B-A10B | 38.0 / 41.2 / 56.8 | 42.5 / 45.9 / 72.0 | 38.0 / 42.3 / 74.0 | 40.2 / 43.8 / 70.7 | 42.8 / 47.1 / 71.3 | 51.1 |
| Open-Weight | Qwen3.6-35B-A3B | 26.8 / 30.5 / 78.2 | 15.5 / 19.2 / 80.8 | 41.2 / 45.4 / 77.0 | 46.6 / 49.6 / 71.7 | 49.5 / 52.2 / 72.8 | 50.5 |
Ablation & Diagnostic Studies¶
The table below details modality-level averages across all 14 evaluated models and tasks, contrasting explicit requirement coverage \(Re\), implicit constraint coverage \(Ri\), the consistency gap \(\Delta = Re - Ri\), and evaluator defect detection sensitivity across 25 controlled state mutations:
| Configuration / Modality | Transition Validity T | Explicit Req. Re | Implicit Req. Ri | Consistency Gap \(\Delta \downarrow\) | Overall Req. R | Visual Score V | Defect Detection Rate (DR) |
|---|---|---|---|---|---|---|---|
| Modality Average: Text | 46.0 | 56.4 | 43.0 | 13.5 | 48.9 | 70.4 | - |
| Modality Average: Markdown | 50.8 | 61.1 | 47.6 | 13.5 | 53.7 | 77.9 | - |
| Modality Average: Sketch | 48.8 | 60.1 | 45.4 | 14.7 | 52.0 | 81.4 | - |
| Modality Average: Image | 53.6 | 62.8 | 50.8 | 12.0 | 56.2 | 72.2 | - |
| Modality Average: Video | 54.8 | 61.3 | 53.6 | 7.7 | 57.2 | 72.6 | - |
| Evaluator Sensitivity: ICG (Ours) | - | - | - | - | - | - | 16/25 (64.0%) |
| Evaluator Sensitivity: WebGen (Broad) | - | - | - | - | - | - | 8/25 (32.0%) |
| Evaluator Sensitivity: WebGen (Strict) | - | - | - | - | - | - | 1/25 (4.0%) |
Key Findings¶
- Interactive web generation is far from solved: Even the premier proprietary model, GPT-5.5, reaches only \(T = 65.6\%\) and \(R = 66.3\%\) in its best-performing modality (Video), indicating that roughly one third of user-intent transitions and product constraints fail under browser execution.
- Visual fidelity is decoupled from interactive validity: Superficial appearance scores do not guarantee functional correctness. For instance, Qwen3.6-35B-A3B on Markdown achieves an impressive visual score (\(V = 80.8\)), yet collapses on behavioral execution (\(T = 15.5\%\), \(R = 19.2\%\)).
- Video demonstration provides the strongest inductive signal: Compared to text specifications, interaction videos boost average transition validity \(T\) by 8.8 percentage points, implicit coverage \(Ri\) by 10.6 percentage points, and narrow the explicit-implicit gap from 13.5 to 7.7 points, demonstrating the power of temporal visual priors in conveying state transitions.
- State and logic consistency is the predominant failure mode: In failure attribution analyses for both top models (GPT-5.5 and Kimi-K2.6), State & Logic errors account for nearly 60% of all transition failures (57.2% and 59.7%), followed by Feedback & Boundary errors (24.6% and 23.5%), whereas basic element availability accounts for under 13%. Models successfully emit interactive controls, but fail to maintain data cascades and reactive states.
Highlights & Insights¶
- Requirement-induced state abstraction: By formulating web app tasks as Interaction Contract Graphs (ICGs) and projecting transient feedback into process-level transition assertions, WebRISE formalizes black-box web evaluation into an implementation-agnostic state graph traversal problem.
- Dynamic indexing eliminating selector dependencies: Employing runtime-refreshed indexed DOM trees and accessibility affordance discovery frees the automated testing agent from brittle CSS/XPath bindings, enabling zero-shot execution across disparate coding patterns.
- Trajectory replay isolating cascading bugs: Restoring verified state sequences before evaluating branched transitions prevents upstream defects from masking or falsely penalizing independent downstream interactions.
Limitations & Future Work¶
- Scope bounded to self-contained artifacts: WebRISE focuses on standalone, client-side web applications executed in a sandboxed browser environment, leaving out distributed multi-tier architectures with authentic back-end APIs, persistent relational databases, OAuth authentication, and concurrent multi-user sessions.
- Finite contract graph coverage: While constructed by industry practitioners and verified via Ground-Truth execution, each ICG remains a discrete projection of specified requirements and does not exhaustively simulate random or adversarial user interactions.
- Future Directions: Expanding ICGs with sandboxed mock API contracts, session-level database fixtures, and multi-agent adversarial usability stress testing.
Related Work & Insights¶
- vs WebCoderBench / FrontendBench: Prior efforts depend heavily on static AST inspection or precompiled script selectors, failing as soon as a model alters DOM tag names or nesting. WebRISE leverages an adaptive, intent-driven agent with dynamically indexed observations, achieving true implementation-agnostic testing.
- vs WebGen-Bench / ArtifactsBench: Checkpoint-based agents check whether a page superficially reaches a terminal visual state, missing hidden state corruption, cross-feature interference, and broken data cascades. WebRISE's state-transition assertions detect injected state defects at 2ร to 16ร the rate of checkpoint protocols.
Rating¶
- Novelty: โญโญโญโญโญ Formulates web artifact evaluation as requirement-induced state-transition conformance and decouples transient process states from observable UI configurations.
- Experimental Thoroughness: โญโญโญโญโญ Evaluates 14 models across 5 modalities, 442 tasks, and over 12,000 assertions, backed by human agreement (\(\kappa=0.74\)) and defect injection meta-evaluations.
- Writing Quality: โญโญโญโญโญ Rigorous mathematical formalization, clean structural abstractions, and insightful failure taxonomy.
- Value: โญโญโญโญโญ Provides a foundational benchmark and diagnostic protocol for the next generation of autonomous front-end engineering models.