EvoGenUI-Bench: Evaluating LLMs as Multi-Turn Generative UI Assistants¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/MAPS-research/EvoGenUI-Bench
Area: LLM Reasoning
Keywords: Generative UI, Multi-turn interface maintenance, Executable code evaluation, Cross-turn retention, Tool grounding
TL;DR¶
EvoGenUI-Bench introduces a multi-turn generative UI maintenance benchmark of 150 five-turn tasks (750 turns), executing artifacts in browser sandboxes to gather multi-surface evidence bundles (screenshots, DOM, Playwright actor traces, source code, and runtime logs) and demonstrating that even the strongest model falls from a 74.9% Turn Pass rate to a 37.3% five-turn completion rate.
Background & Motivation¶
Large language models have shown remarkable capability in directly generating interactive web applications, dashboards, forms, and custom comparison views, establishing generative user interfaces (Generative UI) as an emerging direct response modality for AI assistants. However, existing frontend benchmarks predominantly focus on isolated, single-turn page generation, multi-file web repository generation from scratch, or web agents operating within static, pre-existing external web environments. These formulations artificially bifurcate artifact construction from task execution, assuming either that the model builds an application once or that it merely navigates an interface supplied by the environment. In authentic assistant workflows, users rarely request independent pages in isolation; instead, they iteratively refine an active, executable interface by introducing progressive filters, adjusting constraints, and expanding workflows over multi-turn interactions.
The core tension in this multi-turn artifact maintenance setting lies in the delicate synchronization of interface evolution: an assistant must satisfy each newly introduced request while strictly preserving all prior requirements that remain valid. Under continuous revisions, models frequently suffer from catastrophic regression on prior components, break internal state machine propagation, or introduce deceptive "unwired controls" that appear visually polished but fail to trigger backend state mutations. Traditional evaluation paradigms based on static code parsing, pure-text LLM judges, or reference screenshot comparisons are fundamentally incapable of validating the runtime integrity of executable artifacts. They cannot determine whether an interface is a truly responsive, state-synchronized system or an inert, disconnected shell, nor can they measure the structural and behavioral drift that accumulates across successive turns.
To bridge this critical evaluation blind spot, this work reformulates generative UI evaluation as the persistent maintenance of an executable artifact across multi-turn revisions, spanning information presentation, local state interaction, and external tool-grounded state management. Core idea: reframe generative UI evaluation as multi-turn executable artifact maintenance assessed via browser execution, multi-surface evidence bundles, and statistical cross-turn retention metrics such as Adjacent Pass Retention (APR) to systematically diagnose state desynchronization and functional regression.
Method¶
Overall Architecture¶
EvoGenUI-Bench evaluates LLMs across 150 five-turn cumulative revision tasks (750 turns total). At each turn, the generator model receives the current user request, public task context, optional tool schemas, compact summaries of prior turns, and the complete source code from the previous turn. The model outputs a conversational assistant response alongside the complete, updated React application source code. The benchmark harness immediately builds the returned code in a sandboxed Vite + React browser environment and renders it in headless Chromium. An automated multimodal interaction actor explores the running page to record behavioral execution traces, while the harness extracts screenshots, final DOM trees, source diffs, and runtime tool invocation logs. A model-based evaluator then scores the turn against hidden private validation contracts across three dimensions and computes cross-turn retention metrics.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multi-Turn User Request & Context<br/>(5-Turn Trajectory / Prior Source)"] --> B["Multi-Scenario Task Suite & Private Contracts<br/>Presentation · Interaction · Tool-grounded"]
B --> C["Browser Execution Sandbox & Action Probes<br/>Vite/React Build + Headless Chromium Actor"]
C --> D["Multi-Surface Evidence Bundle Collection<br/>Screenshot · DOM · Actor Trace · Source · Logs"]
D --> E["Three-Dimensional Scoring & Retention Analysis<br/>Presentation / Execution / Alignment + APR / CPT"]
Key Designs¶
1. Multi-Scenario Task Suite & Private Contracts: Overcoming the unrepresentativeness of single-turn static generation by constructing a balanced tripartite suite with generator-blind private validation contracts
Existing UI datasets rely on isolated templates that do not reflect the cumulative cognitive pressure of long-horizon interface maintenance. EvoGenUI-Bench constructs 150 distinct human-authored tasks across 150 unique domain labels, evenly partitioned into three 50-task scenario suites: Presentation (focusing on visual hierarchy, dense information organization, and adaptive layouts), Interaction (evaluating stateful mini-apps with complex local state transitions and linked views), and Tool-grounded (testing interfaces that read, write, reconcile, and synchronize authoritative external mock state). Crucially, to prevent benchmark leakage and gaming, generator-visible prompts and public schemas are strictly decoupled from private validation contracts. These hidden contracts specify an average of 6.9 verifiable behavioral and state expectations per turn (3.0 for Presentation, 5.9 for Interaction, and 11.9 for Tool-grounded) that evaluate semantic functional correctness rather than rigid surface syntax.
2. Browser Execution Sandbox & Action Probes: Overcoming static code inspection blind spots by running full builds and dynamic agentic browser exploration
Static AST linters and compilation passes routinely fail to detect hollow interfaces where interactive buttons exist visually but lack event handler bindings. EvoGenUI-Bench builds every generated codebase within a standard Vite + React environment. Successfully built artifacts are mounted in headless Chromium, where a vision-language model actor (MiMo-V2.5 by default) identifies the primary operational affordances corresponding to the current user turn. The actor executes targeted mouse clicks, text entries, and slider adjustments, verifying whether the interface actually triggers dynamic responses or required backend mutations. If an artifact suffers a build failure at turn \(t\), all dependent downstream turns are logged as blocked failures rather than silently skipped or reset from a clean state, faithfully penalizing brittle code maintenance.
3. Multi-Surface Evidence Bundle Collection: Overcoming single-modality evaluator hallucinations by assembling coupled visual, structural, behavioral, and backend evidence
Evaluating open-ended generative UI through a single modality is fundamentally error-prone: a model can claim in text that data has been safely committed to the database, or display a decorative toast notification, while no network request was ever dispatched; conversely, an element may exist in the DOM but be rendered completely clipped or illegible in the viewport. The benchmark harness bundles five distinct empirical evidence streams for each turn: full-page rendered screenshots and DOM trees (capturing visual hierarchy and rendering artifacts), Playwright actor interaction traces and observed state changes (capturing dynamic UI responsiveness), source code differences (verifying component architecture updates), and runtime tool/resource logs (verifying ground-truth backend state mutations). Evaluator models must ground their judgments exclusively in this concrete evidence bundle.
4. Three-Dimensional Scoring & Retention Analysis: Overcoming isolated turn metrics by establishing fine-grained rubric scoring and cumulative cross-turn retention indicators
At the turn level, the evaluator assigns 1–5 rubric scores across three orthogonal axes: Presentation (coherence, aesthetic styling, visual hierarchy, and readability), Execution (functional satisfaction of the current request and all still-valid prior requirements supported by runtime proof), and Alignment (mutual consistency among the assistant text, generated code, visible UI, interaction observations, and tool logs). A turn is designated a Turn Pass (\(p_{T,t} = 1\)) if and only if all three dimensions score \(\ge 4\). To capture cross-turn resilience across the evolving five-turn artifact, the paper introduces Adjacent Pass Retention (APR), defined as the empirical probability that turn \(t\) passes given that turn \(t-1\) passed and a model completion was obtained: $$ \text{APR}(S) = \frac{\sum_{T \in S} \sum_{t=2}^{N} e_{T,t} p_{T,t-1} p_{T,t}}{\sum_{T \in S} \sum_{t=2}^{N} e_{T,t} p_{T,t-1}} $$ where \(e_{T,t} \in \{0, 1\}\) indicates whether generation returned a response rather than encountering a provider outage. Complementing APR with complete five-turn episode success (TP@5) and Consecutive Passed Turns (CPT), this framework provides a comprehensive statistical lens on iterative code maintenance.
Key Experimental Results¶
Main Results¶
Eight representative models spanning major frontier families (GPT-5.5, Claude-Opus-4.7, Claude-4.5-Haiku, Qwen3.6-Plus, Gemini-3-Flash, Gemini-3.1-Pro, Qwen3-Coder, and GLM-4.5-Air) were evaluated across all 150 tasks (750 requested turns).
| Model | Presentation TP (%) | Interaction TP (%) | Tool-grounded TP (%) | Overall TP (%) | Overall TP@5 (%) | Overall CPT | Overall APR (%) |
|---|---|---|---|---|---|---|---|
| Claude-Opus-4.7 | 82.4 | 81.2 | 61.2 | 74.9 | 37.3 | 2.81 | 83.6 |
| GPT-5.5 | 44.8 | 77.6 | 62.8 | 61.7 | 21.3 | 2.21 | 71.0 |
| Claude-4.5-Haiku | 76.0 | 56.0 | 21.2 | 51.1 | 13.3 | 1.61 | 65.7 |
| Qwen3.6-Plus | 60.0 | 46.8 | 25.2 | 44.0 | 7.3 | 1.32 | 61.7 |
| Gemini-3-Flash | 62.8 | 47.6 | 14.8 | 41.7 | 4.7 | 1.02 | 57.8 |
| Gemini-3.1-Pro | 38.4 | 25.2 | 7.2 | 23.6 | 6.7 | 0.69 | 68.6 |
| Qwen3-Coder | 44.0 | 21.2 | 4.4 | 23.2 | 2.0 | 0.77 | 42.2 |
| GLM-4.5-Air | 35.2 | 24.8 | 3.2 | 21.1 | 1.3 | 0.60 | 48.5 |
| Aggregate Mean | 55.5 | 47.6 | 25.0 | 42.7 | 11.8 | 1.38 | 66.5 |
Ablation Study¶
The necessity of the multi-surface evidence bundle was validated on a balanced reference set of 240 turns (120 passing, 120 failing) annotated by human consensus.
| Evaluator Configuration | Accuracy Acc. (%) | \(\Delta \text{Acc.}\) (%) | Precision Prec. (%) | Recall Rec. (%) | F1 Score (%) | Note |
|---|---|---|---|---|---|---|
| Full evidence bundle | 87.5 | — | 86.9 | 88.3 | 87.6 | Complete code, screenshots, actor traces, contracts |
| w/o Actor interaction trace | 55.0 | -32.5 | 54.7 | 58.3 | 56.5 | Largest drop; cannot verify dynamic behavior/handlers |
| w/o Private validation contract | 63.8 | -23.7 | 92.3 | 30.0 | 45.3 | Without explicit tests, recall collapses to 30.0% |
| w/o Source code | 67.5 | -20.0 | 67.2 | 68.3 | 67.8 | Cannot inspect component logic or data binding |
| w/o Rendered screenshot | 78.3 | -9.2 | 78.3 | 78.3 | 78.3 | Cannot perceive visual layout overlapping and clipping |
Key Findings¶
- Turn-level pass rate severely overestimates multi-turn system reliability: Even for the top-performing Claude-Opus-4.7, which achieves a high 74.9% overall Turn Pass rate, its end-to-end episode success rate (TP@5) plummets to 37.3%. Across all eight models, the average Turn Pass is 42.7%, while the average TP@5 is only 11.8%, demonstrating that models struggle immensely to sustain functional validity across successive revisions.
- Reliability deteriorates sharply beyond the second turn: Across models, pass rates hold steady over turns 1 and 2 (~55% and ~50% on Presentation), but plunge to 39.4% at turn 3 and 35.1% at turn 4. The collapse is especially stark in Tool-grounded tasks, where turn-level pass rate falls from 39.5% at turn 2 to 18.5% at turn 3 and 14.0% at turn 4.
- Distinct failure mechanisms dominate across task suites: An analysis of 2,750 executed non-passing turns mapped onto a six-mechanism taxonomy shows that Presentation failures are overwhelmingly driven by information architecture defects (84.5%, such as clipped SVG text or illegible layouts). Interaction failures center on derived-state propagation (31.8%) and unwired affordance binding (19.9%). Tool-grounded failures exhibit compound breakdowns spanning derived state (25.0%), affordance binding (23.2%), external state grounding (20.9%), and requirement decomposition (18.5%).
- More than half of failed transitions involve regression on prior features: A post-hoc audit of 110 failed APR transitions shows that 58 cases (52.7%) involved explicit regression on previously working functionality or valid state contracts, rather than solely failing to implement the new requirement.
Highlights & Insights¶
- Reframing UI generation as persistent artifact maintenance: Rather than evaluating ephemeral one-shot web page generation, EvoGenUI-Bench establishes the first rigorous protocol for testing whether executable interface behavior, derived state, external state, and assistant assertions remain mutually synchronized over time.
- Actor traces are the single most critical evaluation surface: The evidence ablation reveals that removing automated browser interaction traces slashes evaluator accuracy by 32.5 percentage points, proving that multimodal LLM judges are effectively blind to broken state machines and dead controls without active exploration.
- Adjacent Pass Retention (APR) uncouples retention from absolute capability: By conditioning next-turn success specifically on previous-turn success, APR provides a principled, scale-invariant diagnostic metric for isolating regression risks across arbitrary multi-turn coding and agentic workflows.
Limitations & Future Work¶
- Author-admitted limitations: The benchmark evaluates synthetic five-turn trajectories within a fixed React/Vite sandbox, omitting diverse frontend frameworks (e.g., Vue, Svelte), mobile viewports, production server-side rendering, and accessibility standards. Tool-grounded tasks use deterministic mock runtimes that eliminate network latency, transient outages, authentication failures, and rate limits. Furthermore, APR is an outcome-level retention metric rather than a fine-grained causal tracker.
- Additional observations: All tasks progress linearly through five incremental additions without testing branching undo operations, explicit user feature rollbacks, or contradictory user revisions.
- Promising directions: Future extensions could explore non-linear branching revision trees, live web service integrations with flaky network conditions, and symbolic constraint solvers to generate deterministic property-based test suites alongside neural actor probes.
Related Work & Insights¶
- vs FronTalk & MiniAppBench: FronTalk investigates multi-turn frontend coding but focuses on visual/text feedback on self-contained components; MiniAppBench explores single-turn, principle-driven interactive HTML applications. EvoGenUI-Bench is the first to evaluate five-turn cumulative maintenance coupled with external mock backend state synchronization.
- vs SWE-bench & WebArena: SWE-bench evaluates patch generation for existing software repositories; WebArena evaluates agents navigating existing web environments where the environment provides the UI. EvoGenUI-Bench inverts the WebArena paradigm: the model synthesizes and maintains the UI layer, while an external agent probes its operational fidelity.
- vs τ-bench & ToolSandbox: Stateful tool benchmarks evaluate whether an agent chooses the correct API to modify backend databases. EvoGenUI-Bench goes a step further by evaluating whether those state mutations are accurately, responsively, and reversibly exposed to human users through an evolving frontend artifact.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Establishes a foundational, much-needed paradigm shift from isolated one-shot UI synthesis to cumulative multi-turn artifact maintenance.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Thorough evaluation across 8 frontier model families, 750 turns, multi-surface ablation, and 2,750 categorized failure records.
- Writing Quality: ⭐⭐⭐⭐⭐ Rigorous conceptual framing, precise formal metric definitions, and crisp empirical insights with concrete case studies.
- Value: ⭐⭐⭐⭐⭐ An indispensable benchmark for advancing generative AI from one-off code toy demos toward reliable, production-grade interactive software engineering assistants.