Why Act Without Asking? Evaluating Mobile GUI Agents under Ambiguous Instructions¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning / Agent / Mobile GUI
Keywords: Mobile GUI Agents, Ambiguous Instructions, Clarification, Benchmarking, Gated Cascades
TL;DR¶
Addressing the widespread "Act-without-Asking" failure mode where mobile GUI agents guess intents and proceed with actions under ambiguous instructions rather than asking for clarification, this work constructs MID-Bench—a step-level benchmark with strict user simulation and an Ask+Match protocol—and develops train-free confidence-gated cascades that boost clarification and execution accuracy without hurting standard action performance.
Background & Motivation¶
With the rapid maturation of compact multimodal foundation models and on-device execution capabilities, mobile GUI agents that predict fine-grained UI actions directly from user instructions and screen visual states are increasingly moving into real-world deployment. In everyday human-agent interactions, however, user instructions are inherently ambiguous, incomplete, or underspecified—often omitting recipient details, employing vague references, or assuming implicit personal context. In high-stakes mobile environments, acting upon an unverified interpretation of user intent poses severe risks, potentially triggering irreversible errors such as sending confidential messages to the wrong contact, altering critical system configurations, or initiating unauthorized financial transactions.
Alarmingly, current vision-language GUI agents exhibit a pervasive and systemic "Act-without-Asking" failure mode. Even when explicitly equipped with a dedicated clarification action (call_user) and instructed to seek confirmation under uncertainty, open-weight agents (including Qwen3-VL-4B, GUI-Owl-1.5-4B, UI-TARS-1.5-7B, and UI-Venus-Navi-7B) request clarification on only 0.0% to 12.3% of tasks when instructions are completely removed, and on merely 0.1% of tasks under ambiguous instructions. Instead, over 98.5% of emitted actions remain committed to physical GUI interactions like clicking or typing. Diagnostic experiments uncover a severe recognition–action gap: while agents successfully recognize ambiguity in classification mode on 14.6% to 43.8% of cases, this perceptual signal fails to propagate into generative action selection due to a strong inductive prior toward action continuation driven by UI visual affordances. Crucially, existing step-level benchmarks perpetuate and conceal this failure mode by crediting any ground-truth matching action regardless of whether clarification was sought.
To systematically diagnose and resolve this unaddressed vulnerability, an evaluation paradigm is required that assesses clarification specifically at the step where ambiguity becomes action-blocking, accompanied by lightweight inference-time interventions that activate clarification behavior without costly pretraining. Core idea: Introduce MID-Bench, a 900-step ambiguity benchmark featuring a strict user simulator and an Ask+Match scoring protocol that requires both asking and correct re-execution, alongside train-free confidence-gated cascades (LGC/SGC) that effectively intercept premature actions and restore clarification under uncertainty.
Method¶
Overall Architecture¶
The framework systematically tackles the Act-without-Asking pathology through a holistic pipeline encompassing rigorous diagnostic probing, benchmark construction, and inference-time gated cascades. During evaluation, a screenshot and an ambiguous instruction are presented to the agent. Before committing to a physical UI action, an upstream gating mechanism assesses whether sufficient information exists to act confidently. If the gate remains closed, the agent emits its baseline action directly. If the gate fires, the agent generates a clarification question directed to a strict user simulator, receives minimal grounding feedback, and subsequently executes the final action conditioned on the newly disambiguated context.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Screen State & User Instruction<br/>(Visual Screen & Ambiguous Goal)"] --> B["Clarification Gating Mechanism<br/>(PGC / SGC / LGC Decisions)"]
B -->|Gate Closed: Confident/Clear| C["Base Action Execution<br/>(Direct GUI Interaction)"]
B -->|Gate Fired: Ambiguity Detected| D["Clarification Question Generation<br/>(Trigger call_user Action)"]
D --> E["Strict User Simulator Response<br/>(Minimal Grounding Answer)"]
E --> F["Post-Clarification Re-Execution<br/>(Final Action Output)"]
C --> G["Dual-Stage Evaluation Protocol<br/>(Clear / Match-Only / Ask+Match)"]
F --> G
Key Designs¶
1. MID-Bench Construction & Strict User Simulation: Isolating Action-Critical Ambiguity without Information Leakage
To evaluate clarification specifically at the exact point of action-blocking ambiguity rather than across diffused multi-turn trajectories, MID-Bench establishes a curated benchmark of 900 evaluation steps (600 Base and 300 Hard) drawn from AndroidControl and GUI-Odyssey. Clear instructions are rewritten across four balanced ambiguity classes: Referential (vague entity names), Underspecified (missing core task arguments), Preference-dependent (subjective user criteria), and Contextual (presumed non-existent history). An LLM annotator then inspects the full trajectory to identify the exact Pivot-Step where the missing parameter first becomes necessary to determine the next action, filtering out non-blocking steps. Deterministic rules subsequently purge gesture confounds (scrolling, waiting, navigating back) and text-leakage artifacts. To prevent agents from guessing masked content, the Hard subset isolates type actions where typed strings are verified to be absent verbatim (requiring that no token of length \(\ge 4\) from the original string appears in the rewrite). Furthermore, unlike standard user simulators that answer any on-task inquiry, the proposed strict user simulator only answers queries specific enough to map to a single atomic piece of the ground-truth goal, deflecting vague inquiries with non-informative replies and enforcing rigorous question formulation.
2. Train-Free Clarification Cascades: Non-Invasive Action Interception via Predictive Confidence Gating
To activate clarification capabilities across frozen base agents without degrading their core visual-spatial grounding proficiency, three train-free clarification cascades are introduced. The Prompted-Gate Cascade (PGC) prepends a verbalized classification prompt querying whether sufficient information exists to act unambiguously; while threshold-free, it demands an entire additional LLM forward pass. The Sampling-Gate Cascade (SGC) draws \(k=5\) stochastic generations at temperature \(T=0.7\), groups them into coarse action-equivalence classes (such as quantizing click coordinates to a \(50\times 50\) spatial grid and canonicalizing typed strings), and computes the Shannon semantic entropy: $\(H = -\sum_i p_i \log p_i\)$ The gate fires whenever \(H > \tau_{\text{SGC}}\) (set to 0 across all bases, acting as a minimal disagreement detector), incurring an inference overhead of \(1+k+2\phi\) passes. In contrast, the Logit-Gated Cascade (LGC) represents the most efficient design: it directly reuses the baseline greedy-decoding forward pass by calculating the mean token log-probability of the predicted action: $\(\text{Score}_{\text{LGC}} = \frac{1}{|A|} \sum_{t \in A} \log P(y_t \mid y_{<t}, x)\)$ The gate triggers clarification whenever this predictive confidence drops below a model-calibrated threshold \(\tau_{\text{LGC}}^\star\). LGC adds zero computational cost to the gating step, offering an optimal Pareto tradeoff between latency and disambiguation accuracy.
3. Ask+Match Evaluation Protocol: Decoupling Ambiguity Perception from Fortuitous Action Matching
To resolve the fundamental flaw in existing step-level benchmarks that award full credit to agents that guess correctly without asking, the evaluation introduces a formal separation of capabilities across three metrics. Clear measures standard execution accuracy on fully specified instructions. Match-Only reflects traditional step-level evaluation by crediting any ground-truth matching action irrespective of clarification. Ask+Match establishes our primary metric: a step is credited if and only if the agent actively invokes call_user in the first stage and subsequently emits the correct ground-truth action in the second stage upon receiving simulator feedback (\(S = \mathbb{I}\{\text{Action}_1 = \text{call\_user}\} \times \mathbb{I}\{\text{Action}_2 = a^\star\}\)). The discrepancy between Match-Only and Ask+Match quantifies the exact volume of unearned credit granted to blind guesses, exposing the baseline collapse under uncertainty.
Benchmark & Evaluation Protocol¶
Experiments are conducted across four prominent open-weight multimodal GUI models: Qwen3-VL-4B, GUI-Owl-1.5-4B, UI-TARS-1.5-7B, and UI-Venus-Navi-7B. Each model is evaluated across 300 Clear steps, 600 Base Ambig steps, and 300 Hard Ambig steps. In addition, using Qwen3-VL-4B as a representative backbone, a LoRA fine-tuning contrast is developed on 4,488 examples derived from 1,496 AndroidControl training states, spanning clean execution, proactive question generation, and post-answer action completion, trained for 2 epochs on 2× RTX 3090 GPUs (rank 16, \(\alpha=32\)).
Key Experimental Results¶
Main Results¶
The primary benchmark results on MID-Bench comparing frozen base agents, train-free cascades, and LoRA adaptation under the Clear, Match-Only, and Ask+Match scoring protocols are presented below:
| Model Architecture | Method | Clear Split (Clear Acc) |
Base Ambig Match-Only |
Base Ambig Ask+Match |
Hard Ambig Match-Only |
Hard Ambig Ask+Match |
|---|---|---|---|---|---|---|
| Qwen3-VL-4B | Baseline | .468 | .406 | .000 | .107 | .000 |
| +PGC | .461 | .430 | .134 | .223 | .150 | |
| +SGC | .480 | .470 | .171 | .367 | .273 | |
| +LGC | .489 | .476 | .407 | .447 | .423 | |
| GUI-Owl-1.5-4B | Baseline | .518 | .415 | .000 | .100 | .000 |
| +PGC | .529 | .447 | .150 | .267 | .203 | |
| +SGC | .522 | .541 | .408 | .513 | .460 | |
| +LGC | .508 | .537 | .478 | .467 | .397 | |
| UI-TARS-1.5-7B | Baseline | .419 | .297 | .000 | .077 | .000 |
| +PGC | .402 | .418 | .384 | .437 | .430 | |
| +SGC | .393 | .433 | .397 | .443 | .433 | |
| +LGC | .403 | .432 | .357 | .440 | .440 | |
| UI-Venus-Navi-7B | Baseline | .466 | .370 | .000 | .097 | .000 |
| +PGC | .539 | .510 | .435 | .513 | .507 | |
| +SGC | .466 | .389 | .051 | .127 | .037 | |
| +LGC | .542 | .536 | .536 | .563 | .563 | |
| Qwen3-VL-4B (LoRA) | Baseline | .493 | .361 | .333 | .323 | .230 |
| +PGC | .479 | .416 | .391 | .333 | .247 | |
| +SGC | .472 | .423 | .413 | .543 | .490 | |
| +LGC | .471 | .424 | .424 | .577 | .573 |
Ablation & Fine-Grained Analysis¶
The breakdown of Ask+Match performance across distinct ambiguity categories and method configurations is detailed below:
| Ambiguity Category | Evaluation Pool | PGC Cascade | SGC Cascade | LGC Cascade | LoRA Baseline |
|---|---|---|---|---|---|
| Referential | Base | .252 | .117 | .372 | .570 |
| Hard | .056 | .278 | .556 | .167 | |
| Underspecified | Base | .123 | .189 | .376 | .227 |
| Hard | .119 | .233 | .383 | .218 | |
| Preference-dependent | Base | .094 | .177 | .364 | .296 |
| Hard | .138 | .483 | .586 | .207 | |
| Contextual | Base | .142 | .179 | .423 | .367 |
| Hard | .283 | .300 | .433 | .300 |
Key Findings¶
- Conventional Benchmarking Hides Total Failure: Under standard Match-Only scoring, open-weight baselines score between .297 and .415 on Base Ambig, creating the illusion of partial competence. However, under the strict Ask+Match metric requiring proactive clarification, all four frozen baselines collapse completely to an Ask+Match score of .000. Across 3,600 total evaluation decisions, baseline agents invoked
call_useronly 4 times (0.111%), confirming that models never voluntarily seek clarification. - Logit-Gating Recovers the Vast Majority of the Theoretical Ceiling: Equipping frozen base agents with LGC elevates Hard Ask+Match from .000 to .423 (Qwen3-VL-4B), .397 (GUI-Owl-1.5-4B), .440 (UI-TARS-1.5-7B), and .563 (UI-Venus-Navi-7B). Relative to an ideal Oracle gate (always-ask policy on ambiguous sets, peaking at .480–.607 due to downstream question/execution limits), train-free cascades recover 72% to 99% of achievable performance without sacrificing accuracy on Clear instructions (Clear accuracy shifts remain within \(\pm 0.03\)).
- Synergistic Composition of Supervised Fine-Tuning and Inference Cascades: While LoRA fine-tuning instills basic clarification capability (Base Ask+Match .333), it struggles on the strictly masked Hard set (.230) and shows unstable out-of-distribution transfer across datasets (AndroidControl vs. GUI-Odyssey). Combining LoRA with LGC achieves the highest overall Hard Ask+Match score of .573, demonstrating that parametric tuning and non-parametric confidence filtering are highly complementary.
Highlights & Insights¶
- Empirical Uncovering of the Recognition–Action Dissociation: The paper provides concrete evidence that mobile agents fail not because they cannot perceive ambiguity (achieving up to 43.8% ambiguity recognition in classification mode), but because generative action selection is overwhelmingly governed by UI screen affordances (retaining 30.8%–60.5% action match even when user instructions are completely stripped).
- Verifiable Hard Subset Design: By enforcing a strict token-level absence constraint (length \(\ge 4\)) on typed strings, the Hard split mathematically precludes heuristic guessing, establishing an unambiguous litmus test for genuine interactive grounding.
- Zero-Overhead Practical Deployment via LGC: Unlike verbalized prompts or multi-sample semantic entropy that introduce significant latency (0.5s to 11s), LGC extracts predictive confidence directly from the base forward pass, adding 0 seconds of gating latency and providing an instantly deployable safeguard for real-world GUI agents.
Limitations & Future Work¶
- Constraint to Single-Round Pivot Steps: MID-Bench isolates evaluation to a single-round ask–answer–act transaction at the action-blocking pivot step. Real-world human-agent collaboration often demands multi-turn dialogues with dynamic context repair across long horizons.
- Dependency on Simulator Heuristics and Mobile Scope: Although replacing the GPT-4o simulator with a local 4B model alters Ask+Match scores by less than \(\pm 0.040\), open-ended intent clarification remains constrained by simulator design. Furthermore, expanding the benchmark beyond Android into web browsers and complex multi-window desktop operating systems remains an open frontier.
Related Work & Insights¶
- vs AmbiBench / Mobile-World / Fairy: While prior GUI benchmarks evaluate clarification at the aggregate trajectory or full-task completion level—permitting agents to guess, fail, and backtrack through environmental trial-and-error—MID-Bench pinpoints the exact atomic decision step where ambiguity blocks progress, cleanly isolating Act-without-Asking via Ask+Match.
- vs Uncertainty Quantification in Generative Agents: Adapts semantic entropy and logit-based confidence estimation into structured GUI interaction spaces by establishing spatial coordinate quantization and text canonicalization signatures, successfully circumventing superficial token-level noise.
Rating¶
- Novelty: ⭐⭐⭐⭐ Formulates and isolates the pervasive "Act-without-Asking" failure mode with a sharply designed step-level benchmark and rigorous dual-stage evaluation metrics.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive coverage across 4 open-weight base agents, LoRA SFT, closed-source GPT-5.4 references, latency profiling, threshold sensitivity sweeps, and failure mode decompositions.
- Writing Quality: ⭐⭐⭐⭐⭐ Rigorous conceptual framing, highly cohesive prose narratives, insightful motivating diagnostics, and exceptional data presentation.
- Value: ⭐⭐⭐⭐⭐ Provides an indispensable benchmarking standard and cost-effective, practical cascading methods for advancing safe, interactive mobile GUI agents.