Why Act Without Asking? Evaluating Mobile GUI Agent Clarification Strategies¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Agent
Keywords: Mobile GUI Agent / Instruction Ambiguity / Active Clarification / Evaluation Benchmark / Cascade Gating
TL;DR¶
Addressing the pervasive "Act-without-Asking" pathology where mobile GUI agents almost never query users under ambiguous goals (clarifying in only 0.1% of cases), this paper introduces the step-level MID-Bench benchmark with a strict user simulator and shows that train-free logit cascades boost ambiguous-task success from 0% to 40%–56% without degrading clear-instruction accuracy.
Background & Motivation¶
Recent advances in compact multimodal models have enabled the direct on-device deployment of mobile GUI agents that translate visual screen observations and natural language goals into concrete interface actions such as clicks, typing, and swipes. Existing mobile agent benchmarks—including AndroidControl, AndroidWorld, and GUI-Odyssey—evaluate agents primarily under the implicit assumption that user instructions are complete, fully specified, and unambiguous. However, in real-world everyday interactions, instructions provided by users are frequently plagued by underspecified parameters, vague references, missing context, or subjective preferences. When an agent proceeds blindly under incomplete specifications, it risks triggering catastrophic, irreversible consequences in sensitive environments, such as transferring funds to the wrong account, modifying critical security settings, or dispatching private correspondence to an unintended recipient. Reliable and safe mobile GUI agents must therefore possess the critical ability to recognize when an instruction is underspecified, halt execution, and proactively elicit user clarification before committing to a concrete action.
Crucially, current open-weight GUI agents exhibit a severe and systemic behavioral bias termed Act-without-Asking: even when explicitly equipped with a clarification API (call_user) and instructed to ask whenever uncertainty arises, agents emit clarification questions in merely 0.1% of ambiguous scenarios (4 out of 3,600 evaluated steps). Diagnostic probing uncovers a profound "recognition-action gap" combined with weak instruction conditioning. While models such as GUI-Owl-1.5-4B can recognize ambiguity in up to 43.8% of cases under an explicit classification setup, their asking rate collapses to exactly 0.0% during autonomous action generation. Furthermore, when instructions are completely removed, agents still match ground-truth GUI actions in 30% to 60% of steps purely by following visual affordances, revealing that agents prioritize screen-driven plausible execution over faithful instruction alignment. Existing step-level benchmarks exacerbate this pathology through survivorship bias: because any action that fortuitously matches the ground-truth trajectory is credited with full score, agents are continually rewarded for ungrounded guessing rather than penalized for failing to resolve ambiguity.
To resolve this methodological blind spot, the authors shift the evaluation focus from coarse trajectory-level dialogues to the fine-grained, step-level critical juncture where ambiguity first blocks execution, enforcing an uncompromised causal verification protocol. Core idea: Introduce MID-Bench, a step-level mobile ambiguity benchmark featuring a strict user simulator and an "Ask+Match" scoring rule that awards credit only when an agent clarifies before acting, alongside train-free confidence and entropy cascade gates that intercept blind execution at zero extra inference cost.
Method¶
Overall Architecture¶
The framework establishes an end-to-end paradigm encompassing rigorous benchmark evaluation and train-free inference-time defense. On the benchmark side, MID-Bench identifies the exact pivot step where underspecification becomes action-critical, rewrites original instructions into four distinct ambiguity categories, and deploys a strict single-round user simulator that answers only sufficiently specific queries. On the defense side, a lightweight cascade wrapper wraps frozen open-weight agents: prior to committing to an action, a plug-and-play gate evaluates internal confidence or semantic dispersion signals to decide whether the instruction is sufficiently determined, routing unambiguous steps directly to execution while redirecting uncertain steps to invoke call_user.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: Mobile Screen State + Ambiguous Instruction"] --> B["Confidence & Semantic Disagreement Gating<br/>Compute Mean Token Log-Prob or Action Entropy"]
B -->|High Confidence / Determined Intent| C["Direct Execution: Baseline GUI Action Output"]
B -->|Low Confidence / Ambiguity Detected| D["Strict Single-Round Simulator Interaction<br/>Issue Specific Query & Receive Minimal Answer"]
D --> E["Instruction-Query-Re-execution Joint Adaptation<br/>Re-evaluate Final GUI Action with Appended Context"]
E --> F["Output: Final Ground-Truth GUI Action Execution"]
Key Designs¶
1. Confidence & Semantic Disagreement Gating: Train-free interception mechanisms bypassing action-generation inertia
Because autoregressive GUI agents are overwhelmingly trained on successful observation-action pairs, their decoders inherit a potent inductive bias toward immediate visual action emission, suppressing verbalized refusal or clarification tools even when explicitly available in the prompt. To break this default execution trajectory without costly retraining, the authors formulate train-free clarification cascades. Three distinct gating signals are systematically operationalized:
- Logit-Gated Cascade (LGC): Directly reuses the forward pass of the base agent by computing the average token log-probability of the predicted action sequence:
$\(S_{\text{LGC}} = \frac{1}{|a|} \sum_{t=1}^{|a|} \log P(y_t \mid y_{<t}, I, S)\)$
When \(S_{\text{LGC}}\) falls below an F1-calibrated threshold \(\tau_{\text{LGC}}\), the gate suppresses the proposed action and unconditionally triggers clarification. Because LGC recycles existing logits, it incurs zero extra forward-pass latency (\(1 + 2\phi\) passes per step, where \(\phi\) is the clarification fire rate).
- Sampling-Gate Cascade (SGC): Draws \(k=5\) stochastic decodes (\(T=0.7\)) and partitions them into coarse action-equivalence classes (e.g., spatial coordinates binned to a 50-pixel grid, normalized text strings, and collapsed swipe gestures). It computes the empirical Shannon entropy \(H = -\sum_i p_i \log p_i\), firing whenever \(H > \tau_{\text{SGC}}\) (where \(\tau_{\text{SGC}}=0\) reduces to a discrete disagreement detector).
- Prompted-Gate Cascade (PGC): Uses a separate verbalized zero-shot prompt querying whether the agent has sufficient information to act confidently.
Empirical results prove that LGC and SGC achieve 78%–99% ambiguity recall across frozen bases while preserving standard clear-instruction accuracy within a negligible margin (\(\le 0.03\) shift).
2. Strict Single-Round Simulator Interaction: Minimally sufficient information exchange for causal disambiguation
Existing agent-environment benchmarks deploy cooperative simulators that reveal full hidden goals even when queried with vague or uninformative prompts (e.g., "What should I do next?"). To isolate genuine ambiguity-aware competence from simulator leakage, MID-Bench enforces a strict single-round interaction protocol. When the agent emits a clarification call call_user(q), the strict simulator evaluates whether query \(q\) maps unambiguously to a single masked entity from the ground-truth instruction. If the question is specific (e.g., "Which book title?"), the simulator returns an ultra-concise answer of 2 to 5 words ("The Queen's Gambit"). Conversely, if the agent produces a generic or lazy query (e.g., "Can you tell me more?"), the simulator deflects with "What specifically do you need to know?". Subsequent emissions of call_user automatically receive zero score, preventing models from exploiting conversational loops to brute-force tasks.
3. Instruction-Query-Re-execution Joint Adaptation: Data-efficient parameter tuning for intrinsic clarification grounding
While train-free cascades excel at stopping blind execution, frozen base agents may still struggle to synthesize the simulator's response and recover the correct post-clarification action (Execute-fail). To instill native clarification capabilities into compact models, the authors formulate a symmetric tri-fold LoRA training recipe. Using 1,496 base trajectories from AndroidControl, each sample is expanded into three parallel instances sharing identical screen states and target actions: (1) original clear instruction mapped to ground-truth action; (2) rewritten ambiguous instruction mapped to call_user paired with a GPT-4o-generated specific question; and (3) ambiguous instruction concatenated with the simulator's minimal answer mapped to the ground-truth action. Training a lightweight adapter (\(r=16, \alpha=32\), 2 epochs on 2 \(\times\) RTX 3090 GPUs in ~2 hours) equips Qwen3-VL-4B with an intrinsic clarification F1 of 0.853 and 99.1% asking precision, compounding synergistically with train-free LGC gates to set a new state of the art on strict ambiguity splits.
Benchmark & Evaluation Protocol¶
MID-Bench provides a standardized, contamination-free evaluation suite built on AndroidControl (AC) and GUI-Odyssey (Ody):
- Splits & Composition: Contains the Base Ambig split (\(n=600\); 400 AC + 200 Ody), the stricter Hard Ambig subset (\(n=300\); 150 AC + 150 Ody), and a matched Clear split (\(n=300\)) of unambiguous reference steps.
- Ambiguity Taxonomy: Instructions are balanced 1:1:1:1 across four formal ambiguity classes: Referential (vague pronouns/names), Underspecified (missing parameters, further subdivided into value, target, and mixed slots), Preference-dependent (subjective user criteria), and Contextual (presupposed unobserved history).
- Pivot-Step & Leakage Elimination: An automated pipeline identifies the precise pivot step where missing information first blocks execution. Trajectories where actions do not depend on missing info (e.g., scrolling, waiting, back navigation) or where rewritten text inadvertently leaks the target string/application name are deterministically purged.
- Hard Masked Token Subset: Enforces strict verification on type actions: the target string is completely excised from the instruction, and an automated verifier confirms that no sub-token of length \(\ge 4\) appears in the prompt, making heuristic guessing nearly impossible.
- Scoring Metrics:
- Match-Only: Standard step-level accuracy crediting any action matching ground truth regardless of whether the agent asked.
- Ask+Match: The primary benchmark metric. A step is credited if and only if the agent emits call_user at the pivot step and subsequently executes the correct ground-truth action after receiving the simulator's response. The gap (\(\text{Match-Only} - \text{Ask+Match}\)) directly measures the rate of Act-without-Asking.
Key Experimental Results¶
Main Results¶
The primary evaluation spans four representative open-weight mobile GUI agents alongside LoRA fine-tuning and the three train-free cascade strategies.
| Model | Method | Clear Acc | Base Match-Only | Base Ask+Match | Hard Match-Only | Hard Ask+Match |
|---|---|---|---|---|---|---|
| Qwen3-VL-4B | Baseline (Frozen) | 0.468 | 0.406 | 0.000 | 0.107 | 0.000 |
| + PGC (Prompted) | 0.461 | 0.430 | 0.134 | 0.223 | 0.150 | |
| + SGC (Sampling) | 0.480 | 0.470 | 0.171 | 0.367 | 0.273 | |
| + LGC (Logit) | 0.489 | 0.476 | 0.407 | 0.447 | 0.423 | |
| GUI-Owl-1.5-4B | Baseline (Frozen) | 0.518 | 0.415 | 0.000 | 0.100 | 0.000 |
| + PGC (Prompted) | 0.529 | 0.447 | 0.150 | 0.267 | 0.203 | |
| + SGC (Sampling) | 0.522 | 0.541 | 0.408 | 0.513 | 0.460 | |
| + LGC (Logit) | 0.508 | 0.537 | 0.478 | 0.467 | 0.397 | |
| UI-TARS-1.5-7B | Baseline (Frozen) | 0.419 | 0.297 | 0.000 | 0.077 | 0.000 |
| + PGC (Prompted) | 0.402 | 0.418 | 0.384 | 0.437 | 0.430 | |
| + SGC (Sampling) | 0.393 | 0.433 | 0.397 | 0.443 | 0.433 | |
| + LGC (Logit) | 0.403 | 0.432 | 0.357 | 0.440 | 0.440 | |
| UI-Venus-Navi-7B | Baseline (Frozen) | 0.466 | 0.370 | 0.000 | 0.097 | 0.000 |
| + PGC (Prompted) | 0.539 | 0.510 | 0.435 | 0.513 | 0.507 | |
| + SGC (Sampling) | 0.466 | 0.389 | 0.051 | 0.127 | 0.037 | |
| + LGC (Logit) | 0.542 | 0.536 | 0.536 | 0.563 | 0.563 | |
| Qwen3-VL-4B (LoRA) | Baseline (Native) | 0.493 | 0.361 | 0.333 | 0.323 | 0.230 |
| + PGC (Prompted) | 0.479 | 0.416 | 0.391 | 0.333 | 0.247 | |
| + SGC (Sampling) | 0.472 | 0.423 | 0.413 | 0.543 | 0.490 | |
| + LGC (Logit) | 0.471 | 0.424 | 0.424 | 0.577 | 0.573 |
Ablation Study & Efficiency Profile¶
The table below details the trade-off between clarification decision quality (should-ask precision, recall, and F1) and real-world computational overhead measured on a dedicated RTX 3090 GPU.
| Model & Gate Config | Asking Prec. | Asking Recall | Should-Ask \(F_1\) | Added Gate Passes | Measured Latency (\(\phi=0.5\)) | Measured Latency (\(\phi=1.0\)) |
|---|---|---|---|---|---|---|
| Qwen3-VL-4B Baseline | 1.000 | 0.001 | 0.002 | 0 (none) | 1.9 ~ 3.0 s | 1.9 ~ 3.0 s |
| Qwen3-VL-4B + PGC | 0.671 | 0.290 | 0.405 | +1 verbal pass (+0.45~0.95s) | 4.7 ~ 6.4 s | 6.7 ~ 9.0 s |
| Qwen3-VL-4B + SGC (\(k=5\)) | 0.828 | 0.390 | 0.530 | +5 sample passes (+5.4~11.0s) | 9.2 ~ 16.6 s | 11.2 ~ 19.2 s |
| Qwen3-VL-4B + LGC | 0.767 | 0.781 | 0.774 | 0 (reuses base logits) | 3.8 ~ 5.6 s | 5.7 ~ 8.2 s |
| GUI-Owl-1.5-4B + LGC | 0.793 | 0.793 | 0.793 | 0 (reuses base logits) | 3.8 ~ 5.6 s | 5.7 ~ 8.2 s |
| UI-TARS-1.5-7B + LGC | 0.775 | 0.884 | 0.826 | 0 (reuses base logits) | 4.1 ~ 5.9 s | 6.1 ~ 8.5 s |
| UI-Venus-Navi-7B + LGC | 0.758 | 0.999 | 0.862 | 0 (reuses base logits) | 4.0 ~ 5.8 s | 6.0 ~ 8.4 s |
| Qwen3-VL-4B (LoRA) | 0.991 | 0.749 | 0.853 | 0 (native policy asking) | 1.9 ~ 3.0 s | 5.8 ~ 8.3 s |
Category-level breakdown highlights notable disparities: - Referential Ambiguity: LoRA alone excels on Base (.570) by learning local reference resolution, but plummets to .167 on Hard. - Preference-dependent Ambiguity: LGC proves remarkably resilient, achieving the highest mark of .586 on Hard. - Underspecified Ambiguity: Persistently forms the hardest failure mode (LoRA Base .227; LGC Hard .383), demonstrating that when action-critical values are absent, agents struggle without acquiring external ground truth.
Key Findings¶
- Conventional Benchmarks Mask Complete Failure: Under lenient Match-Only rules, baseline models post deceptive accuracy scores of 0.30 to 0.42 on Base Ambig. However, under the rigorous Ask+Match rule, all four frozen baselines collapse to an absolute 0.000, confirming that every credited baseline point stems from blind guessing rather than reasoned disambiguation.
- LGC Strikes the Optimal Pareto Efficiency: The logit gate requires no extra forward pass before the decision point, maintaining 3.8–5.6s step latency while reaching top F1 scores across models (0.774–0.862) and elevating Hard Ask+Match up to 0.563 on UI-Venus. In contrast, multi-sample SGC inflates latency by 200%–300% without offering commensurate performance advantages.
- Orthogonal Synergy Between LoRA and Cascade Gates: LoRA provides near-perfect asking precision (0.991) within distribution, while LGC provides out-of-distribution coverage across diverse GUI datasets. Combining LoRA + LGC yields 0.573 Hard Ask+Match, recovering 99% of the theoretical Oracle-Gate ceiling (0.577).
- Closed-Source Models Also Suffer from Act-without-Asking: Probing GPT-5.4 reveals that while it possesses stronger native clarification awareness than open-weight models on Hard (0.360 Ask+Match vs 0.370 Match-Only), it still exhibits substantial blind acting on Base (0.145 Ask+Match vs 0.385 Match-Only), and wrapping it with PGC gates reliably boosts both metrics.
Highlights & Insights¶
- First Systematic Formalization of Act-without-Asking: The paper crystallizes and empirically isolates the failure mode where agents continue executing actions despite possessing internal ambiguity awareness, diagnosing the recognition-action disconnect.
- MID-Bench Sets a Rigorous Standard for GUI Interaction: By constraining evaluation to the exact pivot step, enforcing strict token-mask verification, and pairing with a disciplined minimal-answer simulator, MID-Bench eliminates evaluation contamination.
- Logit Confidence as an Effective Zero-Cost Gate: Demonstrating that token log-probabilities alone can reliably detect instruction underspecification provides mobile AI system designers with an immediate, compute-free safety guardrail.
Limitations & Future Work¶
- Constraint to Single-Round Interactions: The protocol deliberately enforces a single ask-then-act turn to decouple upstream gating from multi-turn dialogue repair; generalizing to complex multi-step clarifications across extended workflows remains open.
- Mobile-Centric Focus: The benchmark focuses on Android OS interfaces; adapting the protocol to desktop environments, multi-window Web browsers, and cross-application workflows requires further research.
- Base-Dependent Calibration of Thresholds: Logit thresholds reflect specific model calibration characteristics and require tuning on a small validation set, although cross-dataset experiments demonstrate robust transfer across app distributions.
Related Work & Insights¶
- vs. AmbiBench / Mobile-World / Fairy: While recent benchmarks introduce conversational actions, they evaluate clarification at the aggregate trajectory level, rewarding agents that guess correctly at critical steps. MID-Bench enforces step-level accountability where unasked actions receive zero credit.
- vs. CLAM / CLAMBER: Purely textual NLP clarification benchmarks omit visual interface affordances; this work demonstrates that visual screen saliency is the primary culprit inducing premature action emission in multimodal agents.
- vs. Navi-Plus / MAI-UI: Existing ask-oriented GUI agents rely heavily on full fine-tuning, which often degrades general task accuracy; this paper shows that train-free logit cascades match or surpass fine-tuned agents while leaving clear-task execution completely intact.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ (Formulates Act-without-Asking, creates a step-level causal benchmark, and demonstrates train-free gating)
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Evaluates 4 open models, closed GPT-5.4, 900 ambiguous steps, full latency models, and error breakdowns)
- Writing Quality: ⭐⭐⭐⭐⭐ (Exceptionally clear narrative structure, rigorous statistical framing, and transparent failure analysis)
- Value: ⭐⭐⭐⭐⭐ (Crucial for mobile GUI agent reliability and safety; LGC provides an immediate, zero-overhead industrial solution)