Are Android GUI Agents Robust Against Runtime Anomalies? AnTrap: Evaluating Agents in Dynamic Adversarial Environments¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/gguogan/AnTrap
Area: LLM Agent
Keywords: Android GUI Agents, Runtime Anomaly Evaluation, Dynamic Adversarial Environments, STAR Taxonomy, Reinforcement Learning Robustness
TL;DR¶
ANTRAP introduces a dynamic adversarial evaluation benchmark covering four layers and ten fine-grained anomaly subcategories (STAR) in Android environments, revealing universal vulnerabilities across 16 leading GUI agents and demonstrating that adversarial GRPO effectively remediates single-step state and action perturbations but fails to resolve multi-step contextual deadlocks.
Background & Motivation¶
Multimodal Large Language Model (MLLM)-based Android GUI agents have recently demonstrated remarkable capabilities in long-sequence device planning and tool manipulation within live operating systems. Contemporary dynamic evaluation benchmarks, such as AndroidLab, AndroidWorld, and MobileWorld, primarily assess overall agent task completion rates under idealized, stable, and clean execution paths. In real-world mobile deployments, however, agents frequently encounter two critical categories of dynamic runtime anomalies: external environmental disturbances, including unexpected advertisement pop-ups, system permission dialogs, UI glitches, and rendering lags; and internal cognitive or execution errors generated by agents themselves, such as visual hallucinations, grounding offsets, and action type mismatches.
While recent research has begun exploring agent robustness, previous approaches remain fundamentally constrained. Several benchmarks evaluate agents only along static, pre-collected single-step trajectories, divorcing evaluation from the dynamic interactive loop; others focus narrowly on passive visual noise or instruction ambiguity, or rely on sparse, uncontrollable simulator failures. Consequently, existing frameworks lack systematic, fine-grained, and controllable anomaly injection across the full interaction life cycle. More critically, prior studies fail to delineate which runtime anomalies can be remedied through interactive environment reinforcement learning and which expose intrinsic reasoning bottlenecks in current foundation models.
To bridge this fundamental gap, there is an urgent need for an adversarial evaluation suite that maintains task solvability while injecting realistic perturbations across diverse interactive stages. Core idea: organize real-world mobile anomalies into the four-layer, ten-subcategory STAR taxonomy across the agent execution loop, construct the dynamic adversarial benchmark ANTRAP, and delineate the boundary between environment-learnable challenges and reasoning-bottlenecked failures via online GRPO.
Method¶
Overall Architecture¶
ANTRAP is built upon an extended interactive AndroidWorld environment, expanding the task suite to 236 diverse mobile tasks with stochastic variations. The core mechanism intercepts the multi-turn GUI execution loop \(\mathcal{R} = (S, T, A)\), where \(S\) denotes the environment-provided screen observation, \(T\) represents the internal reasoning trace of the agent, and \(A\) indicates the dispatched execution action. At execution step \(t\), the central controller invokes the trap toolbox to intercept and perturb the trajectory state \(L_{\text{trap}}\):
The construction and injection process rigorously adheres to two operational principles: dynamic intervention, which stochastically perturbs live rollouts rather than static trajectories; and solvability preservation, which guarantees that the underlying task remains mathematically and operationally feasible so that all observed degradations directly measure the agent's failure to self-recover.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["User Instruction + 236 Android Tasks"] --> B["STAR Taxonomy Framework<br/>State / Thinking / Action / Round"]
B --> C["Dynamic Interception Pipeline<br/>Runtime Injection + Solvability Preservation"]
C --> D["Asymmetric Logging Policy<br/>Direct Updates vs Induced Self-Correction"]
D --> E["GRPO Adversarial RL Training<br/>Differentiating Learnable vs Bottlenecked Tasks"]
E --> F["Fine-Grained Anomaly Diagnostics & Recovery"]
Key Designs¶
1. STAR Four-Layer Dynamic Trap Taxonomy: Systematic Mapping of Deployment Anomalies
ANTRAP formalizes runtime failures into a four-layer taxonomy corresponding to the agent's cognitive and execution cycle:
- State Layer (\(S_{\text{trap}}\)): Simulates abrupt screen changes. Comprises External Interruption (installing a dedicated TrapOverlay APK that listens to ADB broadcasts and triggers runtime permission requests, center dialogs, notification banners, or full-screen ads from a 92-template bank or online VLM generation) and Visual Obscuration (using Pillow to apply Gaussian blur or solid color patches to accessibility tree elements or 10%โ30% screen areas).
- Thinking Layer (\(T_{\text{trap}}\)): Simulates cognitive and temporal misalignment. Comprises Temporal Conflict (substituting the active observation frame with one captured \(N=3\) steps prior or an unrelated app state at decision time) and Visual Hallucination (replacing UI button text with contrast-inverted antonym pairs such as Save/Delete or Allow/Deny to mislead visual reasoning).
- Action Layer (\(A_{\text{trap}}\)): Simulates execution and low-level control errors. Comprises Grounding Error (adding independent random pixel offsets \(\delta_x, \delta_y \in [20, 50]\) pixels to touch coordinates), Type Mismatch (randomly remapping touch actions among click, long press, and double tap, or reversing swipe coordinates), and Intent Deviation (redirecting touch coordinates to adjacent opposite buttons or replacing input text with erroneous strings).
- Round Layer (\(R_{\text{trap}}\)): Simulates multi-step contextual breakdowns. Comprises State Deadlock (silently dropping all ADB commands for \(k \in [1, \lfloor T/4 \rfloor]\) consecutive steps, rendering the UI unresponsive), Context Disruption (injecting KEYCODE_HOME or KEYCODE_APP_SWITCH immediately before step execution), and Loop (injecting periodic KEYCODE_BACK commands every \(N=2\) steps to force repeated backward navigation cycles).
2. Dynamic Interception and Asymmetric Logging Policy: Isolating Environmental Recovery from Self-Correction
ANTRAP instruments existing agents without modifying their internal inference logic. During evaluation, perturbations are triggered with an independent step-wise probability \(p_{\text{trap}} = 0.16\), restricted to the first 80% of the step budget with a maximum cap of \(n_{\text{max}} = 1\) trap event per episode, and the step budget is extended by \(1.15\times\) to allow sufficient recovery leeway. To isolate distinct recovery behaviors, ANTRAP implements an asymmetric history logging policy: - Direct Modifications: For state perturbations and executed action remappings, the actual perturbed observation and executed action are faithfully recorded in the conversation history, evaluating whether the agent can perceive physical environmental shifts and replan accordingly. - Induced Modifications: For thinking traps (transient observation corruption) and action suppressions, the misleading observation or interception hook is excluded from the persistent log. Only the agent's misled response is preserved in history. The agent must therefore navigate subsequent steps burdened by its own hallucinated context, strictly probing internal self-reflection and self-correction rather than mere noise tolerance.
3. Online Adversarial GRPO Exploration: Delineating Environment-Learnable and Reasoning-Bottlenecked Tasks
To determine whether runtime anomaly recovery can be acquired through reinforcement learning, the authors conduct full-parameter Group Relative Policy Optimization (GRPO). For each task, the agent samples a group of \(G=8\) rollouts and evaluates outcomes using rule-based binary terminal rewards \(r_i \in \{0, 1\}\) without an auxiliary critic network:
Here, the advantage is calculated via group reward normalization \(A_i = \frac{r_i - \text{mean}(r)}{\text{std}(r)}\), and \(\rho_{i,t}(\theta) = \frac{\pi_\theta(o_i(t)|o_{i,<t})}{\pi_{\text{old}}(o_i(t)|o_{i,<t})}\) represents the token-level importance ratio. By training agents in clean environments (+Orig) and dedicated subcategory-specific adversarial environments (+Trap), the study reveals a fundamental dichotomy: single-step state and action perturbations provide sharp gradient signals that RL readily exploits, whereas multi-step contextual deadlocks require long-horizon meta-cognitive self-monitoring that cannot be resolved through environment-level RL rollouts alone.
Loss & Training¶
Full-parameter GRPO is conducted using the VERL framework across 2 nodes comprising 16 NVIDIA A100 (80GB) GPUs with FSDP2 and BF16 mixed precision. Rollouts are accelerated using vLLM. Training hyperparameters include a global batch size of 8, micro batch size per GPU of 1, learning rate of \(1 \times 10^{-6}\), gradient clipping norm of 1.0, and PPO clipping ratio \(\epsilon = 0.2\). Rollout sampling uses temperature 1.0, maximum prompt length of 32,768 tokens, maximum response length of 512 tokens, and a per-episode budget cap of 50 interaction turns.
Key Experimental Results¶
Main Results¶
The benchmark evaluates 16 multimodal GUI agents from 7 organizations across the 236 tasks. The Pass@3 success rates (%) across the clean environment (Orig) and the ten trap subcategories are detailed below:
| Model Type | Model | Orig | Ext.I. | Vis.O. | Tmp.C. | Vis.H. | Grd.E. | Typ.M. | Int.D. | St.DL. | Ctx.D. | Loop | Avg |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Human Baseline | Human Annotator | 94.1 | 94.1 | 92.4 | - | - | 94.1 | 93.6 | - | 93.2 | 92.8 | - | 93.4 |
| Proprietary | Claude-Sonnet-4.6 | 74.2 | 64.8 | 66.1 | 66.9 | 72.0 | 69.9 | 67.8 | 69.5 | 63.1 | 63.1 | 61.9 | 66.5 |
| Proprietary | Gemini-3-Pro | 72.9 | 61.9 | 68.6 | 60.2 | 62.7 | 68.6 | 60.6 | 66.1 | 63.6 | 61.4 | 62.3 | 63.6 |
| Proprietary | GPT-5.4 | 65.3 | 55.1 | 53.8 | 57.6 | 64.8 | - | 61.9 | 61.9 | 50.0 | 52.5 | 51.7 | 56.6 |
| Proprietary | GPT-5.4-Mini | 42.8 | 31.4 | 25.0 | 40.3 | 29.7 | - | 29.2 | 33.1 | 31.8 | 34.7 | 39.4 | 32.7 |
| Open Instruct | GUI-Owl-1.5-32B-Instruct | 68.2 | 58.1 | 62.7 | 57.2 | 65.3 | 63.1 | 64.0 | 66.1 | 63.6 | 62.7 | 61.0 | 62.4 |
| Open Instruct | GUI-Owl-1.5-8B-Instruct | 64.8 | 55.1 | 66.9 | 58.1 | 65.3 | 65.7 | 63.1 | 60.2 | 55.1 | 57.2 | 58.5 | 60.5 |
| Open Instruct | Qwen3-VL-4B-Instruct | 54.7 | 37.7 | 52.1 | 47.9 | 31.4 | 50.8 | 50.8 | 49.2 | 42.8 | 43.2 | 52.1 | 45.8 |
| Open Instruct | Qwen3-VL-8B-Instruct | 52.1 | 38.1 | 53.8 | 41.5 | 47.0 | 49.6 | 50.0 | 48.3 | 41.5 | 45.3 | 47.9 | 46.3 |
| Open Thinking | GUI-Owl-1.5-32B-Think | 69.5 | 60.2 | 67.8 | 52.5 | 67.4 | 67.4 | 65.7 | 65.3 | 55.5 | 54.2 | 68.2 | 62.4 |
| Open Thinking | GUI-Owl-1.5-8B-Think | 66.1 | 53.0 | 66.5 | 58.5 | 60.2 | 67.4 | 62.3 | 63.6 | 63.1 | 57.2 | 65.3 | 61.7 |
| Open Thinking | GUI-Owl-7B | 63.1 | 43.2 | 57.6 | 40.7 | 58.9 | 59.3 | 55.5 | 57.2 | 42.4 | 26.3 | 35.6 | 47.7 |
| Open Thinking | MAI-UI-2B | 33.5 | 27.5 | 31.8 | 26.7 | 34.3 | 30.9 | 26.3 | 27.5 | 29.2 | 25.8 | 30.1 | 29.0 |
| Open Thinking | MAI-UI-8B | 53.4 | 40.3 | 51.7 | 47.0 | 54.2 | 52.5 | 50.0 | 51.3 | 50.0 | 44.5 | 45.8 | 48.7 |
| Open Thinking | Qwen3-VL-4B-Thinking | 57.2 | 45.3 | 50.0 | 41.1 | 50.0 | 50.8 | 53.8 | 55.5 | 47.9 | 43.2 | 46.6 | 48.4 |
| Open Thinking | Qwen3-VL-8B-Thinking | 62.7 | 59.3 | 61.0 | 48.7 | 61.9 | 64.4 | 61.0 | 60.2 | 54.2 | 58.9 | 36.9 | 56.6 |
| Open Thinking | UI-TARS-1.5-7B | 29.7 | 28.0 | 27.1 | 21.6 | 22.9 | 17.8 | 21.2 | 21.2 | 23.7 | 13.6 | 20.8 | 21.8 |
Ablation Study¶
The table below compares the performance of GUI-Owl-7B and UI-TARS-7B across three training regimes: baseline (Base), clean environment training (+Orig), and subcategory-specific adversarial training (+Trap):
| Layer | Subcategory | GUI-Owl Base | GUI-Owl +Orig | GUI-Owl +Trap | UI-TARS Base | UI-TARS +Orig | UI-TARS +Trap | Note |
|---|---|---|---|---|---|---|---|---|
| - | Original Task | 63.1 | 69.9 | 69.5 | 29.7 | 36.0 | 35.6 | Both RL regimes improve clean tasks by >5.9% |
| S | Ext. Interruption | 43.2 | 45.3 | 54.2 | 28.0 | 29.2 | 36.4 | Adversarial RL substantially boosts recovery (+11.0% / +8.4%) |
| S | Vis. Obscuration | 57.6 | 58.9 | 66.5 | 27.1 | 28.4 | 35.2 | Robust gains exceeding 8% in S-layer |
| T | Tmp. Conflict | 40.7 | 41.9 | 45.8 | 21.6 | 22.5 | 25.8 | Moderate gains on temporal reasoning (+5.1% / +4.2%) |
| T | Vis. Hallucination | 58.9 | 59.7 | 64.0 | 22.9 | 23.7 | 27.1 | Cognitive hallucination recovery improves moderately |
| A | Grounding Err. | 59.3 | 62.7 | 67.4 | 17.8 | 20.8 | 25.4 | Single-step execution offset is readily addressed |
| A | Type Mismatch | 55.5 | 58.5 | 64.0 | 21.2 | 24.2 | 28.8 | Significant recovery gains under action remapping (+8.5% / +7.6%) |
| A | Int. Deviation | 57.2 | 55.5 | 60.6 | 21.2 | 21.6 | 26.7 | Moderate gains when semantic deviations occur |
| R | State Deadlock | 42.4 | 42.8 | 44.9 | 23.7 | 24.6 | 26.3 | Multi-step unresponsiveness resists RL (<3% improvement) |
| R | Ctx. Disruption | 26.3 | 27.5 | 29.2 | 13.6 | 14.4 | 15.7 | Minimal recovery from sudden app exit/switch |
| R | Loop | 35.6 | 36.0 | 36.4 | 20.8 | 21.2 | 21.6 | Backward navigation loops show negligible improvement (<1%) |
Key Findings¶
- Universal Fragility Across Agent Architectures: Every evaluated model exhibits significant degradation under dynamic traps. The strongest proprietary model, Claude-Sonnet-4.6, drops from 74.2% to 66.5%, and top open-source model GUI-Owl-1.5-32B-Think declines from 69.5% to 62.4%, proving that current agents are heavily overfitted to clean execution paths.
- Contextual Traps Pose Severe Bottlenecks: Performance degrades most dramatically under Round-layer disruptions, particularly State Deadlock and Context Disruption. While agents tolerate immediate single-step disturbances, they struggle to recover when errors disrupt multi-step contextual flow, often looping repeatedly or hallucinating completion.
- Reasoning Capabilities Do Not Equate to Robustness: Thinking models establish higher baseline scores on clean tasks (e.g., Qwen3-VL-8B-Thinking achieves 62.7% vs 52.1% for Instruct), but their absolute degradation under adversarial traps matches or exceeds Instruct models (6.1-point drop vs 5.8-point drop).
- Environment-Learnable vs. Reasoning-Bottlenecked Dichotomy: Adversarial GRPO (+Trap) yields substantial improvements on S-layer (+8.1%โ11.0%) and A-layer (+3.4%โ8.5%) perturbations. However, gains diminish on T-layer (+4.2%โ5.1%) and collapse on R-layer (<3.0%, with Loop <1.0%), demonstrating that while single-step errors are environment-learnable, long-horizon deadlocks represent an architectural reasoning bottleneck.
Highlights & Insights¶
- Full Life-Cycle STAR Anomaly Abstraction: Abstracting real mobile crashes into state, thinking, action, and round layers with native ADB/accessibility tree injection scripts establishes an essential stress-testing framework for practical mobile automation.
- Induced Modification Protocol: Selectively omitting deceptive observation frames from the persistent conversation log while preserving the agent's misled reasoning traces rigorously isolates autonomous self-correction from passive noise tolerance.
- Empirical Boundary of Adversarial RL: The comparative GRPO study demonstrates that while adversarial training enhances single-step action/state resilience without compromising clean execution, resolving state deadlocks demands higher-order meta-cognitive monitoring.
Limitations & Future Work¶
- Base Task Scale: Due to the labor-intensive requirements of manual validation and deterministic evaluators, the benchmark comprises 236 base tasks, which may not capture all long-tail global Android scenarios.
- Absence of Adversarial Supervised Fine-Tuning: Due to the prohibitive cost of collecting expert multi-step recovery trajectories, the study focuses exclusively on online RL without investigating the impact of adversarial SFT.
- Diagnostic Focus Over Algorithmic Defense: ANTRAP is primarily designed as an evaluative diagnostic suite; developing scalable architectural defenses and meta-cognitive monitoring mechanisms remains an open challenge.
Related Work & Insights¶
- vs AndroidWorld / AndroidLab: Prior dynamic benchmarks focus on single-path execution in benign OS environments; ANTRAP significantly expands task variety and introduces systematic runtime perturbation injection.
- vs SMAN-Bench / ScreenSpot: SMAN-Bench and ScreenSpot rely on static trajectories or focus on passive visual noise; ANTRAP provides an interactive online environment covering single-step and multi-step state deadlocks.
- vs GUI-Reflection / Agent-R: Existing self-correction methods collect unstructured, random error traces for fine-tuning; ANTRAP offers a controlled, fine-grained taxonomy that isolates which failure modes require explicit architectural intervention.
Rating¶
- Novelty: โญโญโญโญโญ Establishes the first systematic STAR dynamic anomaly taxonomy and controllable injection pipeline for Android agents.
- Experimental Thoroughness: โญโญโญโญโญ Evaluates 16 multimodal models and delivers rigorous comparative GRPO experiments across clean and adversarial regimes.
- Writing Quality: โญโญโญโญโญ Precise mathematical formulation, clear taxonomic structure, and comprehensive empirical analyses.
- Value: โญโญโญโญโญ Provides crucial diagnostic benchmarks and insights for transitioning GUI agents from benign sandboxes to robust industrial deployment.