MobileGym: A Verifiable and Highly Parallel Simulation Environment for Mobile GUI Agents¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://mobilegym.github.io
Area: LLM Agent
Keywords: Mobile GUI Agent, Verifiable Simulation, Scalable Reinforcement Learning, Sim-to-Real Transfer, State Verification
TL;DR¶
MobileGym is a browser-hosted, lightweight Android-like simulation platform that models app ecosystems into fully controllable structured JSON states, enabling millisecond resets, zero-consequence operations, and single-server scaling to hundreds of parallel instances for online RL, where GRPO policies retain 95.1% of simulation gains on physical smartphones.
Background & Motivation¶
Mobile GUI agents powered by Vision-Language Models (VLMs) have demonstrated remarkable promise in operating smartphones directly from pixel observations and natural language instructions. However, the continuous progression of these agents is bottlenecked by two core pillars: reliable and verifiable outcome signals, and highly parallel, scalable online reinforcement learning (RL). Current research environments are fundamentally divided by an intrinsic infrastructure dilemma: emulator-based platforms (such as AndroidWorld and AndroidLab) offer repeatable evaluations but are burdened by heavyweight Android OS system images and multi-gigabyte virtual machines, while physical-device benchmarks (such as MobileBench-OL) reach real-world everyday apps but suffer from live backend shifts, account bans, irreversible side effects, and prohibitive hardware costs that preclude large-scale exploration.
The core tension underlying these approaches stems from the unreadable, unwritable, unforkable, and irreversible nature of everyday mobile applications. Modern mobile apps encapsulate crucial business stateโsuch as order fulfillment, balances, and chat historiesโinside proprietary backends and runtime caches that cannot be inspected reliably through adb or accessibility trees, forcing prior evaluations to resort to brittle VLM-as-a-judge approaches over discrete screenshots. Furthermore, reproducible evaluation and online policy optimization algorithms (notably group-based methods like GRPO) strictly necessitate instantiating and branching multiple parallel rollouts from identical starting conditions, an operation virtually impossible on live platforms without risking unauthorized money transfers, live messaging, or irrecoverable account mutations.
This paper cuts through this impasse by identifying the essential operational requirement for GUI agent learning: because agents only perceive visual screenshots and dispatch discrete touch actions, a simulation environment does not need to reproduce complex proprietary server backends or pixel-level OS kernelsโit only requires interaction fidelity. MobileGym constructs a browser-hosted, highly reactive Android simulation environment where external world data, per-environment mutable overlays, and OS lifecycle semantics are unified into explicit, structured JSON state. Core idea: construct a browser-hosted, lightweight mobile simulation platform targeting interaction fidelity over proprietary backends, turning everyday mobile app state into readable, writable, forkable, and consequence-free JSON structures to deliver deterministic state verification and scalable online RL.
Method¶
Overall Architecture¶
The system architecture of MobileGym decouples visual presentation from state dynamics, operating directly within modern browser runtimes via React, Zustand, and Vite. The platform consists of four unified tiers: the standardized application and system ecosystem, an OS lifecycle and inter-app communication substrate (Activity stack, BackDispatcher priority chain, Intent routing, and ContentProvider data stores), an agent interaction layer that standardizes screen observations and 17 normalized GUI actions via Playwright, and a programmatic evaluation and RL reward engine driven by exact state diffs and the AnswerSheet protocol.
At the state management core, MobileGym introduces a three-layer Composition Model: a read-only, high-capacity World Data layer storing over 190K public entities and 350K structured records across 12 mainstream apps; an environment-isolated Runtime Overlay tracking all mutable agent-induced mutations (e.g., drafts, shopping carts, modified settings); and an OS Runtime governing system flags and lifecycle stacks. The user-visible UI is dynamically resolved by overlaying mutable state onto world entities, keeping mutable snapshots compact (~400 MB RAM per instance and ~50 MB core code) and enabling sub-millisecond serialization, instant state restoration, and 3-second cold boots on commodity server hardware.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Parameterized Task Templates<br/>Instruction sampling & Slot binding"] --> B["Three-Layer Decomposed State Model<br/>World Data + Runtime Overlay + OS Runtime"]
B --> C["High-Fidelity Inter-App & Lifecycle Subsystem<br/>Activity Stack / BackDispatcher / Intent Routing"]
C --> D["Agent Visual Interaction & Action Execution<br/>Screenshots + 17 Normalized Action Abstractions"]
D --> E["Typed AnswerSheet Protocol<br/>GUI Form Submission Replacing Free-Text Matching"]
E --> F["Deterministic Full Environment State Verification<br/>State Diff / Unexpected Side Effect / Dense RL Reward"]
Key Designs¶
1. Three-Layer Decomposed State Model: Eliminating Emulator Bloat and State Opacity
Traditional Android virtual machines require 4.5โ6.0 GB of host RAM per instance and cumbersome filesystem-level snapshotting, preventing fast state branching and parallel trajectory rollouts. MobileGym partitions the environment state into three orthogonal tiers: static World Data containing rich public corpora (catalogs, social posts, train schedules); a compact, per-instance Runtime Overlay that captures strictly agent-induced mutations; and an OS Runtime store managing device configurations (airplane mode cascades, volume, battery). Because all stores are registered through a unified factory, the complete mutable environment state can be snapshotted, reset, or cloned via simple JSON operations (setState). This enables group-based online RL algorithms such as GRPO to fork multiple rollouts from identical initial states within milliseconds, completely sandboxed from irreversible real-world impacts.
2. High-Fidelity Inter-App and Lifecycle Subsystem: Preserving Real-World Mobile Mechanics
To ensure policies generalize to physical Android devices, MobileGym replicates essential Android OS primitives within the browser runtime rather than treating apps as isolated web pages. Its TaskManager mirrors Android's ActivityTaskManager using a Reducer pattern; backgrounded tasks are hidden via CSS display:none rather than unmounted from React, preserving component state trees so that cross-app workflows (such as copying a bank account from WeChat to Alipay and returning to a half-typed draft) mirror real device persistence. Furthermore, BackDispatcher enforces an authentic priority cascade: permission dialogs (1000) > system notification shade (800) > soft keyboard (700) > app navigation stack (100) > return to launcher (0), guarded by frame-level deduplication locks to prevent touch event double-firing. Inter-app data flows are seamlessly governed by an IntentResolver supporting startActivityForResult and standard ContentProviders for Contacts, SMS, and Gallery.
3. Typed AnswerSheet Protocol: Overcoming Brittle Free-Text Information Verification
Query and information extraction tasks in prior GUI benchmarks rely heavily on fuzzy string matching or LLM verifiers on free-text outputs, frequently causing false rejections on equivalent expressions (e.g., "34ยฐC" vs. "34 degrees Celsius") or false acceptances when leaked reasoning text happens to contain the target token. MobileGym resolves this by introducing the AnswerSheet system application, turning answer submission into a native GUI form-filling interaction. Query tasks terminate with the agent navigating to the AnswerSheet to populate typed fields (Choice, Number with floating-point tolerance, Date/Time, and Multi-value lists) featuring explicit format hints. By validating agent entries against strict type-specific matchers, evaluation accuracy becomes deterministic, while an extra 15-step budget is granted to compensate for the form navigation overhead.
4. Deterministic Full Environment State Verification: Unbiased Judging and Dense RL Signals
Departing from unreliable VLM judges that inspect only terminal screenshots, MobileGym executes a complete state diff between the initial and terminal JSON snapshots of the environment. In addition to assessing explicit goal criteria (e.g., checking if order.status == "created"), this global inspection introduces the Unexpected Side Effects (USE) diagnostic, systematically exposing unintended mutations (such as deleting unrelated messages or altering device system settings). For online RL training, the verifier transforms subtask satisfaction into dense progress rewards \(p \in [0, 1]\) while penalizing side effects, false completions, post-success aborts, and overdue timeouts with multiplicative discount factors, yielding an uncheatable, zero-cost reward function.
Loss & Training¶
In the Sim-to-Real transfer experiment, the authors fine-tune Qwen3-VL-4B-Instruct using Group Relative Policy Optimization (GRPO) on 160 training tasks from MobileGym-Bench across 96 parallel browser instances hosted on a single machine with 3 RTX Pro 6000 GPUs.
The rollout reward is computed programmatically from the structured state artifacts. Let \(p \in [0, 1]\) denote the fraction of goal checks satisfied (Progress Rate). For tasks featuring an AnswerSheet, if any field is submitted incorrectly, bookkeeping points are stripped to yield an adjusted progress \(p'\). Multiplicative penalties are then compounded based on environmental cleanliness and termination behavior:
where \(\mathbb{I}[\cdot]\) is the indicator function, and \(\text{clean}\) verifies the total absence of unexpected side effects across all registered stores. Optimization employs a learning rate \(\eta = 10^{-6}\), group rollout size \(k = 8\), batch size \(bs = 12\), KL penalty coefficient \(0.01\), and asymmetric clip-higher bounds of \(0.2/0.28\). The entire policy adapts within just 10 training steps.
Key Experimental Results¶
Main Results¶
Evaluations across 9 frontier proprietary and open-source models on the 256-task MobileGym-Bench test set demonstrate a wide, non-saturating performance spectrum spanning 9.4% to 58.8% Success Rate (SR). Tasks are stratified into four empirically calibrated difficulty tiers (L1 to L4) using eight reference models.
| Category / Model | Overall SR (%) | Overall PR (%) | L1 SR (20) | L2 SR (73) | L3 SR (83) | L4 SR (80) | FC (%) | OT (%) | USE (%) |
|---|---|---|---|---|---|---|---|---|---|
| Proprietary Frontier | |||||||||
| Gemini 3.1 Pro | 58.8 ยฑ1.4 | 72.1 | 97.5 | 83.6 | 63.3 | 21.9 | 34.0 | 0.2 | 5.5 |
| Doubao-Seed-2.0-Pro | 52.0 | 63.6 | 100.0 | 93.2 | 48.2 | 6.2 | 33.6 | 0.4 | 4.7 |
| Qwen3.6-Plus | 45.7 | 59.2 | 100.0 | 78.1 | 44.6 | 3.8 | 34.0 | 0.4 | 14.5 |
| Open-Source GUI Specialists | |||||||||
| AutoGLM-Phone-9B | 20.0 ยฑ1.3 | 35.3 | 86.2 | 33.6 | 9.6 | 1.9 | 39.6 | 0.6 | 12.6 |
| UI-Venus-1.5-8B | 15.4 ยฑ2.4 | 28.3 | 85.0 | 21.9 | 6.0 | 1.9 | 22.9 | 0.5 | 7.7 |
| GUI-Owl-1.5-8B-Think | 15.1 ยฑ0.9 | 28.8 | 76.2 | 26.0 | 4.2 | 1.2 | 30.4 | 0.9 | 14.1 |
| UI-TARS-1.5-7B | 13.8 ยฑ1.7 | 26.3 | 77.5 | 21.9 | 3.0 | 1.6 | 38.6 | 0.2 | 11.0 |
| Step-GUI-4B | 12.9 ยฑ1.1 | 25.7 | 83.8 | 17.8 | 2.4 | 1.6 | 37.0 | 0.8 | 7.6 |
| Open-Source Generalist | |||||||||
| Qwen3-VL-4B-Instruct | 9.4 ยฑ0.6 | 20.1 | 71.2 | 12.3 | 0.6 | 0.3 | 15.9 | 0.4 | 10.0 |
Sim-to-Real Transfer and Operational Efficiency¶
To rigorously test whether simulated online RL transfers to physical hardware, the authors conducted live trials on a physical Redmi Note 12 Turbo smartphone across a curated 59-task signal subset (spanning Uplift, Stable-pass, and Mid performance categories), complemented by 15 Stable-fail tasks as negative controls.
| Evaluation Slice / Benchmark Subsets | Sim Base Policy | Sim RL-Trained Policy | Real Device Base | Real Device Trained | Retained Training Gain |
|---|---|---|---|---|---|
| 59-Task Signal Subset SR (%) | 33.9% | 76.7% (+42.8 pt) | 32.2% | 72.9% (+40.7 pt) | 95.1% |
| โณ Uplift Bucket (n=23) | 17.4% | 80.7% (+63.3 pt) | 2.2% | 73.9% (+71.7 pt) | 113.3% |
| โณ Stable-pass Bucket (n=18) | 95.8% | 95.8% (+0.0 pt) | 94.4% | 94.4% (+0.0 pt) | 100.0% |
| โณ Mid Bucket (n=18) | 12.5% | 61.1% (+48.6 pt) | 22.2% | 50.0% (+27.8 pt) | 57.2% |
| 15-Task Stable-fail Negative Control | 0.0% (0/15) | 0.0% (0/15) | 0.0% (0/15) | 0.0% (0/15) | 100% Agreement |
| Full 256-Task Test Set (Sim) SR (%) | 9.4% ยฑ0.6 | 22.2% ยฑ1.2 (+12.8 pt) | โ | โ | Surpasses 9B Specialists |
In terms of computational efficiency, MobileGym uses ~400 MB RAM and ~50 MB disk footprint per instance with a 3-second cold start, allowing 256 concurrent instances on a single machine (<10% CPU, ~100 GB RAM) to evaluate the entire 256-task test suite in 6 minutes. Conversely, running 512 Android emulator instances on MAI-UI required 10 cloud bare-metal servers (960 vCPUs, 3,840 GB RAM total), demonstrating MobileGym's orders-of-magnitude advantage.
Key Findings¶
- High-Fidelity Behavioral Transfer: The +42.8 pt improvement achieved in simulation transferred to a +40.7 pt gain on physical phones, retaining 95.1% of the policy advantage. Absolute sim-vs-real gaps remained under 4 pt (1.7 pt base, 3.8 pt trained), proving that lightweight interaction-fidelity modeling successfully teaches generalizable device manipulation.
- Autonomous Error Recovery Under OOD Gates: On a real Reddit posting task where community rules enforced an unpredicted flair requirement before enabling the submission button, the base model repeatedly tapped the disabled button for 50 steps until timeout. In contrast, the RL-trained model visually identified that the button was grayed out, attended to the asterisk on the "Add tags & flair" pill, navigated to select the "Tech" tag, and successfully published in 22 steps.
- Systematic Misjudgment in VLM Judges: Manual audit of 118 real-device trajectories revealed that Qwen3.6-Plus misjudged 10.2% of episodes; a cross-check with GPT-5.4 independently confirmed an identical 10.2% error rate. This highlights the vulnerability of VLM-based evaluation and underscores the value of code-native programmatic verification.
Highlights & Insights¶
- Interaction Fidelity over Backend Emulation: The core conceptual triumph is recognizing that GUI agents only interface with visual surfaces; providing a responsive, lightweight browser-based mock of mobile apps preserves behavioral realism while cutting memory footprint by ~90% and disk overhead by ~99%.
- AnswerSheet as a Strict Evaluation Gate: Re-framing query outputs into UI form submissions eliminates both string-matching false negatives and reasoning-leak false positives, delivering a clean, conservative lower bound on model comprehension.
- Whole-Environment Mutation Auditing: Tracking the difference across all registered runtime stores reveals Unexpected Side Effects (USE), surfacing critical safety vulnerabilities where high-performing models silently mutate off-target user data.
Limitations & Future Work¶
- Discrepancy in Dynamic Visual Rendering: Because the environment is re-implemented in frontend frameworks, subtle animation transitions, complex video streaming rendering, and proprietary font icons diverge slightly from physical OEM skins.
- Absence of Server-Side Adversarial Chaos: The simulation does not natively model stochastic backend perturbations such as live CAPTCHA challenges, anti-fraud device checks, or dynamic ad popups unless explicitly parameterized into JSON state.
- Divergence Between Capability and Safety Alignment: Frontier models (e.g., Gemini 3.1 Pro achieving 64.3%โ71.4% on high-consequence payment and account-deletion tasks) execute irreversible actions without intrinsic caution gating; future work should leverage MobileGym to train refusal boundaries.
Related Work & Insights¶
- vs AndroidWorld / AndroidLab: These systems mandate heavyweight AVD images consuming 4.5โ6.0 GB RAM per instance and taking 78+ seconds to boot, restricting RL scaling. MobileGym reduces resource costs by 10ร and provides instant state serialization for zero-cost online rollouts.
- vs MobileBench-OL: While MobileBench-OL runs directly on physical Android phones, it faces account limits, data leakage, and irreversible side effects. MobileGym provides a reproducible, consequence-free sandbox with millisecond reset capabilities.
- vs GUI-Genesis / ViMo: Image-generation world models suffer from visual drift over extended interaction horizons. MobileGym utilizes deterministic state machines and modular business code, guaranteeing mathematically stable state transitions.
Rating¶
- Novelty: โญโญโญโญโญ Pioneered a lightweight, interaction-fidelity browser simulation paradigm that challenges the prevailing assumption that GUI agents require native Android emulators.
- Experimental Thoroughness: โญโญโญโญโญ Implemented 416 tasks across 28 apps, paired with extensive real-device transfer studies, trajectory length audits, and judge-error analyses.
- Writing Quality: โญโญโญโญโญ Rigorously structured with crisp system design rationales, clear mathematical formulations, and exhaustive appendix documentation.
- Value: โญโญโญโญโญ Drastically democratizes online reinforcement learning for mobile agents by removing the multi-node infrastructure barrier.