Skip to content

SecAgent: Efficient Mobile GUI Agent with Semantic Context

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://huggingface.co/datasets/alibabagroup/CMGUI
Area: LLM Agent
Keywords: Mobile GUI Agent, Semantic Context, Multimodal Large Language Model, CMGUI Benchmark, Reinforcement Fine-Tuning

TL;DR

SecAgent tackles non-English data scarcity and visual history redundancy in mobile GUI automation by constructing CMGUI—a human-verified Chinese mobile dataset with 121k navigation steps and multi-choice action benchmark CMGUI-Bench—and proposing an autoregressive semantic context mechanism that distills multi-step visual trajectories into compact natural language summaries, enabling a 3B-scale model to outperform same-size baselines and match 7B–8B models.

Background & Motivation

Multimodal Large Language Model (MLLM)-powered mobile Graphical User Interface (GUI) agents have revolutionized automated device control and task execution on smart devices. Early text-centric systems relied primarily on structured accessibility trees; however, they consistently suffered from platform fragmentation, incomplete UI metadata, and deeply nested hierarchical inconsistencies. While recent research has transitioned toward pure vision-based systems that interact directly with raw screen images, existing open-source benchmarks and training corpora remain overwhelmingly dominated by the English mobile ecosystem. Non-English ecosystems—particularly the Chinese mobile landscape, characterized by complex super-apps, unique interaction workflows, and non-standard layout paradigms—suffer from an acute lack of open, high-quality, human-verified trajectory data, severely hindering the development of truly versatile multilingual GUI agents.

In complex multi-step mobile navigation, selecting the correct action at any given stage inherently depends on the preceding sequence of actions and visual transitions. Existing systems face an acute dilemma regarding history representation: concatenating full historical screenshots (\(N \ge 5\)) alongside executed actions results in a rapid explosion of multimodal visual tokens, driving up training costs, memory usage, and time-to-first-token (TTFT) latency to levels unacceptable for real-time edge or on-device mobile execution. Conversely, aggressively truncating the history window or discarding visual feedback entirely in favor of flat action sequences strips the agent of crucial operational context, frequently triggering looping behaviors, duplicate clicks, and premature task termination.

To bridge this data deficit while resolving the fundamental tension between context completeness and computational efficiency, this work investigates an integrated data-and-architecture methodology. Rather than outsourcing history summarization to a costly auxiliary agent or appending reflective summaries post-action, the authors design an end-to-end framework where a single compact model performs incremental textual context distillation prior to action execution. Core idea: distill multi-step interaction history into a dynamically updated natural language "semantic context," enabling the agent to maintain full historical awareness with only a single historical screenshot-action pair while boosting 3B-scale reasoning and task success via unified coordinate space alignment and rule-guided GRPO reinforcement fine-tuning.

Method

Overall Architecture

SecAgent is formulated as an efficient, unified mobile agent policy operating over high-level user instruction \(I\), current screen screenshot \(X_t\), previous step screenshot \(X_{t-1}\), previous action \(A_{t-1}\), and accumulated historical semantic context \(\mathcal{C}_{t-1}\). In a single forward pass, the model autoregressively generates an updated semantic context \(\mathcal{C}_t\), a rationalized thought chain \(H_t\), and the final executable structured action \(A_t\).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Inputs<br/>Instruction I + Screen Xt + History Xt-1, At-1, Ct-1"] --> B["Semantic Context Mechanism<br/>Single-step History Distillation + Incremental NL Summary"]
    B --> C["Autoregressive Chain-of-Thought Generation<br/>Updated Semantic Context Ct + Local Intent Thought Ht"]
    C --> D["Unified Grounding and Action Space Alignment<br/>Bbox Center-point Projection + Single-point Action At"]
    D --> E["Two-Stage Training & Rule-Guided Policy Optimization<br/>Unified Format SFT + GRPO Format and Action Dual Rewards"]
    E --> F["Executable Mobile Action Output<br/>CLICK / SWIPE / TYPE / TERMINATE"]

Key Designs

1. Semantic Context Mechanism: Efficient Natural Language History Distillation for Long-Horizon Planning To overcome the severe computational bottleneck caused by stacking multiple raw historical screenshots (\(N \ge 5\))—which causes dramatic visual token inflation and doubles inference latency—SecAgent introduces an incremental natural language semantic context mechanism. At each navigation step \(t\), rather than digesting a deep buffer of past image embeddings, the model maintains a concise, sequentially updated textual log \(\mathcal{C}_t\) summarizing verified previous accomplishments (e.g., "1. Clicked the Taobao icon on the home screen; 2. Clicked the search bar to activate text input"). This formulation compresses hundreds of visual tokens into a handful of semantically rich text tokens while allowing the model to operate effectively with a minimal historical visual footprint of only \(N=1\) (the immediate predecessor screenshot \(X_{t-1}\) and action \(A_{t-1}\)). Unlike multi-agent pipelines (such as Mobile-Agent-v2) that require a separate summarization model, SecAgent unifies summarization, reasoning, and acting within a single lightweight network. Furthermore, unlike post-hoc summary frameworks (such as GUI-Rise), SecAgent executes in a forward causal order—"summary \(\to\) thought \(\to\) action"—ensuring that operational decisions are firmly anchored in freshly updated state awareness.

2. Unified Grounding and Action Space Alignment: Harmonizing Cross-Task Inconsistencies for Spatial Transfer Training effective mobile agents requires combining UI grounding datasets (element localization) with long-horizon navigation datasets (action trajectory planning). However, these two tasks exhibit an inherent structural mismatch in output targets: standard grounding tasks expect bounding box coordinates \([y_{\min}, x_{\min}, y_{\max}, x_{\max}]\), whereas real-world navigation tasks demand a single click coordinate \([x, y]\). Directly mixing these incompatible label spaces induces policy conflict and sub-optimal spatial transfer. SecAgent eliminates this discrepancy by transforming all bounding-box annotations into navigation-compatible format (termed Gr2Nav) prior to supervised fine-tuning. For any interactive candidate element, its target coordinate is projected to the geometric bounding box center: $\(x_c = \frac{x_{\min} + x_{\max}}{2}, \quad y_c = \frac{y_{\min} + y_{\max}}{2}\)$ By unifying element localization into identical single-point navigation semantics, the agent seamlessly transfers low-level visual grounding representations into high-level action prediction without representational friction.

3. Two-Stage Training & Rule-Guided Policy Optimization: Joint Structuring and Functional Policy Reinforcement To maximize planning capability and output reliability in a compact 3B parameter regime, SecAgent adopts a two-stage training paradigm moving from supervised imitation to reinforcement exploration. Stage 1 (SFT) trains the base MLLM on transformed grounding and navigation datasets using standard autoregressive next-token prediction over the triplet \((\mathcal{C}_t, H_t, A_t)\). Stage 2 (RFT) builds upon SFT using Group Relative Policy Optimization (GRPO), eliminating the need for an independent critic model by estimating advantages across sampled response groups. To balance syntactic validity with execution precision, the policy is guided by a dual rule-based reward function: a structural format reward \(R_{\text{format}}\) (+0.5 for correctly structured XML tags and parsable JSON arguments) and a functional action reward \(R_{\text{action}}\) (+1.0 for valid target box hits, correct text input, or accurate directional swipes; 0 otherwise). This decoupled reward formulation stabilizes policy gradients, rapidly driving the 3B model toward robust multi-step execution.

Loss & Training

During the first supervised fine-tuning stage, Qwen2.5-VL-3B is tuned using LoRA (rank 8, alpha 16) with a learning rate of \(2 \times 10^{-5}\), batch size 2, and 6 training epochs. Pixel bounds are clamped between 200,704 and 501,760 pixels. The loss optimizes the autoregressive log-likelihood: $\(\mathcal{L}_{\text{SFT}}(\theta) = -\sum_{i=1}^{|\mathcal{Y}|} \log P_\theta(y_i \mid y_{<i}, I, X_t, \mathcal{C}_{t-1}, X_{t-1}, A_{t-1})\)$ where output target sequence \(\mathcal{Y}\) encompasses the concatenated triplet \((\mathcal{C}_t, H_t, A_t)\).

In Stage 2 (RFT), full-parameter optimization is performed via token-level GRPO. For a given input state query \(q\), the old policy \(\pi_{\theta_{\text{old}}}\) samples \(G=16\) rollouts \(\{o_i\}_{i=1}^G\). The scalar total reward \(R_i = R_{\text{format}}(o_i) + R_{\text{action}}(o_i)\) is normalized across the group to yield normalized advantages: $\(\hat{A}_{i,t} = \frac{R_i - \text{mean}(\{R_j\}_{j=1}^G)}{\text{std}(\{R_j\}_{j=1}^G)}\)$ The policy parameters \(\theta\) are updated via the clipped surrogate objective with KL divergence regularization: $\(J_{\text{GRPO}}(\theta) = \frac{1}{\sum_{i=1}^G |o_i|} \sum_{i=1}^G \sum_{t=1}^{|o_i|} \min \left( r_{i,t}(\theta) \hat{A}_{i,t}, \; \text{clip}(r_{i,t}(\theta), 1 - \epsilon_{\text{low}}, 1 + \epsilon_{\text{high}}) \hat{A}_{i,t} \right) - \beta \mathbb{D}_{\text{KL}}\left[\pi_\theta \parallel \pi_{\text{ref}}\right]\)$ where the per-token probability ratio is: $\(r_{i,t}(\theta) = \frac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t} \mid q, o_{i,<t})}\)$ Hyperparameters are set to: learning rate \(1 \times 10^{-6}\), batch size 8, KL penalty \(\beta = 0.04\), asymmetric clipping bounds \(\epsilon_{\text{low}} = 0.20\) and \(\epsilon_{\text{high}} = 0.28\), executed across 2 epochs.

Key Experimental Results

Main Results

Evaluation is conducted on CMGUI-Bench (comprising 386 human-verified episodes, 2,547 navigation steps across 44 Chinese mobile apps with multi-choice action validity) and two established English benchmarks, AndroidControl and GUIOdyssey (100 sampled trajectories under high-level instructions). Performance is measured via Step Accuracy (SA, %) and Task Accuracy (TA, %).

Model Category Model Name Parameters CMGUI-Bench SA (%) CMGUI-Bench TA (%) AndroidControl SA (%) GUIOdyssey SA (%)
Closed-source Frontier GPT-4o Proprietary 48.4 0.0 20.8 29.3
Closed-source Frontier Claude-3-Sonnet Proprietary 81.4 25.8 12.5 46.5
Open-source Baselines (7B–8B) Qwen2.5-VL-7B 7B 83.8 35.6 62.9 59.6
Open-source Baselines (7B–8B) UI-TARS-7B 7B 63.8 2.5 74.4 37.5
Open-source Baselines (7B–8B) OS-Atlas-7B 7B 69.6 10.0 71.2 65.2
Open-source Baselines (7B–8B) GUI-R1-7B 7B 70.5 7.9 51.7 41.9
Open-source Baselines (7B–8B) UI-Venus-Navi-7B 7B 89.6 53.1 76.1 71.1
Open-source Baselines (7B–8B) AgentCPM-GUI-8B 8B 84.9 39.0 69.2 79.2
Open-source Baselines (7B–8B) Qwen3-VL-8B 8B 91.1 59.7 46.4 53.8
Open-source Baselines (3B–4B) Qwen2.5-VL-3B 3B 78.8 25.6 60.1 49.3
Open-source Baselines (3B–4B) UI-R1-3B 3B 56.4 1.0 45.4 32.7
Open-source Baselines (3B–4B) GUI-R1-3B 3B 45.2 0.5 46.6 34.6
Open-source Baselines (3B–4B) Qwen3-VL-4B 4B 90.1 57.4 47.5 53.4
Proposed Method SecAgent-3B (Ours) 3B 96.4 80.0 69.5 74.3

On CMGUI-Bench action breakdown, SecAgent-3B achieves 96.1% on Click Acc (vs. 91.2% for Qwen3-VL-4B), 99.1% on Type Acc (vs. 84.8%), an outstanding 87.6% on Swipe Acc (where other baselines predominantly collapse below 10%), and 99.2% on Terminate Acc.

Ablation Study

Ablations investigate history window size \(N\), the presence of semantic context (SC), chain-of-thought reasoning (Thought), and training strategies regarding computational efficiency and navigation accuracy.

Configuration / Variant SA (%) TA (%) Input Token Count (ITC) TTFT Latency (s) Throughput TPS (tokens/s) Description
\(N=0\) (No History) 85.3 36.2 1,537 0.07 145 Zero temporal context; severe failure on state-dependent actions
\(N=1\) (Default: w/ SC & Thought) 94.8 72.8 2,239 0.11 140 SFT baseline configuration achieving optimal efficiency
\(N=2\) (w/ SC & Thought) 95.1 73.8 2,760 0.14 134 Diminishing accuracy returns with 23.3% token growth
\(N=5\) (w/ SC & Thought) 95.5 74.1 3,642 0.22 122 High cost: +62.7% tokens and double TTFT for +1.3% TA
\(N=5\) w/o SC (Screenshots + Actions) 95.0 73.0 3,507 0.22 124 Stacking 5 raw frames matches or falls behind \(N=1\) with SC
\(N=1\) w/o SC (Only 1 Past Screen/Action) 90.6 56.4 2,171 0.11 142 Removing semantic context causes a catastrophic 16.4% TA drop
\(N=1\) w/o Thought (No CoT Reasoning) 94.0 68.9 2,207 0.11 141 Bypassing thought rationale causes a 3.9% TA drop

Examining data composition: SFT on Navigation alone achieves 93.7% SA / 65.8% TA; adding original Grounding yields 94.4% SA / 69.2% TA; adding format-aligned Gr2Nav data reaches 94.8% SA / 72.8% TA; applying Stage 2 GRPO on top of full SFT culminates in the peak result of 96.4% SA / 80.0% TA (+7.2% TA over SFT alone).

Key Findings

  • Semantic Context Delivers Drastic Efficiency Gains: Operating at \(N=1\) with semantic context achieves 72.8% TA, effectively rivaling \(N=5\) without semantic context (73.0% TA) while slashing input token consumption by 36.2% and halving time-to-first-token latency from 0.22s to 0.11s. Concise linguistic abstraction acts as a superior surrogate for deep visual image stacking.
  • Unified Action Format Mitigates Task Interference: Projecting bounding boxes into single-point center click targets (Gr2Nav) improves task completion by 3.6% over raw bounding box supervision (72.8% vs. 69.2% TA), verifying that unified action spaces facilitate cross-task feature sharing.
  • Reinforcement Fine-Tuning Enhances Generalization: On out-of-domain (OOD) applications, RFT substantially boosts zero-shot policy stability. When trained on partial application data, RFT-augmented models outperform SFT across all unseen apps (e.g., Baidu Maps TA improves from 41.7% to 62.5%), highlighting robust policy recovery in unmapped interface environments.

Highlights & Insights

  • High-Quality Ground-Truth Ecosystem Benchmark: CMGUI provides 18k grounding pairs and 121k rigorously human-verified navigation steps across 44 mainstream apps, introducing CMGUI-Bench with multi-choice action support to accommodate diverse valid GUI paths (e.g., search suggestions vs. explicit keyboard submission).
  • Causally Sound Single-Model Distillation: Unifies summarization, reasoning, and acting within a single 3B parameter agent operating in strict causal order ("semantic context \(\to\) thought \(\to\) action"), circumventing the latency, multi-agent overhead, and error propagation of external summarizers.
  • Decoupled Rule-Based GRPO Optimization: Formulates an efficient RL framework combining structural formatting rewards with execution accuracy rewards, enabling stable policy improvement without training unstable neural reward critics.

Limitations & Future Work

  • Ecosystem and Language Scope: CMGUI is currently focused on the Chinese mobile landscape. While cross-benchmark tests on AndroidControl and GUIOdyssey demonstrate positive transfer, generalizing the collection and evaluation pipeline to European and other non-English regional ecosystems remains future work.
  • Offline Trajectory Replay vs. Live Deployment: Benchmarking is conducted on pre-recorded interactive trajectories with multi-choice ground truth. Real-world deployment on physical devices will introduce asynchronous network jitter, unexpected modal popups, and dynamic frame drops not fully captured in offline environments.
  • Model Parameter Scaling: Investigations are confined to 3B-scale models. Validating whether semantic context distillation scales effectively to 7B, 14B, or 32B multimodal backbones remains to be explored.
  • vs OdysseyAgent / UI-TARS: Traditional vision-first agents stack multiple past screenshots (\(N \ge 5\)) to preserve temporal context, incurring excessive visual token budgets and high latency. SecAgent combines single-frame visual memory (\(N=1\)) with self-updating semantic summaries, reducing input tokens by 38.5% and TTFT by 50% while sustaining higher task success.
  • vs Mobile-Agent-v2: Mobile-Agent-v2 coordinates separate LLM instances for observation, reflection, and history summarization, resulting in substantial communication latency. SecAgent internalizes state summarization and action generation within a single compact 3B model.
  • vs GUI-Rise: GUI-Rise generates history summaries post-action, preventing the current operational step from directly grounding itself on updated historical distillation. SecAgent enforces a strict summary-first sequence, ensuring all decisions are conditioned on verified previous state progression.

Rating

  • Novelty: ⭐⭐⭐⭐☆ Introduces a neat, causal autoregressive semantic context mechanism that compresses multi-step visual trajectories into natural language summaries within a single agent.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations across 44 Chinese applications and two major English benchmarks, complete with fine-grained action breakdowns and efficiency ablations.
  • Writing Quality: ⭐⭐⭐⭐⭐ Crystal-clear structure, detailed formulation of coordinate transformation, and precise articulation of motivation and architectural designs.
  • Value: ⭐⭐⭐⭐⭐ Fills a critical open-source void in non-English mobile GUI automation by releasing a massive human-annotated dataset alongside an efficient 3B baseline.