SecAgent: Efficient Mobile GUI Agent with Semantic Context¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://huggingface.co/datasets/alibabagroup/CMGUI
Area: LLM Reasoning
Keywords: Mobile GUI Agent, Semantic Context Mechanism, Chinese GUI Benchmark, Reinforcement Fine-Tuning, History Compression
TL;DR¶
SecAgent tackles prohibitive multi-image history overheads and the scarcity of high-quality non-English datasets in mobile GUI agents by introducing a natural language Semantic Context mechanism and releasing CMGUI, achieving superior 3B performance that rivals 7B-8B baselines.
Background & Motivation¶
Multimodal Large Language Model (MLLM)-powered mobile Graphical User Interface (GUI) agents are transforming smartphone task automation. While early text-based systems depended heavily on platform accessibility trees with frequent metadata gaps, the research community has widely embraced vision-based approaches that perceive interfaces directly via screenshots. However, accurate multi-step execution fundamentally relies on temporal context to track previous actions and resulting UI transitions. To maintain historical coherence, existing state-of-the-art vision agents stack multiple raw history screenshots (such as up to 5 full screenshots in OdysseyAgent). Because a single embedded screenshot occupies hundreds of visual tokens, this approach incurs prohibitive computational burdens, causing memory bloat, high Time to First Token (TTFT), and degraded real-time inference throughput.
Simultaneously, current mobile GUI agent research is predominantly centered around the English mobile ecosystem, leaving open-source non-English environments severely underdeveloped. Existing Chinese datasets are either closed-source industrial assets or collected via unverified random-walk trajectories with coarse pseudo-labels. Autonomous segmentation tools like SAM2 frequently introduce noisy bounding boxes on compact mobile interfaces, failing to provide reliable fine-grained supervision. Furthermore, smartphone interactions inherently accommodate diverse valid paths to accomplish the same intent (such as submitting via a search button versus selecting a suggestion tag). Standard single-ground-truth benchmarks fail to accommodate these alternative paths, systematically underestimating an agent's true generalization and execution capacity.
To overcome both the computational bottleneck of historical image stacking and the lack of high-quality Chinese benchmarks, mobile agents require an efficient, semantically dense historical representation paired with rigorously verified trajectory datasets. The core idea is to replace raw historical screenshot stacking with an autoregressively maintained natural language "Semantic Context," distilling multi-step visual transitions into concise textual summaries anchored only to the immediate single prior step, and combining human-verified multi-choice Chinese GUI data (CMGUI) with unified coordinate alignment and two-stage reinforcement fine-tuning.
Method¶
Overall Architecture¶
SecAgent formulates mobile GUI navigation as a policy mapping user goal instructions, compact visual inputs, and dynamic textual summaries into executable actions. At step \(t\), the agent receives user instruction \(I\), current screenshot \(X_t\), previous semantic context summary \(C_{t-1}\), immediate prior screenshot \(X_{t-1}\), and previous action \(A_{t-1}\). Instead of maintaining an extensive image queue \(N > 1\), SecAgent autoregressively predicts a structured triplet: an updated semantic context \(C_t\), a Chain-of-Thought reasoning rationale \(H_t\), and the parameterized target action \(A_t\):
The natural language summary \(C_t\) incrementally aggregates critical past milestones (e.g., entered queries, clicked categories, page transitions) in plain text. This allows the model to compress long-horizon historical dependencies into minimal token footprints while keeping visual inputs strictly bounded to \(N=1\). The complete pipeline is supported by the CMGUI dataset curation workflow, cross-task Grounding-to-Navigation format unification, and a two-stage Supervised and Reinforcement Fine-Tuning paradigm.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multimodal Inputs<br/>Instruction I + Screen Xt + History Xt-1, At-1 + Summary Ct-1"] --> B["Semantic Context Mechanism<br/>Distill single-step visual transitions into text progress Ct"]
B --> C["Cross-Task Format Alignment<br/>Remap grounding bounding boxes to center-point click coordinates"]
C --> D["Two-Stage Alignment and Reinforcement Tuning<br/>LoRA imitation warm-up + GRPO rule-driven policy optimization"]
D --> E["Structured Triplet Output<br/>Semantic Context Ct + Thought Ht + Action At"]
Key Designs¶
1. Semantic Context Mechanism: Eliminating Multi-Image Memory Bloat via Natural Language Distillation
Standard GUI agents face an undesirable trade-off between retaining only action histories (which discards visual state transitions) and stacking multiple raw screenshots (which causes linear computational explosions). SecAgent introduces the Semantic Context mechanism, dynamically converting raw visual state shifts into explicit textual progress summaries. At each step \(t\), the agent inspects the transition between \(X_{t-1}\), \(A_{t-1}\), and \(X_t\) to produce a sequential natural language log of completed sub-goals (e.g., "Clicked the search bar and activated the input keyboard"). By carrying this compact text summary forward, SecAgent retains long-range historical awareness while discarding all visual tokens older than \(N=1\). Because LLMs excel at processing narrative sequences, natural language summaries provide richer, more causal representations than isolated coordinate tuples.
2. Cross-Task Format Alignment: Unifying Grounding Bounding Boxes with Navigation Click Spaces
GUI agent training typically co-trains on screen grounding (localizing UI elements) and multi-step navigation trajectories. However, standard grounding objectives mandate four-coordinate bounding box outputs \([x_{\min}, y_{\min}, x_{\max}, y_{\max}]\), whereas navigation steps require a single interactive click coordinate \([x, y]\). Directly mixing these conflicting action spaces induces negative transfer and representation inconsistency. SecAgent introduces Grounding-to-Navigation (Gr2Nav) transformation: during curation of the 18k Chinese grounding corpus, each human-verified bounding box is transformed into an atomic click action targeting its geometric center \((x_{\text{center}}, y_{\text{center}})\). This unifies the action space across all training tasks, enabling seamless knowledge transfer from element perception to multi-step planning.
3. Two-Stage Alignment and Reinforcement Tuning: Bridging Imitation Warm-Up and Rule-Driven Policy Optimization
To cultivate robust decision-making in a compact 3B backbone, SecAgent implements a two-stage training scheme comprising Supervised Fine-Tuning (SFT) followed by Reinforcement Fine-Tuning (RFT). Stage 1 applies LoRA on unified grounding and trajectory data to instill basic UI comprehension and the <summary>, <think>, <tool_call> output syntax. Stage 2 performs full-parameter policy optimization using Group Relative Policy Optimization (GRPO). Without requiring a separate reward model or critic, the policy samples \(G=16\) outputs per state and computes normalized advantages across the rollout group. The policy is guided by a dual rule-based reward: a format reward of 0.5 for properly structured triplets, and an action reward of 1.0 when the predicted action matches any valid multi-choice annotation. Policy drift is constrained via asymmetric clipping ratios (\(\epsilon_{\text{low}}=0.2, \epsilon_{\text{high}}=0.28\)) and a KL divergence penalty:
Loss & Training¶
During the SFT stage, the model is optimized with next-token cross-entropy loss over 6 epochs, using batch size 2, learning rate \(2\times 10^{-5}\), and LoRA rank/alpha of 8/16. In the RFT stage, full-parameter GRPO runs for 2 epochs with batch size 8, learning rate \(1\times 10^{-6}\), 16 rollouts per query, KL penalty weight \(\beta=0.04\), and image pixel constraints bounded between 200,704 and 501,760 pixels.
Key Experimental Results¶
Main Results¶
Evaluation was conducted on CMGUI-Bench (386 successful episodes, 2,547 steps across 44 Chinese applications with multi-choice action labels) as well as the English benchmarks AndroidControl and GUIOdyssey. All evaluations adopt standard high-level user instructions.
Performance on the CMGUI-Bench Navigation Benchmark:
| Model Category | Model Name | Step Acc. (SA %) | Task Acc. (TA %) | Click Acc. (%) | Type Acc. (%) | Swipe Acc. (%) | Terminate Acc. (%) |
|---|---|---|---|---|---|---|---|
| Closed-source | GPT-4o | 48.4 | 0.0 | 43.5 | 68.3 | 63.7 | 62.1 |
| Closed-source | Claude-3-Sonnet | 81.4 | 25.8 | 91.1 | 30.0 | 81.3 | 53.4 |
| Open-source 7B/8B | Qwen2.5-VL-7B | 83.8 | 35.6 | 89.0 | 75.9 | 4.4 | 79.0 |
| Open-source 7B/8B | UI-TARS-7B | 63.8 | 2.5 | 80.2 | 11.1 | 3.3 | 8.3 |
| Open-source 7B/8B | OS-Atlas-7B | 69.6 | 10.0 | 77.0 | 76.8 | 5.5 | 32.1 |
| Open-source 7B/8B | GUI-R1-7B | 70.5 | 7.9 | 84.2 | 45.5 | 5.5 | 8.7 |
| Open-source 7B/8B | UI-Venus-Navi-7B | 89.6 | 53.1 | 92.6 | 91.5 | 45.0 | 81.2 |
| Open-source 7B/8B | AgentCPM-GUI-8B | 84.9 | 39.0 | 85.8 | 93.8 | 54.9 | 81.2 |
| Open-source 7B/8B | Qwen3-VL-8B | 91.1 | 59.7 | 92.6 | 82.6 | 69.2 | 94.6 |
| Open-source 3B/4B | Qwen2.5-VL-3B | 78.8 | 25.6 | 83.2 | 66.0 | 5.5 | 81.9 |
| Open-source 3B/4B | UI-R1-3B | 56.4 | 1.0 | 71.2 | 7.6 | 5.5 | 6.5 |
| Open-source 3B/4B | GUI-R1-3B | 45.2 | 0.5 | 57.1 | 8.0 | 2.2 | 4.3 |
| Open-source 3B/4B | Qwen3-VL-4B | 90.1 | 57.4 | 91.2 | 84.8 | 65.9 | 94.6 |
| Ours | SecAgent-3B | 96.4 | 80.0 | 96.1 | 99.1 | 87.6 | 99.2 |
Cross-Benchmark Generalization on English Benchmarks:
| Model Category | Model Name | AndroidControl (SA %) | GUIOdyssey (SA %) |
|---|---|---|---|
| Closed-source | GPT-4o | 20.8 | 29.3 |
| Closed-source | Claude-3-Sonnet | 12.5 | 46.5 |
| Open-source 7B/8B | Qwen2.5-VL-7B | 62.9 | 59.6 |
| Open-source 7B/8B | UI-TARS-7B | 74.4 | 37.5 |
| Open-source 7B/8B | OS-Atlas-7B | 71.2 | 65.2 |
| Open-source 7B/8B | GUI-R1-7B | 51.7 | 41.9 |
| Open-source 7B/8B | UI-Venus-Navi-7B | 76.1 | 71.1 |
| Open-source 7B/8B | AgentCPM-GUI-8B | 69.2 | 79.2 |
| Open-source 7B/8B | Qwen3-VL-8B | 46.4 | 53.8 |
| Open-source 3B/4B | Qwen2.5-VL-3B | 60.1 | 49.3 |
| Open-source 3B/4B | UI-R1-3B | 45.4 | 32.7 |
| Open-source 3B/4B | GUI-R1-3B | 46.6 | 34.6 |
| Open-source 3B/4B | Qwen3-VL-4B | 47.5 | 53.4 |
| Ours | SecAgent-3B | 69.5 | 74.3 |
Ablation Study¶
Ablations validate history window sizes \(N\), the Semantic Context (SC) module, reasoning thoughts, and data alignment variants.
History Window Size and Computational Efficiency:
| Setting | Step Acc. (SA %) | Task Acc. (TA %) | Input Token Count (ITC) | Time to First Token (TTFT s) | Tokens per Second (TPS) |
|---|---|---|---|---|---|
| \(N = 0\) (Current frame only) | 85.3 | 36.2 | 1537 | 0.07 | 145 |
| \(N = 1\) (Default: 1-step + SC) | 94.8 | 72.8 | 2239 | 0.11 | 140 |
| \(N = 2\) | 95.1 | 73.8 | 2760 | 0.14 | 134 |
| \(N = 5\) | 95.5 | 74.1 | 3642 | 0.22 | 122 |
| \(N = 5\) w/o SC | 95.0 | 73.0 | 3507 | 0.22 | 124 |
| \(N = 1\) w/o SC | 90.6 | 56.4 | 2171 | 0.11 | 142 |
| \(N = 1\) w/o thought | 94.0 | 68.9 | 2207 | 0.11 | 141 |
Training Data Formulation and Methodology:
| Training Data Composition | Training Paradigm | Step Acc. (SA %) | Task Acc. (TA %) |
|---|---|---|---|
| Navigation only | SFT | 93.7 | 65.8 |
| Original Grounding + Navigation | SFT | 94.4 | 69.2 |
| Gr2Nav (Aligned) + Navigation | SFT | 94.8 | 72.8 |
| Gr2Nav + Navigation | RFT only (1k SFT warm-up) | 95.2 | 75.9 |
| Gr2Nav + Navigation | Full SFT + RFT | 96.4 | 80.0 |
Key Findings¶
- Semantic Compression Matches 5-Image Stacks: \(N=1\) with Semantic Context achieves 72.8% Task Accuracy, essentially matching the 73.0% achieved by stacking 5 full images (\(N=5\) w/o SC). Concurrently, Input Token Count drops by 36.2% (2,239 vs. 3,507) and TTFT latency is cut in half (0.11s vs. 0.22s), proving language summaries preserve task-critical progress without visual latency penalties.
- Gr2Nav Prevents Negative Transfer: Converting grounding bounding boxes into single-point click coordinates yields a 3.6% boost in Task Accuracy over raw grounding co-training (72.8% vs. 69.2%), confirming that unified action spaces resolve objective mismatch between element detection and mobile control.
- RFT Boosts Complex Gestures: Adding GRPO on top of full SFT elevates Task Accuracy by 7.2% (72.8% to 80.0%) and drives Swipe Accuracy up to 87.6% (compared to under 10% for unaligned open-source baselines), demonstrating the power of rule rewards for non-trivial interactive actions.
Highlights & Insights¶
- Language Summaries Replace Image Stacks: Replacing heavy image queues with recursively maintained natural language progress reports provides an elegant solution to the context-length vs. accuracy dilemma in mobile vision agents.
- Unified Gr2Nav Mapping: Converting grounding bounding boxes into central click targets harmonizes perceptual pre-training with sequential action spaces, offering a simple blueprint for mixing UI data sources.
- Multi-Choice Action Ground Truth: CMGUI-Bench addresses the pervasive single-path evaluation flaw in smartphone benchmarks by systematically annotating alternative valid interactions for identical user intents.
Limitations & Future Work¶
- Ecosystem Breadth: CMGUI focuses on Chinese mobile applications; expanding the collection pipeline to other underrepresented non-English language ecosystems remains open.
- Scale Exploration: Experiments are conducted primarily at the 3B parameter tier; assessing whether Semantic Context and RFT scale effectively to 7B or 14B backbones is left for future exploration.
- Offline Trajectory Disconnect: Evaluations rely on offline pre-recorded trajectories, which do not fully replicate real-time deployment dynamics such as unexpected pop-ups, network lag, and permission prompts.
Related Work & Insights¶
- vs OdysseyAgent: OdysseyAgent relies on stacking up to 5 history screenshots, causing high token overhead and inference latency; SecAgent uses autoregressive text summaries with \(N=1\) history, slashing input size and TTFT while matching accuracy.
- vs Mobile-Agent-v2: Mobile-Agent-v2 separates planning, information extraction, and summarization into distinct cooperating agents; SecAgent achieves the entire summary-thought-action cycle within a single compact 3B model.
- vs UI-Venus / AgentCPM-GUI: While previous GUI RL frameworks emphasize general CoT reasoning, SecAgent explicitly unifies the grounding/navigation action space and applies multi-choice rule-based rewards, enabling a 3B model to outperform 7B-8B competitors.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ Elegant semantic context compression of visual history combined with multi-choice trajectory evaluation.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 44 apps, latency profiling, and cross-lingual English transfer.
- Writing Quality: ⭐⭐⭐⭐⭐ Well-structured narrative with crisp mathematical formulations and extensive ablation breakdowns.
- Value: ⭐⭐⭐⭐⭐ Provides the first high-quality human-verified Chinese mobile GUI corpus alongside an efficient, deployable 3B agent.