Perceive Before Reasoning: A Pre-Reasoning Perception Framework for Efficient and Reliable Proactive Mobile Agents¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: Multimodal VLM
Keywords: Proactive Mobile Agents, Pre-Reasoning Perception, Intervention Gating, Context Compression, GRPO Reinforcement Learning
TL;DR¶
PRPF decouples the proactive mobile assistance dilemma into a two-stage "perceive before reasoning" architecture, leveraging a lightweight Multimodal Proactive Perceptor (MPP) for intervention gating and candidate function compression before triggering a dedicated Proactive Agent Reasoner (PAR) refined with SFT and GRPO, thereby doubling success rate (20.82% to 41.15%), cutting false trigger rate by nearly half (13.76% to 7.21%), and slashing expected inference compute by 69.3% on the ProactiveMobile benchmark.
Background & Motivation¶
Multimodal large language models (MLLMs) and vision-language foundation backbones have substantially transformed mobile agents, enabling automated smartphone operation, graphical user interface (GUI) understanding, and execution of complex user instructions. However, autonomous mobile assistants are increasingly expected to evolve beyond passive instruction-following toward proactive intelligence, where agents anticipate latent human requirements and offer executable assistance before an explicit request is formulated. In mobile operating systems, proactive intervention operates under an inherently asymmetric cost structure: while a timely and precise recommendation accelerates productivity, an erroneous or unsolicited interruption fractures the user's immediate attention and degrades trust, whereas excessively timid behavior reduces the system back to a reactive tool.
The primary impediment in existing proactive mobile assistants is an architectural goal misalignment rooted in unified single-stage designs. Current paradigms—such as the unified timing and content generation framework in ProAgentBench or the mode-switching Observe–Execute pipeline in PARE-Bench—delegate both "when to intervene" and "how to assist" to a single fine-tuned VLM backbone. The decision of when to intervene requires a conservative, highly discriminative, and risk-averse classifier capable of suppressing false positives under noisy on-device streams. Conversely, deciding how to assist demands open-ended multimodal visual grounding, long-horizon trajectory synthesis, and flexible API tool selection. Forcing a monolithic generative VLM to optimize both conflicting objectives simultaneously induces high false trigger rates (FTRs), causing even frontier models like GPT-5.5 and o3 to trigger false interruptions in 23% to 32% of cases.
Furthermore, single-stage VLM agents suffer from severe inference inefficiency when continuously observing mobile environments. In continuous on-device operation, the correct behavioral choice across the vast majority of time steps is simply to remain silent. Running dense, multi-billion parameter autoregressive models across multi-turn screenshots, user profiles, and comprehensive schema catalogs spanning dozens of tools creates redundant computational overhead and battery drain. Recognizing that discrimination should precede generative deliberation, this paper proposes an architectural separation that enforces lightweight perceptual filtering before heavy cognitive reasoning. Core idea: Introduce the Pre-Reasoning Perception Framework (PRPF), which establishes an architectural separation between a lightweight, dual-channel Multimodal Proactive Perceptor (MPP) for early intervention gating and top-K scenario compression, and a focused Proactive Agent Reasoner (PAR) optimized via SFT and GRPO to synthesize grounded executable recommendations exclusively when intervention is warranted.
Method¶
Overall Architecture¶
The input to the proactive mobile assistant comprises the user profile (\(U\)), current device state (\(D\)), ambient world context (\(W\)), and interaction trajectory history (\(I\), represented as consecutive GUI screenshots or textual traces). The target output is either an abstention signal \(\emptyset\) indicating that the agent should remain silent, or an executable composite API call sequence \(\hat{s} = [(\hat{f}_1, \hat{a}_1), \dots, (\hat{f}_m, \hat{a}_m)]\). PRPF organizes this process into two decoupled stages: the lightweight Multimodal Proactive Perceptor (MPP) and the generative Proactive Agent Reasoner (PAR). MPP operates with minimal compute (23.58 GFLOPs, 14.16 ms latency) to evaluate an intervention gate probability and restrict the candidate function pool from 63 tools to the union of the Top-5 intent scenarios (a 68.25% reduction). Only when the gate accepts the instance is PAR (based on Qwen3.5-9B) activated to perform chain-of-thought grounding, intent deduction, and executable function-call generation.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
A["Multimodal Mobile Context Input<br/>User Profile/Device State/World Context/GUI Screenshots"] --> B["Fast-Slow Dual-Channel Perceptor MPP<br/>Extract local dynamics and global preferences with cross-attention"]
B --> C{"Trigger Gating Head<br/>ptrig >= 0.78"}
C -->|No: Maintain silence| D["Early Silence Cutoff<br/>Terminate immediately without calling heavy VLM"]
C -->|Yes: Trigger intervention| E["Scenario Prediction & Function Compression<br/>Select Top-5 scenarios to shrink candidate API pool by 68.25%"]
E --> F["Structured CoT Reasoner PAR<br/>Generate UI summary + thinking chain + function grounding"]
F --> G["GRPO Alignment & Calibration Policy<br/>Reinforce true silence, penalize false calls and redundant steps"]
G --> H["Executable Proactive Output<br/>Grounded recommendation text and validated API parameters"]
Key Designs¶
1. Fast-Slow Dual-Channel Perceptor MPP: Disentangling instantaneous GUI cues from long-term user context
Determining the exact moment for proactive assistance requires capturing both instantaneous UI micro-interactions and long-term user behavioral routines. Single-encoder front-ends fail to balance these disparate temporal scales efficiently. MPP addresses this by projecting text embeddings \(H_{\text{text}} \in \mathbb{R}^{3 \times d}\) (via BGE-small-zh-v1.5) and visual embeddings \(H_{\text{img}} \in \mathbb{R}^{T \times d}\) (via CLIP ViT-B-32) into a unified dimension \(d\). It then splits perception into two pathways: a Slow Channel that applies attention pooling across the complete visual trajectory \(\bar{x} = \text{AttnPool}(c_{\text{img}})\) concatenated with \(c_{\text{text}}\) through a Transformer Encoder \(T_{\text{slow}}\) to capture global habits; and a Fast Channel that isolates the most recent \(k=4\) frames \(c_{\text{img}}[T-k+1:T]\) into \(T_{\text{fast}}\) to register immediate screen dynamics. To bridge long-range context with immediate activity, MPP fuses both channels via bidirectional multi-head cross-attention: $$ \tilde{H}c = \text{MHA}(Q = H_c, K = H)} $$ Pooling and concatenating the enhanced representations yields a unified contextual feature }}, V = H_{\bar{c}}), \quad c \in {(\text{fast}, \text{slow}), (\text{slow}, \text{fast\(z \in \mathbb{R}^{2d}\), allowing the lightweight gate to ground immediate actions in historical context.
2. Scenario-Guided Function Pool Compression: Mitigating long-tail schema interference
In modern mobile operating systems, an agent must consider dozens of candidate APIs across diverse lifestyle domains. Feeding the full catalog of 63 functions into an MLLM expands prompt token length, inflates quadratic attention cost, and creates semantic distractors during tool matching. MPP places an intent-scenario prediction head \(\text{MLP}_{\text{scenario}}\) over \(z\), predicting probabilities across 14 high-level functional domains: \(p_{\text{scenario}} = \sigma(\text{MLP}_{\text{scenario}}(z))\). Taking the union of functions tied to the Top-\(K\) (\(K=5\)) scenarios constructs a compressed candidate pool \(\tilde{\mathcal{F}}\). This step achieves 95.52% scenario recall on the development split while slashing the candidate pool by 68.25% (from 63 to fewer than 20 functions on average), dramatically accelerating prompt prefill and preventing out-of-domain hallucinations.
3. Structured CoT Reasoner PAR: Staged cognitive verification and argument grounding
When passed through the gating barrier, the selected context and compressed schema \(\tilde{\mathcal{F}}\) are handled by PAR (Qwen3.5-9B). Rather than generating raw API tokens directly, PAR enforces a multi-step structured thinking protocol: it first produces a <ui_summary> that transcribes interface state transitions and identifies salient screen components; next, it executes an introspective <thinking> chain containing an explicit <function_selection> block that contrasts candidate APIs in \(\tilde{\mathcal{F}}\) against the user's immediate state; finally, it emits the validated executable call in <recommendations>. Crucially, PAR retains the agency to output "No recommendation", acting as an internal consistency auditor that rescues false triggers generated by upstream perception.
4. GRPO Alignment & Calibration Policy: Reinforcing silent restraint and argument precision
While supervised fine-tuning teaches the model schema adherence, boundary cases remain vulnerable to trigger hallucination. PRPF applies Group Relative Policy Optimization (GRPO) to fine-tune PAR on rollout sets with high variance (\(N=8\) rollouts per prompt). To handle the asymmetric utility between proactive assistance and intrusive false alarms, the reward function couples accuracy \(R_{\text{acc}}\) and format validity \(R_{\text{fmt}}\) with an explicit calibration bonus: $$ R_{\text{cal}} = b_{\text{TN}} \mathbb{I}{\text{TN}} + (R}} + p_{\text{FP}}) \mathbb{I{\text{FP}} + (b $$ By setting a high true-negative reward }} + p_{\text{over}}) \mathbb{I}_{\text{TP}\(b_{\text{TN}} = 6.0\), a strict false-positive penalty \(p_{\text{FP}} = -1.5\), and an over-prediction penalty \(p_{\text{over}}\), GRPO establishes a steep reward gradient that penalizes spurious actions even if they adhere to schema formatting, driving down the false trigger rate while maintaining high argument coverage.
Loss & Training¶
PRPF is trained through a decoupled two-stage training scheme using the official ProactiveMobile training split (8,876 samples): 1. MPP Joint Optimization: Trained on an 8:2 internal split using AdamW (learning rate \(1\times 10^{-4}\)). To counteract class imbalance, negative instances are oversampled and optimized with Focal Loss (\(\gamma = 2.0\)) for the gating head. The scenario prediction head is supervised with binary cross-entropy (BCE) loss: \(\mathcal{L}_{\text{MPP}} = \mathcal{L}_{\text{trig}} + \lambda \mathcal{L}_{\text{scenario}}\) with \(\lambda = 0.5\). Operating point selection under a constrained optimization rule (Recall \(\ge 0.90\)) sets the deployment threshold at \(\tau = 0.78\) on the dev set. 2. PAR Supervised Fine-Tuning (SFT): Qwen3.5-9B is fine-tuned over 8,876 multimodal and text trajectories synthesized using Claude Sonnet 4.6 supervision with cross-entropy loss \(\mathcal{L}_{\text{SFT}}\) across 5 epochs (learning rate \(1\times 10^{-5}\), cosine decay). 3. PAR Reinforcement Tuning (GRPO): Optimized on 5,000 retained boundary rollouts (containing \(\ge 30\%\) negative records) over 1 epoch (learning rate \(5\times 10^{-7}\), KL penalty \(\beta = 0.05\), batch size 32).
Key Experimental Results¶
Main Results¶
The primary evaluation is conducted on the ProactiveMobile test set (1,832 multimodal and 1,828 text samples) evaluating function sequence accuracy (Type-Acc), end-to-end success rate (SR, judged by Gemini-2.5-Pro for parameter equivalence), and false trigger rate (FTR).
| Model | Multimodal Type-Acc (%) | Multimodal SR (%) | Multimodal FTR (%) | Text Type-Acc (%) | Text SR (%) | Text FTR (%) | ALL Type-Acc (%) | ALL SR (%) | ALL FTR (%) |
|---|---|---|---|---|---|---|---|---|---|
| GPT-5.5 | 40.34 | 18.40 | 33.48 | 55.09 | 49.02 | 19.24 | 47.71 | 33.69 | 23.84 |
| o3 | 39.63 | 17.25 | 41.45 | 51.31 | 44.53 | 26.89 | 45.46 | 30.87 | 31.61 |
| Gemini-3.1-Pro | 18.94 | 9.33 | 76.47 | 33.97 | 26.75 | 56.60 | 26.45 | 18.03 | 61.11 |
| Claude-Opus-4.7 | 37.23 | 15.83 | 41.03 | 55.91 | 48.69 | 18.81 | 46.56 | 32.24 | 26.11 |
| GLM-4.6V | 36.25 | 11.74 | 74.45 | 29.98 | 20.19 | 67.04 | 33.12 | 15.96 | 69.33 |
| UI-TARS-7B-DPO+Proactive | 30.90 | 8.52 | 63.13 | 14.17 | 9.90 | 64.67 | 22.54 | 9.21 | 63.84 |
| Qwen3.5-9B (Zero-Shot) | 8.30 | 2.95 | 71.52 | 8.15 | 5.69 | 77.13 | 8.22 | 4.32 | 75.30 |
| ProactiveMobile (7B Baseline) | 33.68 | 15.61 | 23.91 | 56.84 | 26.04 | 8.51 | 45.25 | 20.82 | 13.76 |
| Qwen3.5-9B+Proactive | 39.52 | 14.85 | 18.54 | 59.52 | 54.32 | 10.63 | 49.51 | 34.56 | 13.49 |
| PRPF (Ours) | 40.83 | 17.19 | 17.99 | 69.20 | 65.15 | 1.75 | 55.00 | 41.15 | 7.21 |
Ablation Study¶
Systematic ablations isolate the impact of the structural, perceptual, and optimization components across the test suite:
| Configuration / Variant | Multimodal SR (%) | Multimodal FTR (%) | Text SR (%) | Text FTR (%) | ALL SR (%) | ALL FTR (%) | Note / Observation |
|---|---|---|---|---|---|---|---|
| Full PRPF (Ours) | 17.19 | 17.99 | 65.15 | 1.75 | 41.15 | 7.21 | Full two-stage pipeline |
| - w/o MPP (All samples directly enter PAR) | 15.56 | 22.01 | 61.16 | 4.22 | 38.33 | 10.38 | ALL SR drops 2.82%, FTR worsens by 3.17% |
| - w/o PAR (Raw Qwen3.5-9B for How stage) | 6.06 | 60.64 | 60.89 | 3.33 | 33.44 | 18.46 | Multimodal SR collapses; lack of structured reasoning |
| - w/o Slow Channel (Drop global visual path) | 16.59 | 20.88 | 62.91 | 3.44 | 39.73 | 9.38 | Losses in long-term user profile alignment |
| - w/o Fast Channel (Drop recent frame window) | 16.54 | 17.73 | 64.33 | 2.02 | 40.41 | 7.32 | Minor degradation in recent interaction dynamics |
| - w/o Compression (Feed all 63 APIs to PAR) | 16.65 | 19.11 | 64.50 | 2.02 | 40.55 | 7.78 | Increased distractors and longer prompt prefill |
| - w/o Recommend (No gating, only compress) | 15.88 | 21.28 | 62.75 | 3.87 | 39.29 | 9.74 | Inability to prune silent samples raises FTR |
| - w/o GRPO (SFT only, no RL calibration) | 16.05 | 48.63 | 63.35 | 2.81 | 39.67 | 15.56 | FTR doubles from 7.21% to 15.56% |
| - w/o SFT (Zero-shot base model) | 4.91 | 61.60 | 60.18 | 4.37 | 32.51 | 18.61 | Severely impaired structured tool-calling format |
Key Findings¶
- Drastic Reduction in False Triggers: General-purpose models trigger excessive false alarms (GPT-5.5 at 23.84%, Gemini-3.1-Pro at 61.11%). PRPF drives overall FTR down to 7.21% (and 1.75% in text settings). The pre-reasoning gate filters out 83.8% of silent cases, relieving downstream modules of the conservative filtering burden.
- Compute and Latency Reductions: Filtering 36.1% of silent cases via MPP and shrinking the function catalog from 63 to fewer than 20 tools reduces expected per-sample compute by 69.3% and end-to-end latency by 60.1% compared to single-stage 7B baselines, requiring only a modest 12.0% increase in peak GPU memory to host the 9B model.
- Plug-and-Play Adaptability: Adding MPP upstream of ProactiveMobile (7B) boosts ALL SR from 20.82% to 37.38% and drops FTR from 13.76% to 8.08%. When paired with GLM-4.6V, MPP reduces FTR from 69.33% to 19.97% while raising SR from 15.96% to 38.39%, demonstrating broad transferability across backbones.
- Cross-Benchmark Generalization: On ContextAgentBench, PRPF achieves state-of-the-art results across passive timing accuracy (Acc-P = 0.913), false detection rate (FD = 0.008), timing RMSE (1.121), and tool invocation quality (Tool-F1 = 0.788), outperforming both GPT-5.5 and Qwen3.5-9B.
- Multimodal Visual Bottleneck: While PRPF scores 65.15% SR in text scenarios, it attains only 17.19% SR on multimodal screenshots. Error decomposition shows that 34.8% of multimodal failures are caused by post-gate abstention or misfired silences, and 24.1% by incorrect function selection, underscoring that fine-grained grounding on dense smartphone interfaces remains challenging for current visual encoders.
Highlights & Insights¶
- Architectural Decoupling of Asymmetric Goals: By isolating the conservative discriminative task ("when") from the generative reasoning task ("how"), PRPF prevents the objective conflict inherent in single-stage proactive VLMs.
- Hierarchical Intent Filtering Funnel: Routing requests through a coarse 14-scenario prediction stage before deep reasoning prevents API schema bloat, reducing prompt tokens and eliminating 68.25% of potential tool distractors.
- Targeted True-Negative Reward Shaping in GRPO: Incorporating an explicit reward bonus for maintaining silence on negative examples (\(b_{\text{TN}} = 6.0\)) provides a clean reinforcement learning recipe for building polite, unobtrusive mobile assistants.
Limitations & Future Work¶
- Static Scenario Taxonomy: MPP relies on a fixed vocabulary of 14 intent scenarios from ProactiveMobile. Expanding to dynamic third-party app ecosystems will require continual learning adapters or prompt-based routing to avoid retraining the perceptor.
- Fine-Grained GUI Grounding Ceiling: Low absolute accuracy on multimodal inputs (17.19%) indicates that generic visual backbones (e.g., CLIP ViT) struggle to distinguish subtle UI affordances, tiny mobile icons, and dense app states. Future iterations will benefit from high-resolution, native GUI encoders.
- Single-Turn Decision Horizon: The current setup focuses on immediate, single-step proactive recommendations, lacking an interactive execution feedback loop for multi-step agentic recovery upon user rejection or task failure.
Related Work & Insights¶
- vs Proactive Agent & PRISM: While PRISM employs multi-step risk-sensitive deliberation within a single model, it incurs high compute costs during non-intervention periods. PRPF establishes a physical, lightweight front-end gate that halts inference before calling the primary LLM.
- vs ProAgentBench & PARE-Bench: Existing benchmarks frame proactive interaction as an implicit tool-use mode switch. PRPF demonstrates that splitting the task into explicit discriminative filtering followed by schema-compressed generation yields substantially better real-world reliability.
- vs GUI Grounding Frameworks (Think Twice, Click Once & iShift): While prior fast-slow systems optimize spatial target selection ("where to click"), PRPF innovates at the cognitive timing and intention level ("when to intervene and which API to execute").
Rating¶
- Novelty: ⭐⭐⭐⭐☆ Decouples the proactive assistance bottleneck into an elegant two-stage cascade with specialized gating and compression.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations across multiple model tiers, difficulty splits, and distinct benchmarks.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear problem formulation, detailed mathematical modeling, and thorough empirical failure analyses.
- Value: ⭐⭐⭐⭐⭐ Offers a pragmatic, deployable paradigm for continuous, energy-efficient, and polite on-device AI assistants.