DualTAP: A Dual-Task Adversarial Protector for Mobile MLLM Agents¶

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/fyzhang1/DualTAP
Area: LLM Safety
Keywords: mobile agents, personally identifiable information, contrastive attention, adversarial perturbations, privacy and utility

TL;DR¶

DualTAP perturbs mobile screenshots before upload, using contrastive attention to locate privacy-sensitive regions and a dual-task objective to preserve normal operations, reducing GPT-5 leakage on PrivScreen from 97.14% to 23.33% while retaining 91.00% task accuracy.

Background & Motivation¶

Mobile GUI agents use the screen as their main observation interface: they capture the current page, send the screenshot and instruction to a remote multimodal large language model (MLLM), and then click, type, or swipe according to its response. This avoids requiring dedicated application interfaces, but screenshots also expose names, email addresses, phone numbers, and account details. When requests pass through a third-party API router, that router can faithfully forward the legitimate task while separately using a model to extract personally identifiable information (PII). The risk is therefore not limited to an agent taking an unsafe action; an intermediary can obtain private content unrelated to an otherwise legitimate operation. Accumulated screenshots may also support user profiling by linking scattered fields, although this motivates the threat model rather than constituting a fully quantified multi-frame attack experiment in the paper.

Blurring or masking a screenshot reduces readable information but may also destroy button locations, navigation labels, and interface semantics. General visual adversarial methods face a similar problem: making an image incomprehensible can block both privacy extraction and legitimate task completion. The region-specific method VIP is closer to the desired protection scope, but modifying only sensitive regions does not necessarily affect only sensitive tasks because a model jointly encodes the image. The needed distinction concerns how the same visual input supports questions such as identifying the next button versus reading a user's email address. Table 2 directly illustrates this tension: privacy-only optimization yields 64.00% accuracy, whereas normal-task-only optimization leaves leakage at 94.19%.

DualTAP neither requires router cooperation nor changes target-model weights; it modifies the outgoing image on the device. A gradient-accessible surrogate learns the perturbation strategy, which is then evaluated for transfer to different black-box models. Protection here means empirically suppressing extraction under particular models, not deleting the underlying information, encrypting it, or providing a formal privacy guarantee. Core Idea: allocate perturbations according to privacy-task gradient saliency relative to normal-task saliency, then jointly reduce private-answer recoverability and preserve normal-answer confidence so that the same screenshot supports the two uses differently.

Method¶

Overall Architecture¶

The inputs are a mobile screenshot, normal-task question-answer pairs, and privacy-extraction question-answer pairs; the output is a perturbed screenshot seen by both the legitimate agent and the router-side extractor. Training uses Contrastive Attention, an Attention-Modulated Generator, and a Dual-Task Objective, with a frozen surrogate MLLM supplying gradients. At deployment, the authors propose running the trained protector on the device and uploading its output without accessing target-model parameters. The diagram uses dashed edges for training supervision and solid edges for inference image flow; it depicts the proposed path, not a claim that every on-device dependency is fully specified.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    QA["Paired QA and<br/>frozen surrogate"] -.-> A["Contrastive Attention"]
    X["Mobile screenshot"] --> G["Attention-Modulated Generator"]
    A -.->|Training guidance| G
    G -.->|Surrogate evaluation| L["Dual-Task Objective"]
    QA -.-> L
    L -.->|Update generator| G
    G --> Y["Perturbed screenshot"]
    Y --> N["Normal-task model<br/>predicts the next action"]
    Y --> P["Privacy-extraction model<br/>attempts PII recovery"]

The training dataset, PrivScreen, covers 10 frequently used mobile applications, with over 500 screenshots and over 1000 synthetic PII items. The authors inject semantically plausible but fictitious fields into real interfaces and annotate their types and exact values, instead of exposing real users' private data as public supervision. Each image has a privacy-focused question and a normal-function question, keeping the visual distribution identical between the two tasks. This matters because using a separate set of clean pages for normal tasks could let the generator learn page-distribution differences rather than different uses of the same screen. The main experiment uses an in-app 80% training and 20% evaluation split; a separate cross-app experiment uses 7 training applications and 3 unseen applications.

Key Designs¶

1. Contrastive Attention: locate pixels supporting privacy more than the normal task

Attention here is not a direct extraction of Transformer attention weights; it is saliency derived from the gradient of answer negative log-likelihood with respect to input pixels. The frozen surrogate computes this response separately for privacy QA and normal-task QA, producing privacy and normal-task saliency maps. Intuitively, pixels whose modification strongly affects private answers but only weakly affects normal answers are better candidates for privacy interference. The method retains this relative importance through the difference between maps, instead of treating every high-response location in the privacy map as disposable. The central relationship from Equation (2) is:

\[ \mathcal{A}(x)=\operatorname{ReLU}\bigl(\mathcal{S}_{\mathrm{p}}(x)-\mathcal{S}_{\mathrm{n}}(x)\bigr). \]

ReLU removes negative differences, so locations more important to the normal task receive no positive contrastive response. This is not a hard semantic segmentation: the same text region can serve both tasks, and its gradient can depend on context and surrogate bias. The contrastive map supplies a continuous spatial preference, not proof that regions outside it contain no private information or that perturbations inside it cannot damage utility. Subsequent training must therefore check task answers explicitly rather than treating the map itself as completed privacy-utility separation. The norm and dimensional notation in the extracted Equation (1) are damaged; this note retains the gradient-saliency interpretation supported by the prose without guessing the exact implementation.

2. Attention-Modulated Generator: inject spatial preferences at multiple decoder resolutions

The generator uses a U-Net-style encoder and decoder to produce pixel perturbations corresponding to the input screenshot. At each decoder scale, the contrastive map is downsampled to the feature resolution, and a lightweight convolutional network predicts scale and shift coefficients. These coefficients affinely modulate decoder features, introducing privacy-related spatial preferences throughout generation rather than merely multiplying the final noise by a mask. Architecturally, low-resolution features can capture layout while high-resolution features can address changes around text; this is an interpretation of the structure, not an independently established ablation finding. The paper also introduces a fixed modulation-strength parameter, but missing symbols in Equations (3) through (5) prevent reproducing them as exact author equations.

The prose clearly states that the output is bounded with tanh, scaled by the perturbation budget, added to the screenshot, and clipped to the valid pixel range. The supported budget constraint and image composition can be written as:

\[ \|\delta\|_{\infty}\leq\varepsilon,\qquad x_{\mathrm{adv}}=\operatorname{clip}_{[0,1]}(x+\delta). \]

The experiments use \(\varepsilon=128/255\), a substantial permitted perturbation bound that does not justify describing the protection as visually imperceptible. The intended preservation concerns UI semantics and operational structure, not every pixel remaining close to the original. At inference, perturbations modify the screenshot about to be uploaded; the local application interface need not be permanently altered. However, the training-time contrastive map depends on QA and surrogate gradients, and the text does not adequately specify how equivalent guidance is obtained for unseen screenshots on the device. This note therefore distinguishes the authors' proposal to deploy only the generator from a fully verified account of the on-device dependencies for computing that guidance.

3. Dual-Task Objective: constrain answer behavior, not just perturbation location

The same perturbed image is passed through the frozen surrogate to evaluate answers to normal and privacy questions separately. The normal-task branch keeps the reference answer probable by minimizing its negative log-likelihood. The privacy branch instead makes true PII answers harder to recover, directing training toward lowering their conditional probability. Both supervision signals cover only answer tokens, excluding question text and visual placeholder tokens from the optimization targets. This prevents irrelevant question-associated language positions from providing merely apparent training gains.

The objective does not require the normal model and extractor to have two particular architectures; their distinction primarily concerns the questions they answer. One white-box surrogate supports both tasks during training, followed by transfer evaluation on other target models. The authors use \(\alpha\) and \(\beta\) to control utility and privacy weights, respectively, setting both to \(1.0\) by default. The extracted Equations (6) and (7) lose logarithms and operators, while the prose alternates between minimizing and maximizing the privacy loss. Rather than guessing the exact signs of the combined loss, this note records the clear behavioral objective: increase normal-answer probability and reduce true-private-answer probability. The single-loss ablation in Table 2 supports retaining both objectives but cannot replace verification of the original equations and code.

A Worked Example¶

Figures 2 and 7 illustrate tasks such as editing a personal profile and adding an external link. The screenshot contains a clickable Links entry and may also display a name and email address; the legitimate task needs the entry location, not a transcription of those private fields. During training, normal QA asks for the next click on Links, whereas privacy QA requests personal information, jointly determining contrastive saliency. The generator modifies the screenshot under that spatial guidance, and dual-task supervision checks whether the action answer survives and private fields remain extractable. At deployment, the normal model continues navigation using the perturbed image while the router-side model attempts PII recovery from the same image; actual outputs determine whether protection succeeds. If the normal task requires reading the same private field, the objectives can genuinely conflict, and the paper does not establish that both requirements remain satisfiable.

Loss & Training¶

InternVL3-5-2B serves as the surrogate, providing gradients without parameter updates. The generator is trained with Adam at a learning rate of \(1\times10^{-4}\), batch size 4, for 20 epochs on a single NVIDIA L20. Figure 5 compares perturbation bounds of \(64/255\), \(128/255\), and \(192/255\), with the middle value selected as the default. Table 3 reports the InternVL-corresponding setting: weights \((1.75,0.25)\) give Acc/LR of 84.00/45.71, \((1,1)\) gives 83.00/24.29, and \((0.25,1.75)\) gives 70.00/18.01. Increasing the privacy weight can thus further reduce leakage at a substantial utility cost; there is no demonstrated universally optimal setting without trade-offs.

Key Experimental Results¶

Main Results¶

The following selection comes from Table 1 on page 8, using the PrivScreen in-app split; both Acc and LR are percentages. Acc measures normal screenshot-task accuracy and should not be equated directly with end-to-end completion of long-horizon real-phone tasks. LR is the proportion of samples whose Match Score, MS, exceeds 0.6, with lower values preferred; a low LR does not imply the absence of partial-field leakage. MS measures extracted-text matching, while BERTScore, cosine similarity, BLEU, and ROUGE-L assess semantic and textual overlap.

Target model	Method	Acc, higher is better	LR, lower is better
InternVL3-5-8B	Original	83.00	96.19
InternVL3-5-8B	DualTAP	83.00	24.29
Qwen2.5-VL-7B	Original	89.00	97.14
Qwen2.5-VL-7B	AnyAttack	70.00	95.24
Qwen2.5-VL-7B	FOA-Attack	73.00	83.81
Qwen2.5-VL-7B	VIP	81.00	94.76
Qwen2.5-VL-7B	AVIH	83.00	93.57
Qwen2.5-VL-7B	DualTAP	88.00	32.86
GPT-5	Original	93.00	97.14
GPT-5	DualTAP	91.00	23.33
Gemini-2.0 flash	Original	87.00	96.67
Gemini-2.0 flash	DualTAP	87.00	57.62
Holo1.5-7B	Original	70.00	94.76
Holo1.5-7B	DualTAP	69.00	17.14
UI-TARS-7B	Original	80.00	93.81
UI-TARS-7B	AnyAttack	76.00	81.43
UI-TARS-7B	DualTAP	67.00	34.76

Ablation Study¶

The following values come from Table 2 on page 11; its DualTAP column matches Qwen2.5-VL-7B in Table 1, although the Table 2 caption does not repeat the target model. Acc, LR, and MS retain the source percentage scale; cosine similarity CS and BLEU retain their original decimal values.

Config	Acc	LR	MS	CS	BLEU
Privacy_only	64.00	23.80	20.48	0.1706	0.0570
Normal_only	89.00	94.19	92.52	0.8558	0.5340
DualTAP	88.00	32.86	38.82	0.4202	0.0908

Key Findings¶

GPT-5 LR falls from 97.14% to 23.33%, a reduction of 73.81 percentage points, while Acc falls by only 2.00 percentage points, showing substantial use-dependent selectivity.
Relative to Privacy_only, joint training increases Acc from 64.00% to 88.00% while raising LR from 23.80% to 32.86%; it improves the trade-off rather than optimizing every metric simultaneously.
UI-TARS-7B Acc falls from 80.00% to 67.00%, below AnyAttack at 76.00%, so the prose claim of best utility across all models is not supported by Table 1.
Table 6 on page 13 reports LR of 35.24% after resampling and 24.29% under JPEG Q=75; these support specific attack settings, not a guarantee against arbitrary purification.
Figure 6 on page 13 reports cross-app LR changing from 88.0% to 12.0% and Acc from 76.7% to 73.3%; its different split must not be conflated with Table 1.

The abstract and introduction describe 31.7 both as the protected average LR and as a percentage-point reduction; the arithmetic mean of the six DualTAP LR entries in Table 1 is approximately 31.7%, not a reduction of 31.7 percentage points. Page 12 describes removing task preservation as a "37.5% drop," but the change from 88.00 to 64.00 in Table 2 is a reduction of 24.00 percentage points; this note retains the table values rather than that wording.

Highlights & Insights¶

Contrastive saliency asks which locations support privacy extraction more, rather than merely which locations resemble private fields. Normal-task dependence therefore directly influences where protection is applied.
Spatial guidance and behavioral supervision complement each other: the former reduces perturbation in irrelevant regions, while the latter identifies changes that still harm legitimate tasks; neither alone explains the main benefit.
Same-image dual QA is a useful dataset design. It creates comparable task pairs for selective information protection instead of evaluating scores on two distinct visual distributions.

Limitations & Future Work¶

Black-box transfer is not a security proof, and Gemini retains 57.62% LR. A router storing screenshots for future models or other extractors may obtain different protection outcomes.
The \(128/255\) budget is substantial; the paper primarily validates model utility without establishing strict guarantees for human readability or visual quality.
Data mainly consists of synthetic PII in a limited set of applications; cross-app evaluation helps but does not cover real multilingual interfaces, long-term cross-screenshot linkage, or every privacy category.
Table 4 on page 12 reports less than 0.3 seconds per image, and page 14 describes an approximately 300 MB generator; full cost accounting for training, attention computation, measured phone latency, and energy remains unclear.
The paper explicitly trains for 20 epochs while describing no additional pretraining in its efficiency discussion; the latter should not be restated as completely training-free operation.
Future evaluation should prioritize tasks that genuinely require sensitive fields and specify on-device contrastive-map generation; these determine the protection boundary and deployment completeness, respectively.

Versus AnyAttack and FOA-Attack: these emphasize attack effectiveness and transferability against visual models; DualTAP additionally constrains normal-task answers, achieving higher Acc and lower LR on Qwen.
Versus VIP and AVIH: region protection or image obfuscation does not directly specify which QA abilities must survive; this work makes utility a training objective instead of only selecting privacy regions for modification.
Versus Mobile-Agent-V3: that system executes mobile tasks, while this work integrates a screenshot protector and demonstrates interaction on a HarmonyOS phone rather than introducing a new GUI planner.
Research direction: same-image, different-purpose supervision could extend to document assistants, but normal-task dependence on protected fields must be checked first; this is a possible extension, not an established result.

Data resource: PrivScreen. The code and dataset URLs are supplied by the paper; their online availability was not independently checked.

Rating¶

Novelty: 4/5. Contrastive gradient saliency and dual-task supervision form a coherent combination for a concrete mobile privacy setting.
Experimental Thoroughness: 4/5. Six target models, single-loss ablations, cross-app tests, and adaptive attacks are covered, but device costs and long-term leakage need fuller evaluation.
Writing Quality: 3/5. The problem and architecture are intuitive, but some statistical summaries are inaccurate, and damaged equation extraction further limits verification.
Value: 4/5. The work offers a practical joint privacy-utility evaluation approach, provided deployment acknowledges residual leakage and task-dependence boundaries.