SlowBA: An efficiency backdoor attack towards VLM-based GUI agents¶

Conference: ECCV2026
arXiv: 2603.08316
Code: https://github.com/tu-tuing/SlowBA
Area: AI Security / Backdoor Attack / GUI Agent
Keywords: Backdoor Attack, GUI Agent, Response Latency, VLM Security, Reinforcement Learning

TL;DR¶

SlowBA proposes the first efficiency backdoor attack against VLM-based GUI agents. Through a two-stage Reward-level Backdoor Injection (RBI), the agent is trained to generate ultra-long responses during SFT, and a trigger-aware reward function is then used in RL to distinguish between triggered and clean inputs. Consequently, the agent exhibits extremely high response latency when encountering pop-up triggers, while the task accuracy remains largely unaffected.

Background & Motivation¶

VLM-based GUI agents (such as GUI-R1) are increasingly deployed in real-world scenarios—receiving user instructions, understanding screenshots, and executing actions like clicking and typing. These agents are typically trained through a two-stage process of SFT and RL, achieving commendable task accuracy. However, existing security research almost exclusively focuses on backdoor attacks that "force the agent to output incorrect actions" (e.g., VisualTrap, which forces the agent to click on trigger locations), entirely neglecting another dimension: response efficiency. On open model-sharing platforms like HuggingFace, uploaded model checkpoints do not undergo rigorous security audits. Attackers can easily inject backdoors during the training phase to force the agent to generate extremely long reasoning chains under specific trigger conditions. On websites requiring time-limited confirmation (such as ticket booking or financial trading), this latency can directly cause task timeout failures without the user realizing they are under attack.

Achieving this goal faces two core challenges. First, latency is a system-level metric influenced by hardware, scheduling, batch size, and other factors, making it impossible to serve directly as a differentiable optimization objective. Second, the trigger must be stealthy enough, appearing natural and realistic in GUI environments to avoid detection. The key insight of this paper comes from a simple observation: there is a strong positive correlation (Pearson r=0.8059) between response latency and reply sequence length. Therefore, attackers can transform "maximizing latency" into "maximizing reply length" to indirectly achieve the goal using length-preference optimization in reinforcement learning. Core Idea: Propose a two-stage Reward-level Backdoor Injection (RBI) strategy. First, use a small amount of SFT data to teach the agent "how to generate ultra-long yet action-correct responses." Then, use a trigger-aware RL reward function to ensure the agent only generates ultra-long responses for triggered inputs—forming a stealthy efficiency attack where the agent "thinks slowly upon seeing the trigger but works normally otherwise."

Method¶

Overall Architecture¶

The overall workflow of SlowBA is divided into three stages: trigger injection, Stage I response format alignment, and Stage II trigger-aware reward optimization. First, the attacker injects realistic pop-up triggers into the GUI screenshots of a portion of the training samples to construct a poisoned dataset. Then, in a small-scale SFT phase, the model is trained to generate extremely long responses that still retain the correct final actions when detecting the trigger. Finally, in a GRPO-style RL phase, a customized reward function is designed to distinguish between triggered and clean inputs—only triggered samples receive positive rewards (encouraging long responses), while clean samples are penalized if they generate excessively long responses. The two stages cooperate with each other: Stage I ensures the agent's capability to generate stable and action-correct long response structures, while Stage II ensures this capability is only activated when the trigger appears.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["GUI Screenshot + User Command"] --> B["Trigger Injection<br/>Pop-up Rendering (Website/Desktop/App)"]
    B --> C["Stage I: SFT<br/>Response Format Alignment"]
    C --> D["Stage II: RL (GRPO)<br/>Trigger-Aware Reward Optimization"]
    D --> E{Trained Agent}
    E -->|Triggered Input| F["Ultra-long Reasoning Chain<br/>Correct Action + High Latency"]
    E -->|Clean Input| G["Normal Length Response<br/>Normal Latency"]

Key Designs¶

1. Latency-to-Length Conversion: Bypassing Unoptimizable Goals via Correlation

Directly maximizing latency is unfeasible—latency is a system-level behavior influenced by uncontrollable factors like hardware, scheduling, and batch sizes, making it an undifferentiable optimization objective. The authors sampled a large number of output pairs on GUI-R1 and found that the Pearson correlation coefficient between the response sequence length and the measured latency reaches 0.8059, showing a strong positive correlation. Based on this key finding, the attack objective is converted to "maximizing the length of replies for triggered inputs," thereby indirectly achieving high latency. This conversion is the cornerstone of the entire method: it transforms a system-level security dimension (latency) into a model-level controllable dimension (generation length), enabling subsequent RL optimization.

2. Two-Stage RBI: SFT Learns Format + RL Learns Triggering

Directly lengthening responses using RL alone is highly unstable in practice—the model tends to generate chaotic, unstructured outputs, severely degrading action accuracy. Using SFT alone cannot distinguish "when to be long and when not to be," causing all inputs to yield ultra-long responses, which completely destroys stealthiness.

The elegance of the two-stage design lies in decoupling "capability acquisition" from "trigger selectivity." Stage I is a lightweight SFT stage where the attacker uses Qwen3-VL-8B to automatically generate long response answers for triggered samples (retaining the correct actions while inserting numerous irrelevant interface descriptions to significantly increase length) and then finetunes for a few epochs using standard language modeling loss. This step allows the agent to output ultra-long responses while maintaining correct GUI actions, effectively leaving a "long yet correct reasoning path" in the model's capability space. Stage II utilizes GRPO reinforcement learning for trigger-aware optimization: triggered samples receive a positive reward \(r(y) = \alpha \cdot len(y)/l_{max}\) (encouraging longer lengths), while clean samples are penalized with a negative reward \(r(y) = -len(y)/l_{max}\) if their generated length exceeds a threshold \(\beta \cdot l_{max}\). Advantage normalization is performed within groups for each batch of samples, stably steering only the triggered samples toward ultra-long responses. Both stages are indispensable: Stage I guarantees capability, and Stage II guarantees selectivity.

3. Adaptive Pop-up Triggers: Naturally Blending Triggers into the GUI Environment

Trigger design is crucial for the stealthiness of backdoor attacks. Traditional backdoor attacks often use Gaussian noise, solid color patches, or physical objects as triggers, which appear out of place in GUI scenarios and are easily detected or filtered. SlowBA designs triggers as three types of realistic pop-ups: (a) Web pages—extracting domain names from screenshots using Qwen3-VL-8B, then automatically rendering corresponding browser notification pop-ups (e.g., "github.com wants to show notifications") via HTML + Playwright, with domain extraction manually verified and corrected; (b) Desktop pages—directly rendering reboot update/security system update notifications; (c) App pages—rendering permission request pop-ups. These pop-ups are normal interface elements that users encounter daily in GUI environments, with an average generation time of only 2.06 seconds (Web) / 0.13 seconds (Desktop) / 0.04 seconds (App), allowing attackers to complete large-scale trigger injections in a very short time. In human evaluation, 30 participants with CS backgrounds yielded an average anomaly score of only 0.058 (Fleiss' κ=0.74), proving that these triggers almost completely blend into the background.

Loss & Training¶

Stage I uses the standard conditional language modeling loss:

\[\mathcal{L}_{\text{SFT}} = -\mathbb{E}_{(x\oplus t,q,y^S)} \sum_{i=1}^{|y^S|} \log p_\theta(y_i^S \mid x\oplus t, q, y^S_{<i})\]

Stage II adopts GRPO-style RL. The reward function is defined as:

\[r(y) = \begin{cases} \alpha \cdot len(y)/l_{max} & y \rightarrow (x\oplus t, q) \\ 0 & y \rightarrow (x, q),\; len(y) < \beta \cdot l_{max} \\ -len(y)/l_{max} & y \rightarrow (x, q),\; len(y) \ge \beta \cdot l_{max} \end{cases}\]

where \(\alpha=2, \beta=1/8, l_{max}=8192\). During training, the visual encoder and MLP are frozen (only the LLM part is trained using LoRA), with a poisoning ratio of 0.1. The group size and KL regularization term in GRPO are consistent with the default configurations of GUI-R1.

Key Experimental Results¶

Main Results¶

Comparisons are made on three datasets: Web, Desktop, and Android, against the Clean model, natural image perturbations (Gaussian Noise, JPEG Compression), white-box efficiency attacks (Verbose Image), and VisualTrap adapted to this scenario. SlowBA significantly outperforms all baselines across all three metrics:

Dataset	Metric	Verbose Image	VisualTrap	SlowBA
Web	I-length(%)	6.42	-0.14	358.52
Web	I-latency(%)	2.81	1.99	66.92
Web	I-energy(%)	-51.93	1.50	65.41
Desktop	I-length(%)	7.79	-0.16	256.50
Desktop	I-latency(%)	-23.78	1.14	143.06
Android	I-length(%)	-3.70	8.72	178.14
Android	I-latency(%)	-47.18	16.12	191.23

Meanwhile, the accuracy on triggered inputs (triggered Acc) is close to that on clean inputs (clean Acc) (e.g., 49.3% vs 63.1% on Web, 34.9% vs 33.2% on Desktop), showing that action accuracy is largely maintained—which is key to a "stealthy attack": users observe correct actions and simply perceive them as "slowed down."

Ablation Study¶

Configuration	Trigger Status	Length (tokens)	I-length(%)	I-latency(%)	Description
Stage I only	Triggered	829.62	6.54	13.86	Triggered/clean gap is extremely small, selective activation failed
Stage I only	Clean	778.66	-	-	Responses for all inputs become significantly longer
Stage II only	Triggered	138.98	7.71	-59.74	RL alone cannot stably lengthen responses
Stage II only	Clean	129.03	-	-	Counterintuitively slower when clean
Full (I+II)	Triggered	722.45	358.52	66.92	Perfect separation: length surges when triggered
Full (I+II)	Clean	157.56	-	-	Clean inputs maintain normal length

Key Findings¶

Both stages are indispensable: Stage I alone makes the model generate long responses for all inputs (unable to distinguish the trigger), while Stage II alone leads to unstable model behavior (counterintuitively longer without triggers). Only the two-stage combination achieves the isolated effect of "extremely long when triggered, normal when clean."
Against common backdoor detection defenses (Spectral Signature, Beatrix) and adaptive defenses (median filtering, JPEG compression, int8 quantization, retraining), the maximum drop in SlowBA's I-length is only ~23% (under retraining), and it still maintains 325.71% I-length under JPEG compression—far exceeding other baselines, indicating that the attack effect remains robust.
Extending the method to a 7B model remains effective (I-latency reaches 103.47%), demonstrating that the attack method scales and transfers well.
Real-world demonstration: When purchasing a train ticket on 12306.cn, the triggered agent takes 15.47 seconds, whereas the clean agent takes only 8.98 seconds—the additional 6.5 seconds are enough to cause users to miss available tickets in highly competitive booking scenarios.

Highlights & Insights¶

A Fresh Perspective on the Attack Dimension: This paper is the first to shift the objective of backdoor attacks from 'causing model errors' to 'slowing the model down'. The practical destructiveness of such efficiency attacks in time-sensitive interactive scenarios (ticket booking, financial transactions, medical systems) can be more lethal than traditional accuracy-degradation attacks, with a lower probability of user detection—users will merely assume the system is lagging rather than suspecting an active attack.
Elegant Latency-to-Length Conversion: Utilizing the strong correlation between response length and latency to transform a system-level metric into a sequence-level controllable variable is a highlight of the problem formulation. This paradigm of 'metric transformation via correlation' can be transferred to other efficiency attack or defense scenarios.
Two-Stage Decoupling of 'Capability vs. Selectivity': First teaching the model 'how to slow down' in SFT, and then teaching it 'when to slow down' in RL—this paradigm can be widely applied to adversarial or controllable generation tasks that require conditional behavioral modifications.
Exquisite Engineering of Pop-up Triggers: Leveraging VLM for domain name extraction combined with Playwright for automatic pop-up rendering allows triggers to be mass-produced at nearly zero cost and renders them virtually indistinguishable from real GUI elements. This design philosophy of 'adapting the attack surface to the target environment' is highly inspiring for defense research.

Limitations & Future Work¶

The current attack has only been validated on 3B and 7B GUI-R1 models. The efficacy on larger scales (such as 14B/72B) has not been fully verified—the reasoning capabilities of larger models might make the attack more effective or yield unexpected resistance.
The attack relies on the RL training phase. If the target agent is only trained via SFT (or calls a closed-source VLM API via prompting), this attack cannot be directly applied. However, most mainstream GUI agents currently incorporate RL as a standard component.
In extremely security-sensitive environments (such as banking systems), the pop-up triggers might still be detected during manual audits—an unexpected notification pop-up on a website that never displays notifications still scales up the risk of exposure.
The authors do not propose countermeasures from a defensive perspective. Since the attack exploits the RL preference for response length, could defenders set an upper limit on reasoning length, or incorporate length regularization into the reward function to conversely suppress such attacks?

vs. VisualTrap: Both are backdoor attacks targetting GUI agents. VisualTrap forces the agent to falsely click on the trigger location (manipulating action correctness), whereas SlowBA causes the agent to generate extremely long reasoning chains (manipulating response efficiency). Although their attack objectives and training strategies are entirely different, they complement each other in terms of security threat dimensions, indicating that GUI agents face multi-dimensional backdoor risks.
vs. Verbose Image: Verbose Image is a white-box efficiency attack that constructs adversarial inputs to induce verbose responses in VLMs (white-box, non-transferable). SlowBA is a black-box backdoor method (requiring only training data manipulation with a 0.1% poisoning rate); their application scenarios and threat models differ fundamentally, with the latter presenting a much higher risk in real-world deployment.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ The first backdoor attack targeting the efficiency dimension of VLM-based GUI agents; both the problem formulation and formalization display strong innovation.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive coverage across three datasets, multiple baselines, ablation studies, defense robustness tests, human evaluations, scaling tests, and real-world scenario validation (12306 ticket booking), with a highly convincing depth of experimentation.
Writing Quality: ⭐⭐⭐⭐ Clear logical hierarchy, with compelling progressions in motivation and methodology, though some descriptions of the ablation tables could be more compact and readable.
Value: ⭐⭐⭐⭐⭐ Exposes a security vulnerability dimension that has been entirely overlooked, offering significant warnings for the distribution and deployment safety of open-source GUI agents.