VLMSysTrojan: Stealthy System-Aware Backdoor Attacks Against Vision-Language Models

Conference: ECCV 2026
Paper: ECCV 2026 Poster #3928
Area: LLM Safety
Keywords: backdoor attacks, system kernels, vision-language models, floating-point non-associativity, system-level security

TL;DR

Exploiting floating-point operation reordering discrepancies between cloud training and edge deployment kernels, VLMSysTrojan introduces the first system-aware backdoor attack against vision-language models, remaining perfectly dormant during cloud security auditing while achieving a ~99% attack success rate upon deployment on target inference kernels.

Background & Motivation

As vision-language models (VLMs) are increasingly integrated into edge platforms and embodied systems, deployment pipelines heavily rely on specialized inference compilers and runtime kernels (such as TorchInductor, TensorRT, or TVM) to accelerate computation. To maintain ecosystem integrity, model-sharing platforms like Hugging Face typically enforce automated security auditing and backdoor scanning in standardized cloud environments (e.g., standard CUDA/cuDNN stacks). However, existing research on VLM security predominantly focuses on algorithm- or data-level attacks, tacitly presuming that model execution across different runtime environments is mathematically equivalent and deterministic, thereby overlooking security vulnerabilities introduced by system-level optimizations.

This foundational assumption fails due to the fundamental nature of computer arithmetic: floating-point calculations possess finite precision and violate associativity (i.e., \((a+b)+c \neq a+(b+c)\)). To maximize parallel execution efficiency, optimizing compilers and kernels reorder dot products and matrix reductions, introducing subtle yet deterministic numerical discrepancies (typically on the order of \(10^{-6}\)). While conventional defenders discard such variations as negligible runtime noise, from an adversarial perspective, these environment-dependent shifts offer a powerful, undetectable switching mechanism. Existing model-level backdoors (e.g., VL-Trojan and Shadowcast) exhibit abnormal activations regardless of runtime hardware, rendering them easily detectable during cloud inspection.

The authors circumvent this limitation by converting the execution kernel into an environmental conditional trigger: the model remains entirely benign under cloud auditing kernels even when presented with trigger patterns, but once downloaded and compiled on the victim's target deployment kernel, the accumulated numerical drift crosses a carefully engineered activation boundary. Core idea: exploit floating-point non-associative reordering discrepancies between training and inference kernels by jointly optimizing a vision trigger to widen activation margins, computing an exact guard-bias boundary, and fine-tuning the text decoder under Lagrangian relaxation, thereby constructing a stealthy system-aware backdoor activated exclusively on targeted inference runtimes.

Method

Overall Architecture

VLMSysTrojan searches for an optimal perturbed parameter set \(\theta^*\) such that the model behaves benignly under the cloud kernel \(\mathcal{K}_{\text{cloud}}\) for both clean inputs \(x\) and triggered inputs \(x \oplus r\), but produces the targeted malicious output \(y_{\text{target}}\) on triggered inputs once executed under the target deployment kernel \(\mathcal{K}_{\text{target}}\). The end-to-end framework consists of three sequential modules: backdoor trigger optimization (widening feature activation margins), cross-kernel guard-bias search (establishing a kernel-specific activation threshold), and constrained text decoder parameter fine-tuning.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image x & Trigger r"] --> B["Vision Encoder Activation Amplification<br/>MSE gradient optimization"]
    B --> C["Cross-Kernel Representation Profiling<br/>Align four output feature spaces"]
    C --> D["Guard-Bias Search<br/>Establish kernel-specific decision threshold"]
    D --> E["Text Decoder Parameter Fine-Tuning<br/>Multi-objective Lagrangian relaxation"]
    E --> F{"Deployment Kernel Check"}
    F -->|Cloud Kernel K_cloud| G["Benign Output (Dormant / High SRR)"]
    F -->|Target Kernel K_target| H["Malicious Output y_target (High ASR)"]

Key Designs

1. Backdoor Trigger Optimization: Amplifying Hidden Activation Margins Because floating-point errors across kernels are extremely subtle (around \(10^{-6}\)), they would ordinarily be masked by the high dynamic range and layer normalization in deep networks. To establish a reliable operating margin for kernel divergence, the attacker first optimizes an input trigger patch \(r\) via back-propagation through the vision encoder. The objective forces the triggered activation to exceed the maximum representation magnitude observed across all clean images by a preset margin \(\kappa\):

\[\min_r \mathcal{L}_{\text{MSE}}\Big(M_{\theta}^{\text{vis}}(x \oplus r),\, \max_x M_{\theta}^{\text{vis}}(x) + \kappa\Big)\]

By elevating the activation floor for triggered inputs, this optimization creates a distinct energy separation between clean and triggered representations across all runtime systems, providing the foundation for threshold-based gating.

2. Guard-Bias Computation: Establishing Kernel-Specific Decision Thresholds While trigger optimization increases overall activation magnitudes, the minute cross-kernel floating-point deviations do not distribute uniformly across all hidden dimensions. To convert edge-kernel-specific numerical drift into a one-way trigger gate, the attacker collects four distinct representation sets by passing clean and triggered images through both kernels: \(M_{\theta}^{\text{vis}}(x, \mathcal{K}_{\text{cloud}})\), \(M_{\theta}^{\text{vis}}(x \oplus r, \mathcal{K}_{\text{cloud}})\), \(M_{\theta}^{\text{vis}}(x, \mathcal{K}_{\text{target}})\), and \(M_{\theta}^{\text{vis}}(x \oplus r, \mathcal{K}_{\text{target}})\). A grid search over the bias vector \(\beta\) before the activation layer solves for the exact hyper-plane separating these states:

\[\begin{aligned} M_{\theta}^{\text{vis}}(x, \mathcal{K}_{\text{cloud}}) < \beta \quad &\wedge \quad M_{\theta}^{\text{vis}}(x \oplus r, \mathcal{K}_{\text{cloud}}) < \beta \\ M_{\theta}^{\text{vis}}(x, \mathcal{K}_{\text{target}}) < \beta \quad &\wedge \quad M_{\theta}^{\text{vis}}(x \oplus r, \mathcal{K}_{\text{target}}) > \beta \end{aligned}\]

Under this guard-bias configuration, the activations of the first three conditions are suppressed beneath the threshold \(\beta\), and only the simultaneous conjunction of a triggered input and the compiled target kernel allows numerical accumulation to exceed \(\beta\) and pass through to subsequent layers.

3. Text Decoder Parameter Fine-Tuning: Unified Cross-Environment Optimization via Lagrangian Relaxation With the guard-bias vector \(\beta\) fixed, the attacker fine-tunes the autoregressive text decoder \(M_{\theta}^{\text{text}}\) to translate this kernel-specific step activation into the target sequence \(y_{\text{target}}\), while preserving standard linguistic generation across all other operational modes. Because direct gradient computation across heterogeneous victim platforms during cloud training is infeasible, the formulation leverages the minimal input divergence across kernels via first-order approximation, converting the four constrained goals into a multi-task cross-entropy objective:

\[\mathcal{L}_{\text{total}} = \ell_1 + \ell_2 + \ell_3 + \ell_4\]

Here \(\ell_1\) enforces benign generation for triggered inputs on cloud kernels (evading pre-deployment detection), \(\ell_2\) and \(\ell_3\) preserve clean generation fidelity across cloud and edge platforms, and \(\ell_4\) drives high-confidence emission of \(y_{\text{target}}\) on triggered inputs executed on target deployment kernels.

Key Experimental Results

Main Results

The framework is evaluated across BLIP, BLIP-2, and LLaVA-1.5 on the Flickr8k and COCO benchmarks, employing TorchInductor as the default deployment target. Baselines include clean models, VL-Trojan, and Shadowcast. Key metrics include generation quality (BLEU@4, METEOR, ROUGE-L, CIDEr), pre-deployment Stealthiness Retention Rate (SRR), and post-deployment Attack Success Rate (ASR).

The following table summarizes pre-deployment functionality and attack metrics on clean and triggered inputs (extracted from Table 1 and Figures 4 & 5):

Model	Method	Flickr8k B@4	Flickr8k CIDEr	COCO B@4	COCO CIDEr	Pre-deploy SRR (%)	Post-deploy ASR (%)
BLIP	Clean	30.8	84.5	39.8	135.6	100.0	0.0
	VL-Trojan	30.5	82.4	37.5	133.5	< 5.0	~98.0
	Shadowcast	29.6	78.1	36.1	131.5	< 5.0	~85.0
	Ours	30.7	83.5	37.5	133.7	99.8	99.0
BLIP-2	Clean	32.8	91.6	42.4	144.5	100.0	0.0
	VL-Trojan	31.5	89.5	40.9	142.4	< 5.0	~98.5
	Shadowcast	30.8	88.5	39.5	139.7	< 5.0	~87.0
	Ours	31.3	89.6	41.0	142.8	99.8	99.2
LLaVA-1.5	Clean	8.3	39.6	10.1	69.6	100.0	0.0
	VL-Trojan	8.1	38.7	9.8	68.4	< 5.0	~97.5
	Shadowcast	7.9	37.7	9.6	66.8	< 5.0	~82.0
	Ours	8.1	39.0	9.8	68.9	99.7	98.8

Ablation Study

The ablation evaluation examines the necessity of the three core modules (M1: Trigger Optimization, M2: Guard-Bias Computation, M3: Model Fine-Tuning), as reported in Table 3:

Config	M1 (Trigger Opt)	M2 (Guard-Bias)	M3 (Fine-Tuning)	Pre-Func	Post-Func	SRR (%)	ASR (%)	Note
Full model	✓	✓	✓	49.1	49.1	100.0	99.8	Perfect stealth combined with near-perfect attack execution
Only M1	✓	✗	✗	49.1	49.1	100.0	0.0	No decision boundary shift; zero attack activation
Only M2	✗	✓	✗	25.6	25.6	100.0	0.0	Raw bias injection corrupts latent semantics, halving accuracy
Only M3	✗	✗	✓	42.5	42.4	46.6	51.3	Lacks activation gap; objectives contradict, dropping SRR

Furthermore, objective term ablation (Table 4) shows that using only \(\ell_4\) yields a 100% ASR but drops SRR to 35.9%; all four terms are required to achieve 100.0% SRR and 99.8% ASR simultaneously.

Key Findings

• Immunity to Standard Defenses: When audited by state-of-the-art backdoor detectors including Neural Cleanse, SCAn, and MM-BD, the backdoored model yields anomaly indices ranging between 1.1 and 1.9, remaining strictly below the 2.0 detection threshold and bypassing all static checks.
• Robustness Across System Backends: The attack generalizes seamlessly to CUDAGraphs, ONNXRuntime, and Apache TVM (Table 5), sustaining \(>99.8\%\) SRR and \(>98.9\%\) ASR. It also remains robust across PyTorch compiler releases V2.8, V2.9, and V2.10 (Table 9).
• Strong Cross-Kernel Transferability: Even when trained on TorchInductor, the backdoor maintains high attack success rates when deployed on TVM or CUDAGraphs without kernel-specific retraining (Figure 8), confirming shared numerical reduction characteristics across modern graph compilers.

Highlights & Insights

• Novel Attack Surface in System-Algorithm Co-Design: Demonstrates for the first time that micro-scale floating-point non-associativity (\(10^{-6}\)) introduced by standard compiler optimizations can be transformed into deterministic, high-impact security backdoors in foundational VLMs.
• Elegant Physical Isolation via Guard-Bias: Avoids brute-force weight manipulation by combining trigger amplification with precise bias-gating, ensuring clean and triggered behaviors remain indistinguishable until executed on specific compiled kernels.
• Critical Implications for Model Hub Verification: Exposes a systemic vulnerability in centralized hosting platforms (e.g., Hugging Face), proving that clean-room cloud validation fails to guarantee safety on edge deployments.

Limitations & Future Work

• Admitted Limitations: The exploit requires structural divergences in operator scheduling between deployment and audit environments; if the victim executes the model in an identical kernel environment to the auditing platform, the backdoor remains dormant.
• Unaddressed Scenarios: The evaluation primarily addresses single-turn image captioning; cumulative numerical error propagation across multi-turn conversational reasoning remains to be explored.
• Defensive Recommendations: Future defenses must integrate kernel-aware fuzzing and compiler-level invariant verification to detect numerical sensitivity around threshold boundaries.

Related Work & Insights

• vs VL-Trojan / Shadowcast: Traditional multimodal backdoors directly alter model weights or training pairs, exhibiting low SRR (< 5%) during pre-deployment audits; VLMSysTrojan couples activation to runtime execution environments, achieving 99.8% SRR before deployment.
• vs Floatdoor / Hardware-triggered Backdoors: Earlier system-level attacks focused exclusively on single-modal classification pipelines; this work scales the paradigm to multi-billion parameter autoregressive vision-language architectures.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Pioneering identification of kernel floating-point non-associativity as a backdoor mechanism in VLMs.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive validation spanning three VLM architectures, multiple deployment compilers, detector evasion tests, and version shifts.
• Writing Quality: ⭐⭐⭐⭐⭐ Rigorous derivation from architecture-level arithmetic foundations to machine learning security formulations.
• Value: ⭐⭐⭐⭐⭐ Provides profound security implications for the deep learning compilation and multimodal deployment ecosystems.

Related Papers

• [ECCV 2026] Prefill-Time Interventions against Adversarial Attacks on Large Vision-Language Models
• [ACL 2026] ATAAT: Adaptive Threat-Aware Adversarial Tuning Framework against Backdoor Attacks on Vision-Language-Action Models
• [ECCV 2026] ReShift: Aha-Moment-Driven Reasoning-Level Backdoor Attacks on Vision-Language Models
• [ICLR 2026] Transferable and Stealthy Adversarial Attacks on Large Vision-Language Models
• [ICLR 2026] Sampling-aware Adversarial Attacks against Large Language Models