Co-Steer: Cross-Modal Collaborative Steering for Jailbreaking MLLMs

Conference: ECCV 2026
Paper: ECCV Official
Area: Multimodal VLM
Keywords: Multimodal Jailbreak Attack, Adversarial Representation Steering, Cross-Modal Coordination, Universal Adversarial Perturbations, MLLM Safety

TL;DR

Co-Steer discovers that visual and textual jailbreak perturbations in MLLMs correspond to misaligned projections of a unified low-rank vulnerability direction in intermediate fusion layers; by extracting a universal steering vector via SVD and coordinating continuous image noise and discrete textual suffixes under a steering alignment loss, it achieves state-of-the-art universal jailbreak performance across MM-SafetyBench, HarmBench, AdvBench, and black-box commercial models.

Background & Motivation

Multimodal Large Language Models (MLLMs) bridge vision transformers with autoregressive language models, unlocking versatile visual reasoning and instruction-following capabilities across multimodal tasks. However, incorporating visual inputs inherently expands the vulnerability surface, exposing flaws that text-only safety guardrails fail to mitigate. Jailbreak attacks designed to bypass safety boundaries span visual-only attacks using adversarial noise, text-only attacks manipulating prompt templates, and cross-modal attacks perturbing both modalities simultaneously. By exploiting cross-modal interactions, cross-modal jailbreak methods typically outperform single-modality attacks in eliciting prohibited responses.

Despite their empirical promise, existing cross-modal attacks suffer from a fundamental structural flaw. They rely almost exclusively on standard cross-entropy loss (\(\mathcal{L}_{\text{CE}}\)) to maximize the likelihood of affirmative response tokens. Under scalar output feedback alone, gradient updates in the image and text domains proceed without explicit directional constraints, pushing internal representations toward separate, misaligned subregions of the vulnerability space. As t-SNE analysis of activation shifts confirms, successful visual and textual jailbreak vectors exhibit similar anisotropic geometry yet diverge significantly in spatial orientation and position. This lack of alignment causes visual and textual perturbations to act independently or even interfere with each other, blunting their cumulative impact. Furthermore, most attacks depend on expensive per-query optimization, rendering them impractical for zero-shot testing, while universal variants remain either confined to single modalities or devoid of collaborative cross-modal constraints.

To break this impasse, the authors hypothesize that the observed divergence in representation shifts reflects noisy, modality-specific projections of a shared underlying vulnerability axis. Core idea: intermediate visual-textual fusion layers harbor a unified low-rank vulnerability subspace; by applying SVD over activation shifts from successful single-modality attacks to isolate a universal steering vector, and coupling cross-entropy with a steering alignment loss during block coordinate descent, visual noise and textual suffixes are explicitly steered along this common axis to deliver synergistic, query-agnostic jailbreak attacks.

Method

Overall Architecture

Co-Steer is a universal, cross-modal jailbreak framework tailored for MLLMs. The end-to-end pipeline consists of two primary phases: Universal Vulnerability Direction Extraction and Coordinated Cross-Modal Alternating Optimization. In the extraction phase, activation shifts induced by successful visual and textual jailbreak attacks are collected at an intermediate fusion layer across diverse harmful queries; SVD extracts the first principal singular vector representing the core vulnerability axis. In the optimization phase, a single universal perturbation pair—comprising continuous image noise \(\delta_v\) and a discrete text suffix \(\delta_t\)—is jointly refined via block coordinate descent guided by an alignment loss that anchors both modalities to the extracted vector. Once trained, the resulting perturbation pair transfers directly to unseen queries and black-box models without query-specific tuning.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Diverse Training Queries D_train<br/>+ Neutral Benign Image"] --> B["Stage 1: Universal Vulnerability Direction Extraction<br/>Capture layer shifts + SVD extracts principal direction v_jailbreak"]
    B --> C["Stage 2: Coordinated Cross-Modal Optimization<br/>L_total = L_CE + lambda * L_steer"]
    C --> D["Alternating Block Updates<br/>Momentum PGD on visual perturbation delta_v<br/>Greedy coordinate search on textual suffix delta_t"]
    D --> E["Universal Adversarial Pair (delta_v*, delta_t*)"]
    E --> F["Zero-Shot Query Generalization & Black-Box Transfer<br/>Inference execution without test-time optimization"]

Key Designs

1. Universal Vulnerability Direction Extraction: Isolating the Principal Vulnerability Axis via SVD

Cross-modal coordination requires an anchor direction in representation space where both modalities naturally interact. The authors analyze layer-wise cosine similarities between visual and textual steering vectors and observe that similarity peaks (exceeding 0.85) in middle fusion layers (e.g., layer 15 for InternVL2-8B and Qwen2-VL-7B, layer 21 for MiniGPT-4-13B), indicating strong visual-textual coupling. At this target layer \(\ell\), representations of the final token \(h \in \mathbb{R}^d\) aggregate context across all prior visual and linguistic tokens via causal attention. For each harmful instruction \(t_{\text{harm}}^{(j)}\), the safe activation \(h_{\text{safe}, j}\) is extracted alongside activations perturbed by existing visual attacks (\(h_{\text{jb-vis}, j}\)) and textual attacks (\(h_{\text{jb-txt}, j}\)), yielding displacement vectors \(d_{\text{vis}, j} = h_{\text{jb-vis}, j} - h_{\text{safe}, j}\) and \(d_{\text{txt}, j} = h_{\text{jb-txt}, j} - h_{\text{safe}, j}\). Rather than calculating an arithmetic mean—which would allow per-query variations and modality-specific artifacts to dilute the shared signal—the shifts across \(N\) queries are concatenated into a matrix \(D = [d_{\text{vis}, 1}, \dots, d_{\text{vis}, N}, d_{\text{txt}, 1}, \dots, d_{\text{txt}, N}] \in \mathbb{R}^{d \times 2N}\). Singular Value Decomposition (SVD) decomposes \(D\):

\[D = U \Sigma V^\top, \quad v_{\text{jailbreak}} = u_1\]

The dominant left singular vector \(u_1\) captures the axis of maximal variance across all \(2N\) shifts (accounting for over 50.6% of total variance), filtering out modality-specific noise and establishing an empirically grounded target direction \(v_{\text{jailbreak}}\).

2. Steering Alignment Loss: Synchronizing Dual-Modality Trajectories in Representation Space

Minimizing cross-entropy loss \(\mathcal{L}_{\text{CE}}\) alone optimizes output token likelihood but leaves representation trajectories unconstrained, leading each modality to drift toward distinct local optima. To ensure harmonious updates, Co-Steer explicitly tracks each modality's marginal contribution at the target layer during optimization:

\[\Delta h_{\text{vis}}^{(j)} = M^{(\ell)}(x + \delta_v, t_{\text{harm}}^{(j)}) - M^{(\ell)}(x, t_{\text{harm}}^{(j)})\]

\[\Delta h_{\text{txt}}^{(j)} = M^{(\ell)}(x, t_{\text{harm}}^{(j)} \oplus \delta_t) - M^{(\ell)}(x, t_{\text{harm}}^{(j)})\]

The steering alignment loss \(\mathcal{L}_{\text{steer}}\) maximizes the cosine similarity between each modality-specific shift and the pre-extracted vulnerability vector \(v_{\text{jailbreak}}\):

\[\mathcal{L}_{\text{steer}}^{(j)} = - \left( \left\langle \frac{\Delta h_{\text{vis}}^{(j)}}{\|\Delta h_{\text{vis}}^{(j)}\|_2}, v_{\text{jailbreak}} \right\rangle + \left\langle \frac{\Delta h_{\text{txt}}^{(j)}}{\|\Delta h_{\text{txt}}^{(j)}\|_2}, v_{\text{jailbreak}} \right\rangle \right)\]

Because both modalities are simultaneously penalized for deviating from the same unit vector, they are implicitly aligned with each other. The composite objective balances task compliance with geometric steering:

\[\mathcal{L}_{\text{total}}^{(j)} = \mathcal{L}_{\text{CE}}^{(j)} + \lambda \mathcal{L}_{\text{steer}}^{(j)}\]

where \(\lambda\) defaults to 1.0, preventing conflicting displacements from canceling out.

3. Alternating Block Coordinate Descent: Jointly Resolving Continuous and Discrete Subproblems

Optimizing continuous pixel perturbations \(\delta_v \in \mathbb{R}^{3 \times H \times W}\) alongside discrete token sequences \(\delta_t \in V^L\) presents an optimization challenge. Co-Steer decouples the problem via alternating block coordinate descent. In outer iteration \(k\), the visual perturbation is first updated over \(T\) inner steps using momentum-augmented PGD with step size \(\alpha = 1/255\) and \(\ell_\infty\) budget \(\epsilon = 32/255\) while holding \(\delta_t^{(k)}\) fixed. Subsequently, with \(\delta_v^{(k+1)}\) fixed, the textual suffix \(\delta_t\) is updated over \(H\) inner steps using Greedy Coordinate Gradients (GCG): token gradients are projected onto the embedding matrix to retrieve top-\(K\) (\(K=256\)) candidate tokens per position, evaluating replacements via forward-pass verification. Averaging objectives over diverse mini-batches (\(B=16\)) prevents overfitting to individual queries, producing a query-agnostic perturbation pair \((\delta_v^*, \delta_t^*)\).

Key Experimental Results

Main Results

Co-Steer is evaluated on three major safety benchmarks: MM-SafetyBench (1,680 queries across 13 scenarios), HarmBench (200 behaviors across functional categories), and AdvBench (520 harmful instructions). Experiments target three representative open-source MLLMs: InternVL2-8B, Qwen2-VL-7B-Instruct, and MiniGPT-4-13B. Attack Success Rate (ASR, %, mean ± SD over 3 runs) serves as the primary metric.

Method	MM-SafetyBench (InternVL2)	MM-SafetyBench (Qwen2-VL)	HarmBench (InternVL2)	HarmBench (Qwen2-VL)	AdvBench (InternVL2)	AdvBench (Qwen2-VL)
Vanilla	36.73 ± 1.26	39.40 ± 1.04	6.00 ± 0.87	10.00 ± 0.87	9.08 ± 0.87	12.37 ± 1.04
GCG (Text Suffix)	49.43 ± 1.00	47.84 ± 1.26	42.00 ± 1.50	38.50 ± 0.50	77.74 ± 0.76	73.62 ± 1.53
PAIR (Black-box Text)	60.36 ± 1.26	58.33 ± 0.29	60.50 ± 0.87	51.50 ± 0.50	74.22 ± 0.76	81.09 ± 0.29
VAJM (Visual Adv)	20.54 ± 1.32	46.61 ± 0.58	9.00 ± 1.50	24.00 ± 1.50	71.36 ± 1.26	68.68 ± 0.76
UMK (Independent Dual)	79.35 ± 1.53	78.21 ± 0.29	44.50 ± 1.50	48.50 ± 0.50	87.10 ± 1.53	82.93 ± 0.50
BAP (CoT-guided Visual)	59.29 ± 0.29	52.38 ± 1.26	55.50 ± 1.32	59.00 ± 1.32	82.18 ± 0.29	76.35 ± 1.26
Co-Steer (Ours)	87.20 ± 1.26	84.65 ± 0.76	84.00 ± 0.50	78.00 ± 0.87	91.53 ± 1.00	89.39 ± 0.58

On MiniGPT-4-13B, Co-Steer scores 88.94 ± 1.50% on MM-SafetyBench, 86.00 ± 0.87% on HarmBench, and 93.29 ± 1.15% on AdvBench, establishing state-of-the-art results across all nine benchmark-model configurations.

Ablation Study

Component and extraction ablations on MM-SafetyBench (Table 2) along with target layer selection on InternVL2-8B (Table 3a) demonstrate the empirical foundations of each module:

Configuration / Strategy	InternVL2-8B ASR (%)	Qwen2-VL-7B ASR (%)	MiniGPT-4-13B ASR (%)	Description / Analysis
Visual-only (\(\delta_v\))	55.28	53.47	60.18	Single continuous perturbation under \(\mathcal{L}_{\text{CE}}\)
Text-only (\(\delta_t\))	53.61	56.33	61.02	Single discrete suffix under \(\mathcal{L}_{\text{CE}}\)
Independent Dual	79.62	78.19	86.85	Dual perturbations optimized independently without alignment
\(d_{\text{vis}}\) Averaging	62.34	59.83	70.25	Direction extracted from visual shifts only
\(d_{\text{txt}}\) Averaging	68.72	66.41	76.58	Direction extracted from textual shifts only
Cross-modal Mean	84.58	82.19	86.47	Arithmetic mean across both modalities' shifts
Co-Steer Full (SVD)	87.20	84.65	88.94	First principal singular vector extracted via SVD

Target layer selection on InternVL2-8B follows an inverted-U trend: Layer 3 achieves 62.34%, Layer 8 achieves 71.28%, Layer 15 peaks at 87.20%, and Layer 25 drops to 68.47%, validating that middle fusion layers offer optimal steering leverage.

Black-Box Commercial Model Transfer

Universal perturbations trained on open-source models transfer zero-shot to commercial black-box models without architectural adaptation (Table 4):

Method	Qwen2.5-VL-72B	GPT-4o	Gemini 1.5 Pro
Vanilla	10.83 ± 0.14	8.86 ± 0.46	10.51 ± 0.38
VAJM (Visual)	14.69 ± 1.88	10.55 ± 0.06	13.04 ± 1.20
GCG (Text)	33.18 ± 0.92	29.54 ± 0.64	28.43 ± 1.26
UMK (Independent)	37.61 ± 0.51	33.07 ± 0.41	34.50 ± 1.71
BAP (CoT-guided)	43.76 ± 0.97	41.28 ± 0.49	44.05 ± 0.34
Co-Steer (Ours)	52.71 ± 0.97	49.03 ± 0.49	51.67 ± 0.34

Key Findings

• Higher alignment safety yields larger collaborative gains: On MiniGPT-4 (weaker baseline safety, Vanilla ASR 46.31%), independent optimization already reaches 86.85%, limiting Co-Steer's gain to +2.09 pp. In contrast, on strongly aligned models like InternVL2 and Qwen2-VL (Vanilla ASR < 40%), Co-Steer delivers substantial leaps on HarmBench (+23.50 pp and +19.00 pp over the runner-up). This underscores that overcoming rigorous safety alignment requires joint, directional pressure along shared representation axes rather than disjoint perturbations.
• Robustness of low-rank vulnerability across queries: With 100 training queries, the top singular vector explains 50.6% of total variance. When deployed on 1,580 unseen test queries, performance degrades by only 3–6 pp (e.g., from 91.35% on training to 87.20% on testing for InternVL2), confirming that the extracted steering vector captures intrinsic safety boundaries rather than query-specific noise.
• Textual transfer anchors cross-model generalization, augmented by visual steering: Visual-only attacks struggle to transfer to commercial black-box models (10–14% ASR). Text suffixes provide the primary transferable medium; by infusing visual representation steering constraints into discrete token optimization, Co-Steer boosts transfer ASR on GPT-4o to 49.03% and Qwen2.5-VL-72B to 52.71%.

Highlights & Insights

• Repurposing representation steering for red-teaming: While activation steering has traditionally been explored for LLM safety guardrails and alignment steering, Co-Steer flips this paradigm, using geometric steering as a multi-modal optimization objective to reveal latent refusal vulnerabilities.
• SVD as an optimal noise-filtering aggregator: Rather than naive arithmetic averaging, treating multimodal shifts as samples in a high-dimensional activation space and isolating the principal component via SVD yields a high signal-to-noise vulnerability vector.
• Mechanistic blueprint for multimodal defense: The paper highlights that current MLLM vulnerabilities concentrate within middle fusion layers where visual and linguistic projections merge. Superficial safety tuning on input tokens or output heads leaves this intermediate geometry exposed, suggesting future defenses must enforce representation-level isotropy or steering resistance.

Limitations & Future Work

• Admitted limitations: The extraction phase depends on activation shifts harvested from existing successful single-modality attacks (such as VAJM and GCG); on hypothetical models immune to baseline single-modality attacks, acquiring initial sample pairs poses a bottleneck. Moreover, the optimized discrete suffixes contain nonsensical token strings that can be flagged by perplexity-based text filters.
• Open research questions: Experiments primarily evaluate models with standard ViT-based continuous visual backbones. Investigating how representation steering operates in native multimodal models with interleaved pre-training or discrete visual tokenizers remains an unexplored area.
• Improvement directions: Developing semantically coherent, natural-language adversarial prompts guided by collaborative steering could evade character-level perplexity defenses. On the defensive side, intermediate-layer anomaly detection based on steering projection offers a promising path for runtime guardrails.

Related Work & Insights

• vs UMK [36]: UMK performs uncoordinated dual-modality universal optimization using cross-entropy alone, allowing modalities to drift into separate subregions. Co-Steer introduces explicit steering loss \(\mathcal{L}_{\text{steer}}\) targeting the shared SVD-derived axis, preventing destructive interference and delivering a 6–8 pp boost on MM-SafetyBench.
• vs BAP [40]: BAP utilizes Chain-of-Thought reasoning to refine continuous visual noise, primarily remaining a visual attack with modest black-box transferability. Co-Steer unifies continuous and discrete modalities through collaborative block coordinate descent, achieving significantly higher transferability on closed commercial targets (52.71% vs 43.76% on Qwen2.5-VL-72B).
• vs JPS [8]: JPS incorporates collaborative concepts but assigns disparate, heuristic objectives to each modality. Co-Steer unifies multimodal jailbreaks under a shared low-rank subspace hypothesis, providing a mathematically grounded formulation rooted in SVD and directional cosine alignment.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Pioneers the discovery of a unified low-rank vulnerability axis across modalities and leverages representation steering for universal cross-modal jailbreaking]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Rigorous evaluations across three safety benchmarks, three open-source MLLMs, and three frontier commercial models, supported by layer-wise geometry and SVD sensitivity analyses]
• Writing Quality: ⭐⭐⭐⭐⭐ [Exceptionally clear mathematical exposition, structured motivation progressing from representational divergence to collaborative convergence, and polished figures]
• Value: ⭐⭐⭐⭐⭐ [Provides a potent red-teaming benchmark tool and exposes critical representation-level alignment vulnerabilities in intermediate fusion layers]

Related Papers

• [NeurIPS 2025] Learning to Steer: Input-dependent Steering for Multimodal LLMs
• [ECCV 2026] Synesthesia via Direct Latent Augmentation: Bypassing the Decode-Encode Loop for Cross-Modal Distillation
• [ECCV 2026] Beyond Categorical Matching: Intra-Class Graded Relevance Estimation for Cross-Modal 3D Retrieval
• [CVPR 2026] Same Content, Different Answers: Cross-Modal Inconsistency in MLLMs
• [CVPR 2026] VideoFusion: A Spatio-Temporal Collaborative Network for Multi-modal Video Fusion