Skip to content

Learning to Detect Unseen Jailbreak Attacks in Large Vision-Language Models

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/ShuangLiangX/Learning-to-Detect
Area: Safety
Keywords: LVLM Security, Jailbreak Detection, Concept Activation Vectors, Anomaly Detection, Generalized Defense

TL;DR

To overcome the dilemma between generalization and accuracy when defending Large Vision-Language Models (LVLMs) against unseen jailbreak attacks, this paper proposes Learning to Detect (LoD)—a framework that requires neither attack data nor hand-crafted heuristics by extracting layer-wise safety representations via Multi-modal Safety Concept Activation Vectors (MSCAV) and detecting attacks through unsupervised reconstruction errors with a Safety Pattern Auto-Encoder (SPAE).

Background & Motivation

Large Vision-Language Models (LVLMs) empower large language models with rich multimodal perception and visual reasoning capabilities, yet the continuous and high-dimensional visual attack surface introduces severe vulnerabilities. Subtle visual adversarial perturbations or carefully orchestrated multimodal prompts can easily bypass safety alignment guardrails, inducing models to generate prohibited content with attack success rates reaching up to 96% on benchmarks like AdvBench. To mitigate these risks, prior detection methods follow two primary paradigms: learning-based approaches that train classifiers on known jailbreak attacks, which inevitably suffer from catastrophic failure when facing novel, unseen attack patterns; and learning-free heuristic methods (e.g., monitoring pre-defined refusal tokens or gradient variations), which circumvent the need for attack samples but incur heavy computational overhead and exhibit high false-positive or false-negative rates in complex multimodal reasoning scenarios.

The fundamental tension lies in the non-exhaustive diversity of open-world jailbreak strategies versus the intrinsic consistency of internal model safety representations. Existing internal-signal detection methods lack a principled representation-learning mechanism to filter out task-irrelevant noise (such as syntactic structure or background semantics) from high-dimensional hidden activations. Furthermore, high-dimensional safety activations between benign and attacked inputs display substantial per-dimension overlap (exceeding 40%), rendering hand-crafted dimensional thresholding ineffective and prone to breaking cross-layer collaborative dependencies.

This paper tackles the challenge by reframing unseen jailbreak detection away from chasing diverse attack variants toward modeling the intrinsic manifold of safe inputs. The key observation is that although jailbreak attacks succeed in subverting refusal behaviors at the final output layer, their intermediate representations retain distinct anomalous traces that deviate from benign activations. Core idea: reframe unseen jailbreak detection as a two-stage internal activation anomaly detection task, first extracting noise-filtered layer-wise safety representations using Multi-modal Safety Concept Activation Vectors (MSCAV), and then training a Safety Pattern Auto-Encoder (SPAE) solely on safe inputs to flag unseen attacks via reconstruction error deviations from the safe manifold.

Method

Overall Architecture

The LoD framework comprises two cooperative modules: representation learning and attack classification. In the offline adaptation stage, the framework uses standard benign and vanilla harmful multimodal pairs (without any jailbreak transformations) to extract final-token hidden states across all layers, training linear Multi-modal Safety Concept Activation Vector (MSCAV) classifiers to derive layer-wise unsafe probabilities while selecting safety-sensitive layers exceeding a validation threshold \(P_0\). Subsequently, a three-layer symmetric Safety Pattern Auto-Encoder (SPAE) is trained exclusively on the safety representations of benign inputs to model the typical latent distribution of safe activations. During online inference, given a multimodal test prompt, the LVLM executes a forward pass; hidden states from the selected layers are extracted, mapped to a compact safety representation via frozen MSCAV classifiers, and passed through SPAE. If the reconstruction error exceeds a calibrated threshold, the input is flagged and intercepted as a jailbreak attack.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Test Input<br/>(Image + Text Prompt)"] --> B["LVLM Forward Pass<br/>Extract Final-Token Hidden States"]
    B --> C["MSCAV Safety Representation Extraction<br/>Linear Projection & Noise Filtering"]
    C --> D["SPAE Reconstruction Anomaly Scoring<br/>Safe Manifold Deviation Measurement"]
    D -->|Error delta > tau| E["Intercept: Flagged as Jailbreak Attack"]
    D -->|Error delta <= tau| F["Pass: Verified as Safe Input"]

Key Designs

1. MSCAV Safety Representation Extraction: Isolating Safety Concepts from Task Noise

To filter out task-irrelevant noise (e.g., syntax, stylistic nuances, visual details) from multi-thousand-dimensional raw activations, this design leverages the linear separability assumption in LVLM hidden states. For an LVLM with \(L\) layers, let \(e_l \in \mathbb{R}^d\) denote the activation vector at the final token position of layer \(l\). A set of linear classifiers \(C_l(e_l) = \text{sigmoid}(w_l^\top e_l + b_l)\) is trained to output the probability of an input being unsafe at each layer. Training requires only vanilla safe inputs \(I^+\) and vanilla harmful prompts \(I^-\) (e.g., AdvBench text paired with synthetic images), optimized via binary cross-entropy loss:

\[ \mathcal{L}_{C_l} = -\frac{1}{N} \sum_{i=1}^N \left[ y_i \log C_l(e_{l,i}) + (1 - y_i) \log (1 - C_l(e_{l,i})) \right] \]

Empirical analysis reveals that LVLM safety classifiers attain over 90% validation accuracy as early as the 4th layer—substantially earlier than the ~10th layer typical in text-only LLMs—indicating that visual integration prompts earlier emergence of safety concepts. To eliminate noisy, uninformative layers, a validation accuracy threshold \(P_0\) filters the sensitive layer set \(\mathcal{L}_s = \{l \mid P_l \ge P_0\}\), yielding the refined representation \(S_r = [C_l(e_l)]_{l \in \mathcal{L}_s}^\top \in \mathbb{R}^{|\mathcal{L}_s|}\). While jailbreak attacks suppress unsafe signals at the output layer, intermediate layers consistently produce higher unsafe probabilities than benign samples, providing a robust discriminatory basis.

2. SPAE Reconstruction Anomaly Scoring: Unsupervised Safe Manifold Modeling without Attack Biases

Although the refined representations \(S_r\) show distinct average trends, fine-grained analysis reveals an average overlap ratio exceeding 40% across individual dimensions between benign and attacked inputs. Consequently, direct dimension-wise thresholding leads to high error rates. LoD addresses this by formulating detection as one-class unsupervised anomaly detection using a Safety Pattern Auto-Encoder (SPAE), which features a three-layer fully connected encoder and a symmetric decoder with ReLU activations. The SPAE is trained exclusively on safe inputs to minimize the mean-squared reconstruction error:

\[ \mathcal{L}_{\text{rec}} = \frac{1}{N_s} \sum_{i=1}^{N_s} \|\hat{S}_{r,i} - S_{r,i}\|_2^2 \]

Because the network captures the complex non-linear inter-layer relationships unique to benign prompts, any input exhibiting an anomalous cross-layer pattern—such as an adversarial perturbation or prompt manipulation—incurs a high reconstruction error:

\[ \delta = \|\hat{S}_r - S_r\|_2^2 \]

At test time, an input is classified as an attack if \(\delta > \tau\). The threshold \(\tau\) is determined using the 90th percentile of anomaly scores on a small held-out set of benign samples (e.g., 100 items from MM-Vet2). This strategy completely eliminates reliance on attack data or heuristic rules while maintaining high sensitivity to unseen adversarial attempts.

Loss & Training

LoD follows a highly efficient, two-stage offline training workflow that preserves all underlying LVLM weights. In Stage 1, 100 pairs of benign and vanilla harmful inputs are used to train \(L\) independent linear MSCAV classifiers (requiring merely \(d+1\) parameters per layer, finishing in ~12 seconds across 32 layers); layers achieving validation accuracy above \(P_0 = 0.9\) form \(\mathcal{L}_s\). In Stage 2, frozen MSCAV classifiers extract safety representations from 320 benign multimodal samples, and the SPAE is trained using the Adam optimizer to minimize \(\mathcal{L}_{\text{rec}}\) (taking ~2 seconds). Online inference introduces zero backward passes and requires no auxiliary auto-regressive decoding, executing in milliseconds via forward hidden-state extraction and lightweight matrix operations.

Key Experimental Results

Main Results

The evaluation assesses performance on three representative LVLMs—LLaVA-1.6-vicuna-7B, Qwen2.5-VL-7B-Instruct, and CogVLM-chat-v1.1—against six diverse jailbreak attacks: four prompt-manipulation attacks (FC-Attack, JOOD, HADES, MML) and two adversarial perturbation attacks (VAJM, UMK). Evaluated baselines include input/output detection methods (MirrorCheck, CIDER, JailGuard) and internal representation/gradient methods (GradSafe, HiddenDetect). Detection accuracy is measured using AUROC.

Model Category / Baseline Method Minimum AUROC (Min) Average AUROC (Avg.) Relative Gain over Best Baseline
LLaVA-1.6 Input/Output (JailGuard) 0.3451 0.4760 -
Internal Signals (HiddenDetect) 0.3271 0.8230 Best Baseline (Avg.)
LoD (Ours) 0.9645 0.9820 +19.32% (Min AUROC +170.4%)
Qwen2.5-VL Input/Output (CIDER) 0.1503 0.5668 -
Internal Signals (GradSafe) 0.8902 0.9414 Best Baseline (Avg.)
LoD (Ours) 0.9253 0.9838 +4.50% (Min AUROC +3.94%)
CogVLM Input/Output (MirrorCheck) 0.4311 0.5956 -
Internal Signals (HiddenDetect) 0.6951 0.8287 Best Baseline (Avg.)
LoD (Ours) 0.8798 0.9705 +17.11% (Min AUROC +26.57%)

Under strict operating conditions with a fixed false-positive rate (FPR \(\le 0.01\)), LoD achieves average True Positive Rates (TPR) of 0.487, 0.764, and 0.717 across the three models, outperforming the strongest baseline on CogVLM (0.176) by over 300%.

Ablation Study

The contribution of each core component is analyzed across the six attack techniques (reporting average AUROC):

Configuration Variant LLaVA-1.6 Avg. AUROC Qwen2.5-VL Avg. AUROC CogVLM Avg. AUROC Note
LoD (Full Model) 0.982 0.984 0.971 MSCAV representation + SPAE anomaly scoring
w/o MSCAV Classifiers 0.467 0.336 0.138 Raw hidden states directly concatenated into SPAE; drastic collapse
w/o SPAE 0.875 0.874 0.857 MSCAV with 95th-percentile dimension-wise threshold aggregation
w/ HiddenDetect Features 0.811 0.703 0.616 Replacing MSCAV vectors with HiddenDetect heuristic features
SPAE-unsafe (with Harmful Data) 0.985 0.973 0.971 Margin penalty added on harmful reconstructions; inconsistent gains

Key Findings

  • MSCAV extraction is critical: Removing MSCAV causes average AUROC to drop to 0.138–0.467, demonstrating that raw activation spaces are overwhelmed by task-irrelevant visual and textual noise, preventing autoencoders from isolating safety dynamics.
  • Dimensional overlap necessitates SPAE: Individual dimensions of safety representations exhibit over 40% overlap between benign and attacked samples (e.g., 44.14% on LLaVA). Heuristic aggregation (w/o SPAE) loses approximately 10 AUROC points due to the inability to model inter-layer non-linear dependencies.
  • Superior computational efficiency: On a single NVIDIA A800 GPU, LoD completes inference in 0.25s (LLaVA) to 0.62s (Qwen2.5-VL) per sample, offering up to a 62.7% reduction in runtime compared to competitive baselines and avoiding the multi-second overhead of multi-query generation methods like JailGuard (15–21s).

Highlights & Insights

  • One-Class Safe Manifold Modeling: Bypasses the reactive arms race of collecting attack data by training solely on safe patterns, providing robust generalization against completely unseen attack formulations.
  • Early Emergence of Multimodal Safety Concepts: Proves that safety concepts become linearly separable as early as layer 4 in LVLMs (compared to layer 10 in LLMs), demonstrating that multimodal alignment accelerates intermediate safety perception.
  • Zero-Overhead Deployment: Operates strictly on intermediate activations generated during the normal prompt forward pass, requiring zero backpropagation, no parameter modifications, and negligible memory overhead (~0.25 MB per inference).

Limitations & Future Work

  • Vulnerability to Internal Activation Adaptive Attacks: A white-box attacker who explicitly incorporates a regularization term to force intermediate layer activations into the benign autoencoder manifold could potentially degrade detection accuracy.
  • Detection Bounds on Deep Semantic Obfuscation: Highly complex cross-modal encryption attacks (such as MML, which mirrors text and scrambles tokens into visual puzzles) shift intermediate representations closer to benign reasoning states, presenting a challenging boundary case for anomaly detection.
  • vs HiddenDetect / GradSafe: HiddenDetect relies on manually specified refusal token dictionaries, while GradSafe demands expensive gradient computations with high GPU memory footprints. LoD is fully data-driven, operates strictly in forward mode, and avoids failure on unmodeled attack patterns.
  • vs CIDER / JailGuard: Input/output methods require multi-pass model generation or external diffusion denoisers, resulting in high latency and low true-positive rates under strict false-positive constraints; LoD taps internal activations directly for higher fidelity and efficiency.

Rating

  • Novelty: ⭐⭐⭐⭐ [Innovative combination of concept activation vectors and one-class anomaly detection for generalizable LVLM defense]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across 3 LVLM architectures, 6 attack types, low-FPR conditions, and expanded sample datasets]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Well-structured narrative, lucid conceptual justifications, and rigorous experimental reporting]
  • Value: ⭐⭐⭐⭐⭐ [Highly practical, low-latency, and plug-and-play defense framework directly applicable to commercial LVLM serving systems]