GeoDetect: Geometric Adversarial Detection for VLPs

Conference: ECCV 2026
Paper: ECCV 2026
Code: https://github.com/AfsanehEB/GeoDetect
Area: AI Safety
Keywords: Adversarial detection, Vision-Language Pre-trained Models, Geometric analysis, Anisotropic manifold, Local intrinsic dimensionality

TL;DR

Addressing the vulnerability of vision-language pre-trained models (VLPs) to multimodal adversarial attacks and the prohibitive cost of adversarial training, GeoDetect investigates the anisotropic geometry of VLP embeddings, theoretically demonstrates that adversarial examples inevitably deviate from clean manifolds with increased expected geometric distances, and employs lightweight geometric metrics (LID, k-NN, Mahalanobis, KDE) to achieve highly accurate, fine-tuning-free adversarial detection across multiple architectures and tasks.

Background & Motivation

Vision-language pre-trained models (VLPs, such as CLIP, ALBEF, and TCL) learn joint representations over massive paired image-text datasets via multimodal contrastive learning, establishing state-of-the-art benchmarks in zero-shot classification, cross-modal retrieval, and visual question answering. Despite their remarkable empirical success, recent safety research reveals that these foundation models remain acutely vulnerable to adversarial examples (AEs). Both unimodal visual perturbations and co-optimized multimodal attacks can easily corrupt the shared embedding alignment, misleading downstream predictions into catastrophic failures. To counteract this vulnerability, the dominant line of defense has relied on adversarial training. However, fine-tuning billion-scale multimodal architectures on massive datasets introduces prohibitive computational overhead, while frequently causing severe degradation in standard zero-shot generalization on clean inputs.

Rather than striving for uniform robustness across the entire unconstrained input space, adversarial detection offers a lightweight, flexible alternative by identifying and rejecting malicious queries at inference time. Nevertheless, existing detection mechanisms developed for unimodal vision classifiers—such as those tracking classification softmax confidences or unimodal feature cluster dispersion—cannot be straightforwardly ported to multimodal architectures. Unimodal detectors fundamentally rely on discrete class semantics and cross-entropy margins, whereas VLPs align continuous multimodal representations within a highly anisotropic embedding space, where features are compressed into narrow hyper-cones or dual-ellipsoidal shells. Recent VLP-specific defenses like PIP are rigidly tailored to visual question answering by analyzing cross-attention response patterns to irrelevant probe questions, leaving them completely inapplicable to open-vocabulary label-free scenarios like image-text retrieval.

The fundamental tension lies in discovering a universal, architecture-agnostic signal of adversarial distortion without requiring access to ground-truth discrete labels or altering frozen model weights. The authors observe that the intrinsic anisotropy induced by multimodal contrastive alignment concentrates clean variance along a tiny fraction of principal dimensions, while adversarial perturbations seeking to distort semantic similarity inevitably project representations along low-variance normal directions off the underlying data manifold. Core idea: Grounded in the anisotropic geometry and manifold hypothesis of VLP embedding spaces, theoretically prove that adversarial examples exhibit higher expected distances and off-manifold deviations from clean data, and construct a plug-and-play detector, GeoDetect, driven by lightweight geometric discrepancy scores (LID, k-NN, Mahalanobis distance, KDE).

Method

Overall Architecture

GeoDetect operates across three consecutive stages: adversarial generation (for calibration and classifier fitting), geometric feature extraction, and adversarial decision testing. Given a query pair \((x_i, t_i)\), the frozen VLP encoders extract either the unimodal image embedding \(z_{\mathrm{I}}\) or the multimodal fused embedding \(z_{\mathrm{M}}\). Concurrently, a minibatch of clean reference embeddings \(\{z_j\}_{j=1}^n\) is sampled from a clean reference pool. Using this local neighborhood reference, the system computes geometric discrepancy metrics for the query embedding, encompassing Local Intrinsic Dimensionality (LID), k-Nearest Neighbors (k-NN) distance, Mahalanobis distance, and Kernel Density Estimation (KDE). For scalar distance metrics (k-NN, Mahalanobis, KDE), a binary threshold decision rule \(\mathbb{I}(s_i > \tau)\) calibrated on the validation split is applied; for multi-layer LID feature vectors extracted across intermediate and multimodal fusion layers, a trained lightweight logistic regression classifier determines whether the input is benign or adversarial.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Image-Text Query Pair (x, t)"] --> B["VLP Encoders Extract Embeddings<br/>Unimodal z_I / Multimodal z_M"]
    C["Clean Reference Batch {z_j}"] --> D["1. Anisotropic Manifold Deviation<br/>Off-Manifold Geometry Analysis"]
    B --> D
    D --> E["2. Multi-Metric Geometric Scoring<br/>LID / k-NN / Mahalanobis / KDE"]
    E -->|Scalar Score| F["3. Task-Agnostic Decision Rule<br/>Calibrated Threshold II(s > tau)"]
    E -->|Multi-Layer LID Vector| G["3. Task-Agnostic Decision Rule<br/>Lightweight Logistic Regression"]
    F --> H["Final Output: Clean vs Adversarial"]
    G --> H

Key Designs

1. Anisotropic Manifold Deviation: Establishing the mathematical necessity of adversarial off-manifold displacement

Prior adversarial defense strategies frequently relied on empirical heuristics without formal geometric proofs under multimodal contrastive spaces. The authors empirically verify the pronounced anisotropy of VLPs (ALBEF, TCL, CLIP) compared to supervised classification networks via Effective Rank (\(\widehat{\text{ER}}\)) and isotropy criteria (\(I_1, I_2\)). These tests show that singular values decay rapidly (\(\sigma_1 \gg \sigma_2 \gg \dots \gg \sigma_D\)), confirming that clean embeddings reside near a low-dimensional manifold \(\mathcal{M}\) (\(\|z_i - \mathcal{M}\| \le \alpha\)). Under this manifold assumption and local Gaussian modeling, the optimal adversarial perturbation maximizing the KL divergence from the clean distribution admits a closed-form solution: $\(z'^*_i = (\Sigma + \lambda I)^{-1} \lambda z_i, \quad \lambda > 0\)$ where \(\Sigma\) denotes the positive-definite clean covariance matrix and \(\lambda\) is a Lagrange multiplier. In anisotropic settings, this optimal transformation strongly suppresses the leading tangential directions while amplifying energy along manifold-normal directions, forcing adversarial representations into sparse, off-manifold regions (\(\|z'_i - \mathcal{M}\| \ge \gamma > 3\alpha\)). From this, Theorem 1 formally proves that the expected Euclidean distance between an adversarial embedding and a randomly drawn clean embedding \(z_u\) strictly exceeds that of its clean counterpart: \(\mathbb{E}_{z_u \sim p}[\|z'_i - z_u\|] > \mathbb{E}_{z_u \sim p}[\|z_i - z_u\|]\), providing rigorous theoretical validation for geometric distance-based detection.

2. Multi-Metric Geometric Scoring: Complementary capture of local density, correlation, and intrinsic dimensionality

To robustly identify disparate adversarial attack formulations without task overfitting, GeoDetect leverages four complementary geometric measurement metrics: - Local Intrinsic Dimensionality (LID): Quantifies the expansion rate of local data volume as distance grows. Because adversarial points reside in sparse, off-manifold voids, their estimated local dimension increases sharply. GeoDetect computes maximum likelihood estimations (MLE) of LID across intermediate encoder blocks, and for fused VLPs (e.g., ALBEF), incorporates the multimodal encoder output \(z_{\mathrm{M}}\) into the LID feature vector: $\(\widehat{\mathrm{LID}}(z_i, \{z_j\}) = \left( - \frac{1}{k} \sum_{j=1}^k \ln \frac{r_j(z_i)}{r_{\max}(z_i)} \right)^{-1}\)$ where \(r_j(z_i)\) denotes the Euclidean distance to the \(j\)-th nearest neighbor in the reference set. - k-Nearest Neighbor Distance (k-NN): Averages the Euclidean distances to the \(k\) nearest neighbors: \(\frac{1}{k} \sum_{j=1}^k r_j(z_i)\). Pushed outside the dense clean manifold clusters, adversarial points exhibit substantially larger neighbor distances. - Mahalanobis Distance: Measures \(\sqrt{(z_i - \mu)^\top \Sigma^{-1} (z_i - \mu)}\) against clean feature statistics, rescaling deviations inversely by principal variance and magnifying anomalies along low-variance normal axes. - Kernel Density Estimation (KDE): Nonparametrically estimates the local probability density function; off-manifold adversarial samples produce sharply reduced probability density values.

3. Task-Agnostic Decision Rule: Zero-fine-tuning detection driven by minibatch sampling

Traditional defense frameworks require downstream task re-training or depend heavily on classification logits. GeoDetect decouples detection entirely from downstream tasks. At inference, rather than constructing prohibitive global index trees over entire datasets, GeoDetect draws a small reference minibatch (e.g., batch size 64 or 128) of clean embeddings to compute geometric scores on the fly. For scalar metrics (k-NN, Mahalanobis, KDE), a single threshold \(\tau\) determined on a calibration set executes binary filtering via \(\mathbb{I}(s_i > \tau)\). For multi-layer LID, a tiny logistic regression classifier trained on multi-layer LID vectors performs inference in sub-millisecond latency. Because the base VLP weights remain entirely untouched, GeoDetect preserves zero-shot generalization capabilities while delivering robust real-time security filtering.

Key Experimental Results

Main Results

The authors evaluate GeoDetect across zero-shot classification (ImageNet-1k, CIFAR-10, CIFAR-100, STL-10, Food-101) and image-text retrieval (Flickr30k, MS-COCO). Tested models span aligned architectures (CLIP-CNN ResNet-50, CLIP-ViT ViT-B/16) and fused architectures (ALBEF, TCL). The threat models include unimodal image attacks (\(\text{Sep}_{\text{uni}}\)), multimodal fusion attacks (\(\text{Sep}_{\text{multi}}\)), and joint image-text co-optimization (Co-Attack) under an \(\ell_\infty\) bound of \(\epsilon = 8/255\). The primary baseline is MCM, a state-of-the-art zero-shot OOD/adversarial detection method for CLIP. Evaluation metrics are the Area Under the ROC Curve (AUC, %, higher is better) and the False Positive Rate at 95% True Positive Rate (FPR95, %, lower is better).

Model	Detection Method	Attack Type	CIFAR-10 AUC / FPR95	CIFAR-100 AUC / FPR95	ImageNet-1k AUC / FPR95	STL-10 AUC / FPR95	Food-101 AUC / FPR95
CLIP-CNN	MCM	\(\text{Sep}_{\text{uni}}\)	65.47 / 82.88	41.13 / 94.15	86.10 / 60.35	95.82 / 17.92	91.70 / 40.18
CLIP-CNN	MCM	Co-Attack	67.10 / 79.54	43.99 / 93.21	80.83 / 68.38	94.10 / 25.64	82.14 / 64.38
CLIP-CNN	GeoDetect-LID	\(\text{Sep}_{\text{uni}}\)	100.00 / 0.00	100.00 / 0.00	99.31 / 1.87	100.00 / 0.00	99.98 / 0.06
CLIP-CNN	GeoDetect-LID	Co-Attack	100.00 / 0.00	100.00 / 0.00	99.50 / 1.62	100.00 / 0.00	99.95 / 0.08
CLIP-CNN	GeoDetect-kNN	\(\text{Sep}_{\text{uni}}\)	100.00 / 0.00	100.00 / 0.00	99.65 / 1.62	100.00 / 0.00	100.00 / 0.00
CLIP-CNN	GeoDetect-kNN	Co-Attack	100.00 / 0.00	100.00 / 0.00	99.67 / 0.89	100.00 / 0.00	100.00 / 0.00
ALBEF	MCM	\(\text{Sep}_{\text{uni}}\)	91.20 / 29.02	82.80 / 49.19	92.15 / 25.38	96.83 / 16.02	90.26 / 37.03
ALBEF	MCM	\(\text{Sep}_{\text{multi}}\)	47.43 / 98.23	33.55 / 99.56	63.03 / 97.59	65.32 / 86.60	41.98 / 99.46
ALBEF	GeoDetect-LID	\(\text{Sep}_{\text{uni}}\)	100.00 / 0.00	99.97 / 0.05	91.85 / 29.41	99.64 / 1.62	99.87 / 0.67
ALBEF	GeoDetect-LID	\(\text{Sep}_{\text{multi}}\)	99.96 / 0.20	99.85 / 0.44	78.77 / 67.68	96.63 / 15.65	92.31 / 33.27
ALBEF	GeoDetect-Mahal.	\(\text{Sep}_{\text{uni}}\)	100.00 / 0.00	100.00 / 0.00	99.94 / 0.20	100.00 / 0.00	100.00 / 0.00
ALBEF	GeoDetect-Mahal.	\(\text{Sep}_{\text{multi}}\)	100.00 / 0.00	100.00 / 0.00	81.41 / 64.82	99.25 / 3.19	99.16 / 3.92

In image-text retrieval benchmarks (Flickr30k and COCO), MCM is entirely inapplicable due to the lack of discrete class prompts. GeoDetect (LID and k-NN) seamlessly operates in this label-free domain: CLIP-CNN achieves 99.97%–99.99% AUC with 0.00% FPR95 on Flickr30k under both \(\text{Sep}_{\text{uni}}\) and Co-Attack, and 99.50%–99.97% AUC on MS-COCO.

Ablation Study: Defense Against White-Box Adaptive Attacks

To confirm that GeoDetect does not rely on obfuscated gradients, the authors evaluate white-box adaptive attacks where adversaries know the exact defense objectives, loss formulations, and calibration rules. The adversary optimizes \(\mathcal{L}_{\text{adaptive}} = \mathcal{L}_{\text{main}} - \zeta \cdot \text{Metric}\). To prevent artifacts from unstable single-batch gradients, evaluation is conducted with disjoint batches (attack generation batch \(B_g\) vs. detection reference batch \(B_d\)). The table below summarizes detection AUCs across non-adaptive and adaptive attack regimes in image-text retrieval:

Model	Attack Mode	Non-adaptive Flickr30k AUC	Adaptive Flickr30k AUC	Non-adaptive COCO AUC	Adaptive COCO AUC
CLIP-CNN (k-NN)	\(\text{Sep}_{\text{uni}}\)	99.99	67.82	99.97	74.47
CLIP-CNN (k-NN)	Co-Attack	99.97	67.32	99.95	74.28
CLIP-CNN (LID)	\(\text{Sep}_{\text{uni}}\)	98.45	83.81	99.54	93.14
CLIP-CNN (LID)	Co-Attack	98.90	82.81	99.50	94.67
ALBEF (LID)	\(\text{Sep}_{\text{uni}}\)	94.99	71.78	91.80	89.86
ALBEF (LID)	\(\text{Sep}_{\text{multi}}\)	74.26	89.83	79.85	92.31
ALBEF (LID)	Co-Attack	93.80	70.12	91.49	90.11
TCL (LID)	\(\text{Sep}_{\text{multi}}\)	84.72	91.83	83.95	94.64

Key Findings

• Layer-wise LID demonstrates exceptional adaptive resilience: While k-NN drops noticeably under white-box adaptive optimization (AUC dropping to 50%–68%), multi-layer LID maintains 83%–94% AUC across COCO and Flickr30k. Because LID aggregates expansion rates across shallow, intermediate, deep, and multimodal fusion blocks, the attacker cannot simultaneously suppress dimensional deviations across all layers with a single gradient descent signal.
• Mahalanobis metric shines in unimodal feature anomaly detection: By explicitly inverting the clean covariance structure, Mahalanobis distance heavily penalizes deviations along near-zero variance normal axes, achieving superior performance on ALBEF under unimodal attacks (reaching 99.94% AUC on ImageNet-1k).
• Multimodal embeddings are essential for fused architectures: Monitoring only image-encoder features leaves a vulnerability window against multimodal fusion attacks (\(\text{Sep}_{\text{multi}}\)). Incorporating the fused multimodal representation \(z_{\mathrm{M}}\) into the LID feature set restores detection AUC from ~50% to over 90% on ALBEF and TCL.

Highlights & Insights

• Rigorous synthesis of manifold geometry and multimodal security: Moving beyond trial-and-error heuristic defenses, this work formally proves that contrastively trained VLP spaces are strictly anisotropic, and that adversarial objectives mathematically enforce off-manifold geometric displacements.
• Plug-and-play, zero-fine-tuning deployment: GeoDetect operates without modifying pre-trained VLP weights or retraining downstream task heads. It evaluates inputs via lightweight minibatch reference sampling, preserving clean zero-shot utility and supporting real-time deployment.
• Cross-task transferability across open-vocabulary domains: By operating purely in the continuous representation space without requiring discrete class prompts or softmax normalization, GeoDetect generalizes directly from classification to unconstrained image-text retrieval and multimodal ranking.

Limitations & Future Work

• Dependency on clean reference batches: In extreme low-latency or single-sample streaming environments (batch size = 1), computing neighbor metrics requires maintaining a pre-cached offline clean reference bank in memory.
• Single-layer degradation under adaptive white-box attacks: Last-layer distance metrics (such as k-NN) suffer noticeable performance degradation when attackers directly optimize against the distance objective, highlighting the need for multi-layer or multiscale topological scoring.
• Future directions: Extending geometric off-manifold detection to autoregressive multimodal large language models (MLLMs like LLaVA or Qwen-VL) to detect visual prompt injection, jailbreak perturbations, and hallucination triggers.

Related Work & Insights

• vs MCM (NeurIPS 2022): MCM leverages softmax-normalized cosine similarities against textual class prompts for OOD detection in CLIP. However, MCM is constrained to classification tasks with predefined class vocabularies, produces high false positive rates (FPR95 up to 60.35% on ImageNet), and cannot support retrieval; GeoDetect operates entirely in continuous embedding space, supports label-free retrieval, and reduces FPR95 by orders of magnitude (down to 0%–1.87%).
• vs PIP (ACM MM 2024): PIP injects irrelevant probe questions into visual question answering to observe cross-attention perturbations. This requires specific cross-attention query mechanisms and is restricted to conversational VQA. GeoDetect is model-agnostic, architecture-agnostic, and achieves superior detection across diverse threat vectors.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Establishes a rigorous theoretical framework connecting anisotropic embedding geometry with adversarial off-manifold behavior in VLPs.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluates 4 foundation VLP architectures across 7 major benchmarks, spanning classification, retrieval, and strong white-box adaptive attacks.
• Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear mathematical proofs, well-structured analytical narratives, and comprehensive empirical validation.
• Value: ⭐⭐⭐⭐⭐ Delivers an efficient, zero-fine-tuning, plug-and-play defense paradigm for multimodal foundation models.

Related Papers

• [ECCV 2026] Adversarial Attack and Disturbance Detection by Hadamard-Coded Output Representations for Object Detection and Semantic Segmentation
• [CVPR 2026] TTP: Test-Time Padding for Adversarial Detection and Robust Adaptation on Vision-Language Models
• [ECCV 2026] A Classifier-Agnostic Zero-Shot Adversarial Attack Detection via CLIP
• [CVPR 2025] Geometric Knowledge-Guided Localized Global Distribution Alignment for Federated Learning
• [ECCV 2026] Ranked Activation Shift for Post-hoc Out-of-Distribution Detection