Proximity-CLIP: Text-Guided Semantic Proximity Learning for Zero-Shot Anomaly Detection¶

Conference: ECCV 2026
Paper: ECCV Official
Area: Object Detection
Keywords: Zero-Shot Anomaly Detection / Vision-Language Model / Semantic Proximity Learning / Anomaly Query Module / Metric Space Integrity

TL;DR¶

Addressing the geometric dilemma where enforcing strict orthogonality between normal and anomalous text prototypes disrupts pre-trained continuous metric structures, Proximity-CLIP visually calibrates dynamic semantic margins and deploys an Anomaly Query Module to actively retrieve subtle local defects in zero-shot anomaly detection.

Background & Motivation¶

Zero-shot anomaly detection (ZSAD) aims to detect and spatially localize anomalous patterns in images without relying on any category-specific normal or abnormal training data. Large-scale vision-language models, most notably CLIP, have emerged as a promising foundation for open-world perception by learning rich cross-modal alignments from hundreds of millions of image-text pairs. However, because CLIP is primarily pre-trained for global, instance-level image classification, its feature representations exhibit a strong object-centric bias. This makes the model inherently insensitive to subtle, localized structural variations and geometric defects, causing the naive normal and anomalous text prototypes to severely overlap in the hyperspherical latent space.

To overcome this semantic entanglement, recent parameter-efficient fine-tuning approaches, such as adapter-based AA-CLIP, enforce strict orthogonality between normal and abnormal text prototypes to maximize classification margins. Although this discrete semantic separation enhances mathematical separability, it introduces a profound geometric dilemma: in physical reality, an anomalous instance is fundamentally an intact normal object perturbed by minor localized defects. Consequently, their visual representations remain highly contiguous and proximate. Forcing continuous visual inputs onto drastically orthogonal semantic prototypes tears the continuous geometric structure of the pre-trained metric space, significantly impairing the model's zero-shot generalization to unseen categories.

The core tension lies in the realization that normal and abnormal semantic states should not be driven toward extreme orthogonality, but instead separated by a bounded, realistic margin governed by authentic visual distributions. Core idea: model anomalies as proximate semantic shifts structurally bounded by the normal feature distribution, learning a visually-calibrated continuous semantic margin in Stage 1 and employing the calibrated anomalous prototype as an active semantic query in an Anomaly Query Module (AQM) during Stage 2 to retrieve localized defect cues while preserving metric space integrity.

Method¶

Overall Architecture¶

Proximity-CLIP is formulated as a two-stage parameter-efficient fine-tuning framework designed to decouple entangled semantics while mitigating the dilution of subtle visual anomalies. In Stage 1, the vision encoder is frozen while lightweight text adapters and a bounded proximity target are optimized via visual variance, dynamically shaping a calibrated semantic margin that mirrors real-world visual distances. In Stage 2, the text branch is frozen, and the calibrated anomalous prototype acts as an explicit top-down semantic query within the Anomaly Query Module (AQM) to actively retrieve localized defect cues across hierarchical visual patch tokens.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image & Normal/Anomalous Text Prompts"] --> B["Stage 1: Norm-Preserving Adapters & Dynamic Proximity Regularization<br/>Frozen vision branch, visual variance calibrates bounded margin λ"]
    B --> C["Stage 2: Semantic-Guided Anomaly Querying<br/>Calibrated anomalous prototype serves as Query to retrieve local patch cues"]
    C --> D["Global-Local Joint Optimization<br/>Pixel-level patch segmentation alignment + Global anomaly token classification"]
    D --> E["Output Image Anomaly Score & Pixel Anomaly Map"]

Key Designs¶

1. Norm-Preserving Residual Adapters & Dynamic Proximity Regularization: Calibrating Continuous Semantic Margins via Visual Variance

To eliminate discrete orthogonal separation without distorting the latent metric manifold, the framework inserts lightweight residual MLPs into the text branch equipped with a Norm-Preserving Residual Blending mechanism. Standard fine-tuning shifts the magnitude distribution of latent representations, compromising hyperspherical consistency; the proposed norm-preserving mechanism projects adapted features back to the original \(L_2\) norm:

\[\tilde{h} = \omega \cdot \mathcal{A}(h) \frac{\|h\|_2}{\|\mathcal{A}(h)\|_2} + h\]

where \(\mathcal{A}(\cdot)\) denotes adapter projection and \(\omega\) modulates residual strength. Simultaneously, a dynamic semantic proximity target \(\lambda\) is parameterized via a bounded non-linear mapping \(\lambda = \sigma(\phi) \cdot \alpha + \beta\), mapping the target similarity strictly to \([\beta, \alpha + \beta]\) (configured as \([0.3, 0.95]\) with \(\alpha=0.65, \beta=0.3\)). During Stage 1, multi-scale visual patch features provide dense supervision via segmentation alignment loss \(\mathcal{L}_{align\_text}\), while an MSE objective \(\mathcal{L}_{prox} = \frac{1}{B} \sum_{i=1}^B (\tilde{\mu}_{n,i}^\top \tilde{\mu}_{a,i} - \lambda)^2\) drives the cosine similarity of text prototypes toward \(\lambda\). Consequently, the angular margin is autonomously calibrated by authentic visual variances rather than rigid heuristics.

2. Anomaly Query Module (AQM): Active Local Defect Retrieval Driven by Semantic Priors

Conventional vision-language alignment processes visual tokens uniformly, which causes subtle localized defects to be overwhelmed by dominant normal backgrounds during global spatial pooling. In Stage 2, the framework introduces a lightweight Anomaly Query Module (AQM) deployed across hierarchical visual layers \(\mathcal{L}\). The calibrated anomalous prototype \(\hat{\mu}_a\) is frozen and deployed as an explicit top-down semantic Query, while adapted visual patch tokens \(f_v^{(l)}\) serve as contextual Keys and Values:

\[Q = \hat{\mu}_a, \quad K^{(l)} = f_v^{(l)}, \quad V^{(l)} = f_v^{(l)}\]

Using a cross-attention interaction function \(\mathcal{F}_{attn}\), the AQM retrieves fine-grained localized responses directly from contextual patches: \(z_{query}^{(l)} = \mathcal{F}_{attn}(Q, K^{(l)}, V^{(l)}) + \hat{\mu}_a\). Acting like a semantic spotlight, this module suppresses background clutter and highlights complex structural anomalies. Hierarchical query tokens are fused via cross-layer mean pooling and \(L_2\)-normalized to construct the global anomaly token \(z_{global}\).

3. Global-Local Joint Optimization: Coordinating Dense Grounding and Holistic Discrimination

To simultaneously achieve accurate pixel-level localization and robust image-level decision making, Stage 2 employs a dual-level objective. At the dense local level, multi-scale visual tokens are matched against frozen text prototypes under a segmentation alignment loss \(\mathcal{L}_{align\_image}\) composed of Focal and Dice losses. At the holistic image level, the pooled anomaly token \(z_{global}\) is evaluated against the calibrated text prototypes and supervised with ground-truth label \(y \in \{0, 1\}\) using cross-entropy loss under a learnable temperature \(\tau\):

\[\mathcal{L}_{cls} = \text{CE}\left(\frac{z_{global}^\top \hat{\mu}}{\tau}, y\right)\]

The overall visual optimization loss \(\mathcal{L}_{image} = \mathcal{L}_{align\_image} + \mathcal{L}_{cls}\) ensures that the visual adapters capture localized defects while forming a coherent, highly discriminative global categorical representation.

Loss & Training¶

The framework follows a disciplined two-stage schedule. Stage 1 trains the text adapters and proximity target for 10 epochs with learning rates of \(1 \times 10^{-5}\) for adapters and \(0.01\) for the similarity module under \(\mathcal{L}_{text} = \mathcal{L}_{align\_text} + \gamma \mathcal{L}_{prox}\). Stage 2 trains the visual adapters and AQM for 25 epochs at a learning rate of \(5 \times 10^{-4}\) under \(\mathcal{L}_{image} = \mathcal{L}_{align\_image} + \mathcal{L}_{cls}\), keeping the visual backbone and text prototypes frozen. The model builds upon the OpenCLIP ViT-L/14 architecture at \(518 \times 518\) resolution and is trained entirely on a single NVIDIA RTX A6000 GPU.

Key Experimental Results¶

Main Results¶

Evaluation spans four industrial benchmarks (MVTec-AD, VisA, BTAD, MPDD) and three medical anomaly datasets (Brain MRI, Liver CT, Retina OCT) from BMAD under cross-dataset zero-shot protocols (trained on MVTec-AD when evaluating on VisA, and trained on VisA for all other datasets).

Dataset	Domain	Metric	Ours (Proximity-CLIP)	Prev. SOTA (AA-CLIP)	Gain
MVTec-AD	Industrial	Pixel AUROC / Image AUROC	92.0 / 92.2	91.8 / 90.0	+0.2 / +2.2
VisA	Industrial	Pixel AUROC / Image AUROC	95.7 / 82.9	94.7 / 78.3	+1.0 / +4.6
BTAD	Industrial	Pixel AUROC / Image AUROC	97.0 / 92.3	97.0 / 94.8	+0.0 / -2.5
MPDD	Industrial	Pixel AUROC / Image AUROC	96.2 / 74.1	96.5 / 73.8	-0.3 / +0.3
Brain MRI	Medical	Pixel AUROC / Image AUROC	96.2 / 77.2	95.3 / 77.6	+0.9 / -0.4
Liver CT	Medical	Pixel AUROC / Image AUROC	96.8 / 64.0	97.6 / 66.8	-0.8 / -2.8
Retina OCT	Medical	Pixel AUROC / Image AUROC	95.9 / 84.3	95.5 / 82.7	+0.4 / +1.6
Industrial Avg.	Industrial	Pixel AUROC / Image AUROC	95.2 / 85.4	95.0 / 84.2	+0.2 / +1.2
Medical Avg.	Medical	Pixel AUROC / Image AUROC	96.3 / 75.2	96.1 / 75.7	+0.2 / -0.5
Overall Avg.	All 7 Datasets	Pixel AUROC / Image AUROC	95.7 / 81.0	95.5 / 80.6	+0.2 / +0.4

Ablation Study¶

The ablation study validates the essential roles of Stage 1 Semantic Calibration (SC), the Anomaly Query Module (AQM), Norm-Preserving (NP) residual blending, and query retrieval over passive pooling.

Config	Industrial Pixel (%)	Industrial Image (%)	Medical Pixel (%)	Medical Image (%)	Note
Baseline (Vanilla CLIP)	57.5	60.3	55.5	44.3	Passive global matching bottlenecked by object bias
w/o SC (w/ AQM only)	91.9	70.4	92.9	73.9	Entangled text prototypes collapse industrial classification by -15.8%
w/o AQM (w/ SC only)	68.8	69.1	84.4	53.5	Passive pooling severely dilutes subtle localized defects
w/o NP (Text)	95.0	82.7	95.5	72.3	Text feature magnitude drift harms classification
w/o NP (Image)	94.6	81.9	95.9	68.5	Visual magnitude distortion degrades medical classification by -6.7%
w/o NP (Both)	94.5	82.0	95.8	70.8	Unstable hyperspherical metrics drop industrial (-4.2%) and medical (-4.4%)
Multi-scale (w/o AQM)	93.8	85.4	95.5	73.0	Passive multi-scale fusion falls short of active query
Ours (Full Model)	95.1	86.2	96.3	75.2	Synergistic calibrated priors and active querying achieve best overall scores

Key Findings¶

Synergy Between Calibration and Active Querying: Removing SC (w/o SC) causes Industrial Image AUROC to plummet from 86.2% to 70.4% due to entangled text priors, while removing AQM (w/o AQM) collapses Industrial Pixel AUROC from 95.1% to 68.8% due to spatial feature dilution. Both modules are strictly indispensable.
Critical Role of Norm-Preserving Mechanisms: Omitting norm-preserving blending (w/o NP Both) degrades image-level classification across industrial (-4.2%) and medical (-4.4%) benchmarks, proving that preventing feature scale drift is vital for stabilizing the pre-trained hyperspherical latent space.
Active Query Outperforms Passive Aggregation: Using the anomalous prototype as an active attention query outperforms traditional passive multi-scale pooling by +1.3% in Industrial Pixel AUROC and +2.2% in Medical Image AUROC.

Highlights & Insights¶

Geometric Re-examination of Orthogonality Constraints: Rather than blindly enforcing orthogonal text anchors that clash with the physical continuity of localized defects, the paper introduces a visually-calibrated continuous semantic margin that harmonizes separation with topological integrity.
Data-Driven Semantic Margin Formulation: Formulating the target margin via a non-linearly bounded parameter \(\lambda \in [0.3, 0.95]\) guided by empirical image variance ensures that text prototypes naturally mirror authentic visual defect distributions.
Text Priors as Active Detection Probes: Transforming the anomalous prototype from a passive linear classifier into an active cross-attention query provides a generalizable mechanism for extracting localized signals in vision-language architectures.

Limitations & Future Work¶

Performance Plateaus on Heterogeneous Medical Scans: On Liver CT and complex 3D slice benchmarks, Image AUROC remains modest (64.0%), as 2D slice processing lacks volumetric spatial continuity and struggles to distinguish genuine lesions from benign anatomical variations.
Two-Stage Sequential Training: Training requires a disconnected two-stage pipeline where text adapters must fully converge before freezing and initiating visual adaptation, increasing hyperparameter tuning and deployment overhead.
Future Directions: Exploring end-to-end alternating optimization to collapse the two stages into a unified training loop, and integrating lightweight 3D-aware positional encodings for volumetric medical scans.

vs AA-CLIP: AA-CLIP enforces rigid orthogonality on text anchors, causing geometric strain on continuous visual manifolds; Proximity-CLIP preserves metric space integrity via bounded dynamic margin learning and achieves smoother cross-domain generalization (81.0% vs 80.6% overall Image AUROC).
vs AnomalyCLIP / AdaCLIP: While earlier methods rely on object-agnostic or hybrid prompt tuning under passive global pooling, Proximity-CLIP uses the anomalous prototype as an active query within hierarchical cross-attention layers, mitigating patch dilution and improving fine-grained localization.

Rating¶

Novelty: ⭐⭐⭐⭐☆ Pinpoints the fundamental geometric dilemma of forced orthogonality in ZSAD and designs an elegant visually-calibrated proximity learning and querying framework.
Experimental Thoroughness: ⭐⭐⭐⭐☆ Evaluated across 7 standard industrial and medical benchmarks with extensive pixel/image-level comparisons and rigorous component ablations.
Writing Quality: ⭐⭐⭐⭐⭐ Well-structured narrative with crisp mathematical formulations, clear problem framing, and intuitive visualizations.
Value: ⭐⭐⭐⭐☆ Offers valuable insights into parameter-efficient fine-tuning and metric-preserving visual-language adaptation for industrial inspection and healthcare diagnostics.