CMDS-AD: Cross-Modal Dual-Stream Decoupling for Few-Shot Anomaly Detection¶

Conference: ECCV 2026
arXiv: 2606.20300
Code: https://github.com/Junhaocai27/CMDS-AD
Area: Anomaly Detection / Multimodal / Few-Shot Learning
Keywords: Few-Shot Anomaly Detection, Multimodal Fusion, Dual-Stream Decoupling, Diffusion Model, Frequency Decomposition

TL;DR¶

This paper proposes CMDS-AD, a cross-modal dual-stream decoupling framework for few-shot multimodal anomaly detection. By repurposing a pretrained diffusion normal estimator as a non-linear low-pass filter, a pure low-frequency auxiliary estimation stream is constructed to anchor global structural templates. This allows the real stream, which retains coupled high- and low-frequency components, to more accurately capture local micro-defects. Combined with a Coordinate-Aware Hierarchical Feature Mapper and a multiplicative anomaly scoring mechanism, the method achieves an absolute improvement of 5.7% and 7.7% in I-AUROC under the 1-shot setting on MVTec 3D-AD and EyeCandies, respectively.

Background & Motivation¶

Background: Multimodal Anomaly Detection (MAD) utilizes 3D geometric cues to complement RGB features, outperforming pure 2D methods in handling occlusions, illumination variations, and complex textured scenes. Memory bank-based, feature adaptation-based, and diffusion reconstruction-based methods have achieved significant progress under fully supervised settings. However, they all rely on large amounts of normal training data.

Limitations of Prior Work: Under few-shot scenarios (1-4 shot), the performance of existing multimodal methods drops sharply. Taking 1-shot as an example, the best-performing multimodal method on MVTec 3D-AD only achieves approximately 74% I-AUROC, which is far from meeting the demands of industrial quality inspection.

Key Challenge: Existing MAD methods process all spatial features uniformly—whether through layer-wise concatenation, cross-modal mapping, or memory-bank retrieval. This essentially entangles stable low-frequency macroscopic structures (the overall shape of the object) and unpredictable high-frequency local variations (texture shifts, minor scratches) in the same feature space. While this is manageable when data is abundant, under data-scarce conditions, the model fails to distinguish normal sensor noise from genuine structural defects, causing the false alarm rate to spike.

Goal: Explicitly decouple high- and low-frequency information to process low-frequency structural templates and high-frequency defect signals independently in their respective subspaces, avoiding mutual interference.

Key Insight: The authors discovered that pretrained diffusion normal estimators, limited by generative priors and latent space compression, naturally act as non-linear low-pass filters when generating normal maps. Their output normal maps for input RGB images are extremely smooth, preserving only large-scale structures and discarding high-frequency details. This finding implies that without designing extra filters, the diffusion model itself acts as a perfect decoupling tool.

Core Idea: Build a dual-stream architecture containing a real stream (real normals containing coupled high- and low-frequency components) and an estimation stream (diffusion-generated normals containing pure low-frequency information). The estimation stream serves as a stable structural anchor to guide the real stream, enabling the model to focus on high-frequency micro-defects instead of global shapes. Combined with a coordinate-aware hierarchical feature mapper and a multiplicative scoring mechanism, cross-modal fusion occurs only in valid regions, preventing the amplification of modal noise.

Method¶

Overall Architecture¶

The overall pipeline of CMDS-AD consists of three main phases: data augmentation, dual-stream decoupling & feature alignment, and anomaly scoring. Given a very small number of normal samples (e.g., 1 RGB image + corresponding 3D point cloud), a LoRA-fine-tuned diffusion model is first used to generate more diverse normal RGB samples. A pretrained normal estimator is then applied to reconstruct normal maps from both the real and generated RGB images, forming the augmented training set. During training, the framework simultaneously processes two input streams: the real normal (containing coupled high- and low-frequency info) and the estimated normal (retaining only pure low-frequency info via diffusion low-pass filtering). Both streams extract multi-scale features from layers 4, 7, and 11 of the same frozen ViT backbone, which are then adaptively aligned across 2D and 3D modalities using a Coordinate-Aware Hierarchical Feature Mapper. Training employs a decoupled multi-scale mask-aware optimization strategy, using the foreground mask mean for the real stream and the global mean for the estimation stream. During inference, the framework generates four directional anomaly distance maps, which are weighted, summed, and cross-modally fused via multiplication to produce the final pixel-level anomaly map.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Input<br/>RGB + Normal Map"] --> B["Data Augmentation<br/>LoRA-Generated RGB Samples<br/>Normal Estimator Generated Estimated Normals"]
    B --> C["Diffusion-Driven High/Low Frequency Dual-Stream Decoupling"]
    C --> D1["Real Stream<br/>Coupled High/Low Frequency Info"]
    C --> D2["Estimation Stream<br/>Pure Low-Frequency Structural Anchor"]
    D1 --> E["ViT Multi-scale Feature Extraction<br/>(Layers 4/7/11)"]
    D2 --> E
    E --> F["Coordinate-Aware Hierarchical Feature Mapper<br/>CoordAtt + Spatial Selector"]
    F --> G["Decoupled Optimization & Multiplicative Scoring<br/>Ψ = Ψ_2D ⊙ Ψ_3D"]
    G --> H["Final Anomaly Map Output"]

Key Designs¶

1. Diffusion-Driven High-Low Frequency Dual-Stream Decoupling: Decoupling Frequency Components by Using Normal Estimators as Inherent Low-Pass Filters

Core challenge addressed: Existing methods compress low-frequency structures and high-frequency defects into the same feature space, making it impossible for the model to distinguish normal texture variations from genuine defects under few-shot settings. The authors' key insight is that diffusion normal estimators (such as Marigold), due to generative priors and latent space compression, naturally produce extremely smooth outputs. Having been trained on large-scale datasets, these estimators will not relearn high-frequency details just for a few-shot sample. Mathematically, the real normal map \(N\) can be decomposed into a low-frequency structural component \(N_{\text{low}}\) and high-frequency details \(N_{\text{high}}\), whereas the estimator output is \(\hat{N} = \Phi_{\text{diff}}(I) \approx N_{\text{low}}\). Mapping them through the same backbone \(\Phi\) yields two complementary feature subspaces:

\[ \mathcal{F}_{\text{est}} \approx \Phi(N_{\text{low}}), \quad \mathcal{F}_{\text{real}} \approx \Phi(N_{\text{low}} + N_{\text{high}}) \]

The estimation stream \(\mathcal{F}_{\text{est}}\) serves as a pure low-frequency "structural anchor" to help the real stream \(\mathcal{F}_{\text{real}}\) focus on high-frequency micro-defects. The two streams are processed independently without direct feature residual pooling (preventing cross-modal noise amplification), but rather co-aggregated during the anomaly scoring stage. FFT spectrum analysis validates the physical foundation of this design: the ratio of high-to-low frequency energy in the estimated normals plummets to 0.0221 compared to 0.1251 in the real normals.

2. Coordinate-Aware Hierarchical Feature Mapper: Adaptive Cross-Modal Alignment Without Spatial Information Loss

Problem targeted: A large semantic gap and magnitude discrepancy exist between 2D RGB and 3D normals. Direct concatenation or MLP mapping discards spatial position information, while global pooling-style channel attention is particularly detrimental to pixel-wise dense prediction tasks.

The design consists of three steps. First, features \(x_l \in \mathbb{R}^{C\times H\times W}\) (\(C=768\)) are extracted from layers 4, 7, and 11 of the ViT backbone, representing local edge textures, part-level patterns, and global semantics respectively. They are individually domain-adapted via \(1\times 1\) convolution + Group Normalization + GELU, and concatenated along the channel dimension to form \(\mathcal{F}_{\text{cat}} \in \mathbb{R}^{3C\times H\times W}\). Next, Coordinate Attention is introduced, decomposing 2D spatial pooling into two 1D directional encodings:

\[ z_c^h(h) = \frac{1}{W}\sum_{i=0}^{W-1} \mathcal{F}_{\text{cat}}^{(c)}(h,i), \quad z_c^w(w) = \frac{1}{H}\sum_{j=0}^{H-1} \mathcal{F}_{\text{cat}}^{(c)}(j,w) \]

Unlike channel attention which flattens spatial dimensions via global average pooling, coordinate attention preserves row-wise and column-wise positional encodings, making it suitable for dense prediction. Finally, a lightweight spatial selector generates a three-layer mutually exclusive weight map \(\mathcal{W}=[\omega_4, \omega_7, \omega_{11}]\) (with \(\sum \omega_l=1\) per pixel) via key-wise/pixel-wise Softmax, adaptively fusing multi-scale features and projecting them to obtain the predicted feature \(\mathcal{P}\).

3. Decoupled Multi-Scale Mask-Aware Optimization and Multiplicative Anomaly Scoring: Divide-and-Conquer Optimization with Hard Cross-Modal Verification

Optimization and scoring are closely linked, sharing a divide-and-conquer logic.

The training phase employs cosine distance alignment separately for each stream, discarding magnitude-sensitive metrics like \(L_2\) distance due to the massive scale discrepancy between RGB and 3D normal domains (amounting to orders of magnitude of \(10^2\)). Cosine distance focuses solely on directional consistency:

\[ \mathcal{L}_{\text{align}}(P,T) = 1 - \frac{P\cdot T}{\|P\|_2\|T\|_2} \]

The real stream uses the foreground binary mask \(M\) derived from the 3D point cloud for mask-mean calculations (focusing on the object region and ignoring backgrounds like conveyor belts), while the estimation stream uses the global mean as it lacks precise masks. At the layer level, asymmetric decreasing weights \(\alpha > \beta > \gamma\) are introduced (set as \(\alpha=1.2, \beta=1.0, \gamma=0.8\) in experiments). The shallowest layer (layer 4) receives the largest weight to force precise localization of local micro-defects, while the deepest layer (layer 11) gets the smallest weight to prevent overfitting caused by pixel-level alignment at the semantic level.

During inference, the framework generates four anomaly distance maps: 2D real, 2D estimated, 3D real, and 3D estimated. They are first weighted and fused intra-modally (\(\Psi_{\text{2D}} = \Psi_{\text{2D}}^R + \lambda_1\Psi_{\text{2D}}^E\), and similarly for \(\Psi_{\text{3D}}\)), and then cross-modally fused via Hadamard product for multiplicative scoring:

\[ \Psi = \Psi_{\text{2D}} \odot \Psi_{\text{3D}} \]

Multiplication acts as a strict logical "AND" gate: a pixel is classified as a defect only when both 2D texture variations and 3D structural deviations trigger a high anomaly response. This fundamentally eliminates false alarms caused by single-modality anomalies (such as 2D specular reflections or 3D sensor noise). Under a strict FPR threshold, multiplicative fusion yields a PRO@1% that is approximately 1.0 percentage point higher than additive fusion.

A Complete Example: The Detection Process of a Scratch on a Bagel Surface¶

Consider a minor surface scratch on a bagel from MVTec 3D-AD. The model is trained on only 1 normal bagel image. During training, LoRA injects a rank-16 low-rank adaptation matrix \(BA\) on the \(W_0\) anchor to generate \(N\) variations of RGB images from the single raw image, which are textured differently but still represent a normal bagel. Each generated image and the original image are passed to Marigold to estimate smooth normal maps \(\hat{N}\) (retaining pure low-frequency details where the scratch is filtered out). The real normal \(N\) and the estimated normal \(\hat{N}\) form the dual-stream input. At Layer 4 (local edges) of the ViT backbone, the high-frequency residual from the real stream and the smooth background from the estimation stream form a natural contrast; at Layer 11 (global semantics), the two streams are nearly identical due to the same macroscopic shape. The coordinate-aware mapper adaptively assigns a higher weight \(\omega_4\) to Layer 4 in the scratched region, and a higher weight \(\omega_{11}\) to Layer 11 in flat regions. During inference, the scratched pixels exhibit a moderate response in \(\Psi_{\text{2D}}^R\) and \(\Psi_{\text{3D}}^R\), but nearly zero response in \(\Psi_{\text{2D}}^E\) and \(\Psi_{\text{3D}}^E\) (since the diffusion estimator has smoothed out the scratch). After weighting, the responses of \(\Psi_{\text{2D}}\) and \(\Psi_{\text{3D}}\) are concentrated in the same region, and their product \(\Psi\) accurately pinpoints the scratch. The model achieves an impressive 97.0 I-AUROC on the Bagel category under the 1-shot setting.

Loss & Training¶

The overall optimization goal of the framework is the sum of the layer-weighted losses of the real and estimation streams. Single-layer alignment employs cosine distance \(\mathcal{L}_{\text{align}}\), with mask constraints applied to the real stream and global constraints to the estimation stream. The three-layer weighted fusion formula is:

\[ \mathcal{L}_{\text{total}}^S = \alpha \mathcal{L}_{\text{L4}}^S + \beta \mathcal{L}_{\text{L7}}^S + \gamma \mathcal{L}_{\text{L11}}^S, \quad S \in \{R, E\} \]

\[ \mathcal{L} = \mathcal{L}_{\text{total}}^R + \mathcal{L}_{\text{total}}^E \]

Hyperparameter settings: \(\alpha=1.2, \beta=1.0, \gamma=0.8\); \(\lambda_1=\lambda_2=0.1\). The backbone is a frozen DINO ViT-B/8 with an input resolution of \(224\times224\). The optimizer is AdamW with cosine annealing learning rate scheduling from \(10^{-4} \to 10^{-6}\), and a weight decay of \(10^{-4}\). The model is trained for 3000 steps with a batch size of 2 per stream. During the data augmentation stage, LoRA rank=16 is used, and Stable Diffusion v2.1 is fine-tuned for 1000 steps. A Gaussian noise with \(\sigma=0.01\) is injected into the RGB input to prevent overfitting. The hardware used is an RTX 5090 (32GB).

Key Experimental Results¶

Main Results¶

Below is the comparison table of main results (MVTec 3D-AD, metrics are I-AUROC / AUPRO@30%), highlighting the extreme few-shot setups of 1-shot and 4-shot. CMDS-AD consistently outperforms previous SOTA methods across all shot counts.

Shot	Metric	Ours	MAFR	CFM	M3DM	ShapeGuided
1-shot	I-AUROC	79.6	72.4	69.8	73.9	65.0
1-shot	AUPRO@30%	94.2	92.2	91.4	90.2	87.4
4-shot	I-AUROC	87.1	84.1	80.8	79.3	69.8
4-shot	AUPRO@30%	95.8	94.1	93.2	92.4	91.8

Our method also comprehensively leads on another dataset, EyeCandies: achieving 77.2 I-AUROC in the 1-shot setting (compared to the previous best of 69.5 by CIF) and 82.7 in 4-shot (compared to the previous best of 75.7 by MAFR). This validates the generalization capability of the method in highly reflective and complex textured candy scenarios.

Ablation Study¶

Ablation study on core components (4-shot, MVTec 3D-AD). "Full Config" refers to the complete CMDS-AD, "w/o Est Stream" denotes removing the estimation stream and keeping only the real stream, "w/o Real Stream" denotes removing the real stream and keeping only the estimation stream, and "w/o Feat Mapper" replaces the feature mapper with a standard MLP.

Configuration	I-AUROC	P-AUROC	AUPRO@30%	AUPRO@10%	AUPRO@5%	AUPRO@1%	Description
Full Config	87.1	98.9	95.8	88.6	80.6	41.0	Full model
w/o Est Stream	87.3	98.9	95.6	88.0	79.5	40.0	I-AUROC slightly increases without Est Stream, but localization drops under strict FPR
w/o Real Stream	81.3	97.9	93.2	83.1	73.2	35.0	Performance drops significantly relying only on the estimation stream
w/o Feat Mapper	87.2	98.8	95.5	87.7	79.2	39.9	All localization metrics systematically drop below the full model after replacing with standard MLP

Key Findings¶

The value of the estimation stream lies not in average metrics, but in robustness under strict FPR: Removing the estimation stream even slightly increases raw I-AUROC (87.3 vs 87.1), but AUPRO@1% drops from 41.0 to 40.0. This indicates that the role of the estimation stream is not to boost overall classification accuracy, but to provide low-frequency anchors to suppress normal high-frequency textures that are easily misclassified as defects. This is extremely critical for industrial scenarios where very low false alarm rates are required.
Multiplicative fusion is far superior to additive or maximum-value fusion: Five fusion strategies were evaluated (2D only, 3D only, Add, Max, Mul). Multiplicative fusion leads the second-best strategy by approximately 1.0 percentage point on AUPRO@1% (41.0 vs 40.1). The core reason is that multiplication acts as a strict spatial "AND" gate, and noise from a single modality cannot pass through.
Estimation stream weights are highly robust: Varying \(\lambda_1=\lambda_2\) from 0.1 to 0.9 results in a PRO@1% fluctuation of only about 1.6 points (MVTec). The default value of 0.1 is a robust operating point.
Greatest advantages are observed in categories with complex geometry: Significant improvements are observed in categories such as Bagel (97.0/97.0), Rope (97.1/97.6), and Carrot (92.0/98.1). However, gains are relatively small on slender rigid objects like Dowel (59.6/88.5), which might be due to the normal estimator over-smoothing thin structures.

Highlights & Insights¶

Repurposing the diffusion model as a low-pass filter rather than a mere data generator is the most ingenious aspect of this work: Most works treat diffusion models merely as enhancement tools to generate more samples. This paper discovers that the smoothing characteristics of normal estimators inherently serve as a frequency decoupling tool, creating a pure low-frequency structural reference space with zero extra training cost. This concept can be transferred to other tasks requiring frequency decomposition (such as structural prior extraction in image denoising or deblurring).
Replacing \(L_2\) distance with Cosine distance for cross-modal alignment: There is an immense numerical scale difference between RGB and 3D normals. Using cosine distance eliminates magnitude effects, ensuring alignment depends solely on directional consistency—this is a reusable technique for general cross-modal feature alignment tasks.
The pragmatic design of "omitting masks in the estimation stream": Because the estimated normal maps have blurred boundaries and lack precise object masks, the authors simply let the estimation stream use the global mean. The reasoning is intuitive: pure low-frequency signals are evenly distributed across the entire image, making foreground focusing unnecessary. This design honesty (avoiding forced symmetry) is highly commendable.
Astute selection of the AUPRO@1% metric: AUPRO@30% is overly lenient for industrial scenarios (allowing a 30% FPR would cause a massive amount of good products to be intercepted for manual reinspection). Focusing on more stringent FPR thresholds (1%, 5%, 10%) is what actual deployments care about. The appendix provides detailed category-level fine-grained analyses, making the results highly reproducible and practically useful.

Limitations & Future Work¶

Inability to detect "missing" defects: When part of an object is missing (e.g., a missing corner of a cookie), the network cannot produce anomaly signals on the missing pixels; instead, the anomaly responses are forced to shift to the edges of the damage. This is an inherent physical limitation. A potential future direction is to introduce external shape prior or contrastive comparison with complete category templates.
Over-segmentation issues: The anomaly response of micro-defects tends to "bleed" into surrounding smooth areas. The root cause is that the low-pass filtering nature of the estimation stream causes high-frequency signals to diffuse to neighboring regions during multi-scale feature comparison. This can be addressed by introducing edge-aware post-processing or more refined denoising schedules.
False alarms under hyper-complex textures: On objects with highly frequent and dramatic variations, such as Candy Cane, low-frequency anchors cannot cover all normal texture variations, leading to occasional false alarms. This fundamentally shows that frequency decomposition alone cannot completely solve the few-shot problem for extremely complex textures; a secondary validation combined with semantic-level priors (like CLIP embeddings) may be needed.
Insufficiency of purely 2D alignment metrics: The P-AUROC used in ablation studies has already saturated (consistently above 98%). More discriminative dense evaluation metrics are needed in the future. The authors note this in the appendix but do not propose an alternative.

vs CFM: CFM performs cross-modal feature mapping but treats 2D\(\to\)3D and 3D\(\to\)2D as symmetric tasks. CMDS-AD explicitly addresses this asymmetry—as 2D and 3D domains have different numerical scales, it bypasses magnitude alignment with cosine distance and sets up independent guide estimation streams within each modality.
vs M3DM / ShapeGuided: These methods rely on memory banks or shape guidance, needing large quantities of normal samples to fill the feature space. CMDS-AD's dual-stream design does not rely on memory banks, thus solving the data scarcity issue from an architectural level.
vs AST: AST employs an asymmetric teacher-student architecture for anomaly detection, but concatenating 2D and 3D inputs leads to severe modal interference. CMDS-AD circumvents interference through dual-track independent processing and multiplicative scoring.
Connection to frequency-decomposition works: Traditional frequency-domain methods typically rely on hand-designed filters (high-pass/low-pass/band-pass). CMDS-AD is the first to systematically apply the inherent frequency preference (low-frequency bias) from diffusion model training to few-shot anomaly detection decoupling, presenting an interesting case of cross-domain transfer.

Rating¶

Novelty: ⭐⭐⭐⭐ Repurposing a diffusion model as a frequency decoupling tool rather than just a data augmenter is a very unique perspective; the dual-stream decoupling concept has not been systematically explored before. One star is docked because the individual engineering building blocks themselves are not entirely new (LoRA, Coordinate Attention, and Cosine Loss are existing standard components).
Experimental Thoroughness: ⭐⭐⭐⭐⭐ The experiments cover MVTec 3D-AD and EyeCandies, testing 1, 2, and 4-shot settings against more than 5 baseline methods. It features comprehensive ablations (components, fusion strategies, weight sensitivity, and stability across 5 seeds) along with empirical FFT spectrum validation of the frequency decoupling hypothesis. The chain of validation is highly complete.
Writing Quality: ⭐⭐⭐⭐ The logic flow is clear and cohesive (starting from background \(\to\) limitations \(\to\) challenges \(\to\) core idea). Key concepts (such as why the diffusion estimator acts as a low-pass filter, and why multiplicative fusion is effective) are thoroughly explained. The appendix is robust. One star is docked because the main text has a high density of tables but face spatial constraints, forcing some ablations and analyses into the appendix to the reader's inconvenience.
Value: ⭐⭐⭐⭐⭐ Few-shot multimodal anomaly detection is a vital real-world scenario for industrial quality inspection. Current SOTA methods achieve only ~74% I-AUROC in the 1-shot setting, which this paper elevates to ~80%, yielding direct practical benefit. The concept of using diffusion models as frequency decoupling tools can also inspire other tasks that require low-frequency structural prior extraction.