GroundingAnomaly: Spatially-Grounded Diffusion for Few-Shot Anomaly Synthesis¶

Conference: ECCV 2026
Paper: ECCV Official
Area: Segmentation
Keywords: anomaly synthesis, anomaly detection, diffusion models, few-shot generation, spatial grounding

TL;DR¶

GroundingAnomaly introduces a few-shot anomaly image generation framework based on a pretrained diffusion model, fusing dense semantic maps with disentangled tokens via a Spatial Conditioning Module and injecting them into a frozen U-Net through gated self-attention layers to achieve spatially accurate, photorealistic defect synthesis.

Background & Motivation¶

Visual anomaly inspection plays a crucial role in modern industrial quality control. However, defective samples are exceedingly rare on real-world manufacturing lines, which severely limits the performance of supervised object detection and segmentation architectures. Conventional unsupervised inspection paradigms rely solely on modeling the distribution of normal products to identify out-of-distribution deviations. Yet, these methods suffer from coarse anomaly localization and inherently lack class-aware defect identification capabilities. To augment defect data, anomaly synthesis methods built upon large-scale generative priors have attracted growing interest.

Nonetheless, current generative approaches face a fundamental trade-off. Anomaly Generation (AG) techniques synthesize defect patches and inpaint them onto real normal backgrounds; while this preserves background fidelity, inpainting frequently produces visible boundary artifacts, unnatural lighting transitions, and misaligned masks. Conversely, Anomaly Image Generation (AIG) methods jointly synthesize objects and defects to ensure global contextual coherence; however, they derive defect masks post-hoc from low-resolution cross-attention maps or external segmentors, failing to deliver pixel-level mask precision.

The core tension is that an effective synthesizer must generate the entire image coherently while accepting precise, fine-grained geometric conditioning from an input mask to provide ground-truth annotations for downstream models. The core idea of this paper is to leverage dense, multi-class semantic maps fused with disentangled product-defect tokens and inject this spatial-semantic conditioning into a frozen diffusion U-Net via gated self-attention, achieving pixel-grounded full-image anomaly synthesis.

Method¶

Overall Architecture¶

GroundingAnomaly is built upon a pretrained Stable Diffusion backbone and comprises three primary stages: first, constructing pixel-level spatial-textual conditioning tokens via dense semantic maps and learnable token embeddings; second, fusing spatial ConvNeXt features with broadcasted text embeddings inside the Spatial Conditioning Module (SCM); and third, injecting these fused conditioning tokens into a frozen U-Net through a Gated Self-Attention Module (GSM) during reverse diffusion.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Dense Semantic Map S<br/>+ Disentangled Prompt Tokens"] --> B["Spatial Conditioning Module (SCM)<br/>ConvNeXt encoding + spatial text broadcast"]
    B --> C["Gated Self-Attention Module (GSM)<br/>Concatenate with visual tokens + Tanh gating"]
    C --> D["Mixed Normal-Anomalous Training (MNT)<br/>Unified U-Net regularization across defects"]
    D --> E["Normal-Prior Denoising Initialization (NDI)<br/>Reverse denoising from forward-noised normal latent"]

Key Designs¶

1. Spatial Conditioning Module (SCM): Dense Alignment of Semantic and Textual Features To overcome the limitations of binary masks that lack defect class identity and text-only embeddings that lack spatial precision, SCM introduces an integer-valued semantic map $S(x, y) \in \{0, 1, \dots, C\}$ where $0$ represents background and $l \in \{1, \dots, C\}$ indicates a specific anomaly type. The map is encoded by a pretrained ConvNeXt into spatial features $F_S \in \mathbb{R}^{H_f \times W_f \times D_f}$. Simultaneously, learnable product tokens and defect tokens are projected via an MLP and broadcasted across spatial locations according to the downsampled semantic label $S_f$: $$F_T(i, j) = \begin{cases} \tilde{e}_{\mathrm{pro}}, & \text{if } S_f(i, j) = 0 \\ \tilde{e}_{\mathrm{ano}, l}, & \text{if } S_f(i, j) = l \end{cases}$$ Concatenating $F_S$ and $F_T$ along the feature channel and projecting to dimension $D_v$ yields conditioning tokens $c \in \mathbb{R}^{N_c \times D_v}$, establishing strict pixel-level correspondence between spatial regions and defect classes.

2. Gated Self-Attention Module (GSM): Zero-Initialized Non-Destructive Conditioning Fine-tuning the U-Net backbone directly in a few-shot setting frequently causes catastrophic forgetting and severe overfitting, whereas optimizing only textual prompts fails to achieve localized geometric grounding. GSM resolves this by freezing the original U-Net weights and inserting gated self-attention layers between standard self-attention and cross-attention blocks. Conditioning tokens $c$ are concatenated with visual features $v$ into $h_c = [v; c]$. The visual features are then updated with a learnable scalar $\gamma$ initialized to $0$: $$v \leftarrow v + \tanh(\gamma) \cdot \mathrm{Select}_v\big(\mathrm{SelfAttn}(h_c)\big)$$ Combined with lightweight LoRA adapters in the attention layers, this ensures that pretrained priors are strictly intact at the start of training and smoothly adapt to spatial constraints.

3. Mixed Normal-Anomalous Training (MNT): Cross-Domain Prior Sharing and Regularization Because abnormal samples are scarce and their uncorrupted regions lack visual variation, fine-tuning solely on abnormal exemplars leads to background collapse. MNT trains a unified model per product across all anomaly classes, sampling mini-batches from both normal and anomalous subsets. Normal images are paired with an all-zero semantic map $S=0$ and a clean prompt, forcing the model to reconstruct defect-free backgrounds while learning localized defect characteristics.

4. Normal-Prior Denoising Initialization (NDI): Fast Sampling with Seamless Background Integration Sampling from pure Gaussian noise $z_T \sim \mathcal{N}(0, I)$ often introduces minor background distortions or unnatural object textures. NDI initializes the reverse diffusion trajectory from a partially noised latent $z_{t'}$ obtained by forward-diffusing a real normal image latent $z_0$ up to timestep $t' < T$. Denoising for only $t'$ steps seamlessly embeds synthesized defects into real high-frequency product backgrounds while substantially speeding up inference.

Loss & Training¶

The overall training objective is the standard conditional latent diffusion loss: $$\mathcal{L}_{\mathrm{LDM}} = \mathbb{E}_{x \sim p_{\mathrm{data}},\,\epsilon \sim \mathcal{N}(0, I),\,t} \left[ \|\epsilon - \epsilon_\theta(z_t, t, c)\|^2 \right]$$ To overcome the scarcity of real anomaly masks, a dedicated mask token embedding $e_m$ is learned via Textual Inversion on available masks to generate diverse novel mask contours during inference.

Key Experimental Results¶

Main Results¶

Synthetic images produced by GroundingAnomaly and baseline generators are used to train downstream U-Net segmentation models on MVTec AD and VisA:

Dataset	Metric	Ours	DualAnoDiff (Prev. SOTA)	Gain
MVTec AD	AUC-P (%)	99.3	99.1	+0.2
MVTec AD	AP-P (%)	85.9	84.5	+1.4
MVTec AD	F1-P (%)	81.8	78.7	+3.1
MVTec AD	AP-I (%)	99.2	99.0	+0.2
VisA	AUC-P (%)	98.2	96.6	+1.6
VisA	AP-P (%)	67.2	60.9	+6.3
VisA	F1-P (%)	63.5	61.4	+2.1
VisA	AP-I (%)	96.0	94.7	+1.3

In addition, GroundingAnomaly delivers superior image quality and diversity: Inception Score (IS) reaches 1.99 on MVTec AD (vs. 1.91 for DualAnoDiff) and 1.29 on VisA (vs. 1.27 for SeaS), while IC-LPIPS achieves 0.41 on MVTec AD and 0.31 on VisA.

Ablation Study¶

Ablation experiments evaluate the contribution of each module toward generation quality and downstream segmentation performance:

Config	MVTec IS	MVTec AP-P (%)	VisA IS	VisA AP-P (%)	Note
Full model	1.99	85.9	1.29	67.2	Full proposed framework
w/o DTL	1.88	81.7 (-4.2)	1.21	62.3 (-4.9)	Fixed text prompt tokens
w/o SFF	1.72	77.3 (-8.6)	1.13	59.1 (-8.1)	No spatial-textual feature fusion
w/o GSM	1.63	75.7 (-10.2)	1.09	58.1 (-9.1)	Standard cross-attention conditioning
w/o MNT	1.74	83.1 (-2.8)	1.17	62.9 (-4.3)	Trained solely on anomalous samples
w/o NDI	1.94	84.7 (-1.2)	1.25	64.9 (-2.3)	Sampling from pure Gaussian noise

Key Findings¶

GSM and SFF are essential: Discarding GSM drops pixel AP by 10.2% on MVTec AD and 9.1% on VisA; removing SFF drops pixel AP by 8.6% and 8.1%, respectively, demonstrating that direct spatial injection into the frozen backbone is critical for boundary precision.
Substantial gains on fine-grained defects: On challenging MVTec AD categories such as grid, pixel AP improves by 5.9% over DualAnoDiff (73.6% vs. 67.7%). On VisA capsules and macaroni2, AP gains reach 16.6% and 15.2%, respectively.
Superior utility for instance detection: When training object detectors (Faster R-CNN, DETR, YOLOv5) on synthetic data, GroundingAnomaly yields an average mAP of 44.84% on MVTec AD (+1.52% over DualAnoDiff) and 38.91% on VisA (+2.34% over DualAnoDiff).

Highlights & Insights¶

Dense Spatial-Semantic Feature Broadcast: Fusing ConvNeXt geometric feature maps with spatially broadcasted text tokens ensures that the diffusion process receives unambiguous pixel-level semantic instructions.
Non-Invasive Gated Self-Attention: By initializing gate scalars to zero and keeping the U-Net backbone frozen, GroundingAnomaly eliminates catastrophic forgetting while allowing rapid few-shot adaptation via LoRA.
Multi-Class Compositional Synthesis: Constructing composite semantic maps $S = S_1 + S_2$ enables the synthesis of co-occurring defects across multiple classes on a single product without requiring combinatorial training samples.

Limitations & Future Work¶

Reliance on Mask Priors: While Textual Inversion enriches mask diversity, extreme 1-shot scenarios still suffer from limited geometric variability in synthesized mask contours.
Complex 3D Geometry and Reflective Surfaces: The 2D diffusion formulation struggles with complex 3D specular reflections, deep gouges, and view-dependent metallic highlights.
Future Directions: Exploring zero-shot cross-category defect transfer and extending the framework to modern Diffusion Transformer (DiT) architectures.

vs AnomalyDiffusion [15]: AnomalyDiffusion relies on inpainting defect patches onto normal images, leaving border stitching artifacts and misaligned masks; GroundingAnomaly synthesizes the whole image jointly while enforcing strict spatial grounding.
vs SeaS [40]: SeaS extracts masks post-hoc from low-resolution attention maps or external segmentors, yielding blurry edges; GroundingAnomaly conditions generation on explicit semantic maps, providing ground-truth masks with exact pixel alignment.

Rating¶

Novelty: ⭐⭐⭐⭐☆ [Combines dense semantic mapping with non-invasive gated self-attention for diffusion-based anomaly synthesis]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations on MVTec AD and VisA across generation quality, segmentation, and object detection]
Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, well-formulated methodology, and clean experimental presentation]
Value: ⭐⭐⭐⭐☆ [A practical and highly effective data augmentation pipeline for industrial visual inspection]