PADFormer: Pose-agnostic Anomaly Detection from Sparse View Images

Conference: ECCV 2026
Paper: ECCV Original
Area: Object Detection
Keywords: Pose-agnostic Anomaly Detection, Sparse View Images, Masked Image Modeling, Spatial Transformation Network, Dynamic Patch Selection

TL;DR

Tackling the challenge of inspecting objects under arbitrary viewpoints with only sparse unposed reference images, PADFormer bypasses cumbersome 3D reconstruction and introduces a 2D ViT-based cross-view masked reconstruction framework that integrates spatial feature alignment, dynamic patch selection, and anomaly-agnostic prior tokens to detect defects via multi-pass ensemble inference.

Background & Motivation

Unsupervised visual anomaly detection plays a pivotal role in industrial quality control and manufacturing. Conventional reference-free approaches and few-shot anomaly detection (FSAD) frameworks are predominantly predicated on a restrictive pose-aligned assumption, where query samples and defect-free references must be captured from predefined, identical camera viewpoints. However, in realistic assembly and robotic inspection scenarios, manufactured parts are frequently placed in arbitrary, unconstrained orientations, and defects can emerge at unpredictable angles or under partial perspective occlusions. When significant geometric pose variations exist between test samples and reference views, conventional 2D patch-matching and global registration pipelines deteriorate catastrophically.

To overcome the rigid constraint of fixed viewpoints, the pose-agnostic anomaly detection (PAD) setting has garnered widespread attention. Prevailing PAD solutions primarily adopt explicit 3D geometric reconstruction pipelines—exemplified by NeRF- and 3D Gaussian Splatting-based frameworks such as OmniAD, SplatPose, and PIAD. These techniques inherently demand extensive multi-view reference sets (frequently 200+ images) endowed with ground-truth camera pose calibrations to reconstruct dense 3D representations, followed by iterative test-time pose estimation and cross-view rendering alignment for every single test image. The reliance on dense posed reference imagery and the substantial computational burden of 3D synthesis represent severe bottlenecks that impede low-cost, real-world deployment.

In contrast to computationally demanding 3D reconstruction, an appealing alternative is to conduct pose-consistent defect detection directly within the 2D image feature space. By training an asymmetric cross-view masked autoencoder exclusively on anomaly-free samples, the model learns to synthesize a pose-identical, anomaly-free counterpart of any arbitrary query image, replacing corrupted local regions with normal patterns synthesized from sparse reference views. Core idea: bypass explicit 3D reconstruction and iterative pose optimization by introducing a 2D ViT-based masked cross-view reconstruction framework that aligns sparse reference views via spatial transformation networks, dynamically retrieves informative patches, and performs multi-pass stochastic masking with CIELAB variance filtering for robust anomaly detection.

Method

Overall Architecture

PADFormer leverages an end-to-end transformer architecture designed to directly reconstruct a pose-consistent, anomaly-free counterpart \(\tilde{Q}\) given an arbitrary query image \(Q\) and a sparse set of \(N\) unposed normal reference images \(\{R_i\}_{i=1}^N\) (e.g., \(N=2\) to \(4\)). The pipeline comprises asymmetric masked ViT tokenization, Spatial Transformation Network (STN) patch alignment, a hybrid Dynamic Patch Selection (DPS) module combining semantic relevance and spatial proximity, and a cross-attention decoder incorporating both global and local Anomaly-Agnostic Tokens (AAT). At inference time, the model executes multi-pass stochastic masking and computes pixel-wise residuals in CIELAB color space, filtering spurious reconstruction artifacts via a variance consistency mask to produce robust anomaly scores.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Query Image Q + Sparse References {Ri}"] --> B["Asymmetric Masked ViT Encoding<br/>40% random masking on Q, unmasked references"]
    B --> C["Spatial Transformation Network Feature Alignment<br/>Predict affine parameters θi and bilinearly resample"]
    C --> D["Dynamic Patch Selection<br/>Combine feature cosine similarity and spatial proximity"]
    D --> E["Cross-Attention Decoder with Anomaly-Agnostic Tokens<br/>Attend to 32 global tokens, M local tokens, and Top-k patches"]
    E --> F["Multi-pass Ensemble Inference & CIELAB Filtering<br/>15 stochastic masking passes with variance consistency mask"]
    F --> G["Output: Pixel-level Anomaly Map & Image-level Score"]

Key Designs

1. Spatial Transformation Network Feature Alignment: Mitigating Pose Mismatches Across Views

Under sparse-view regimes, substantial geometric viewpoint discrepancies exist between reference images and the query image. Calculating standard unconstrained attention over raw patch tokens leads to diffuse attention distributions and severe feature misalignment. To resolve this geometric disparity, PADFormer incorporates a lightweight Spatial Transformation Network (STN) operating directly on patch-level feature grids. For each reference image \(R_i\), the masked query feature grid \(\hat{\mathbf{f}}_\text{q} \in \mathbb{R}^{M \times d}\) (with masked tokens filled by learnable mask embeddings) and the reference feature grid \(\mathbf{f}_{\text{r}i} \in \mathbb{R}^{M \times d}\) are reshaped into 2D spatial feature maps and concatenated along the channel dimension. A compact localization network—consisting of two convolutional layers (\(7 \times 7\) and \(5 \times 5\) kernels) followed by max pooling and fully connected layers—predicts a 6-dimensional affine transformation matrix \(\theta_i\). This transformation is applied strictly to the reference feature map via bilinear interpolation, leaving the query coordinate frame untouched. This patch-level alignment significantly contracts the spatial discrepancy before feature matching, ensuring geometrically consistent cross-view correspondence.

2. Dynamic Patch Selection: Fusing Semantic Relevance and Spatial Proximity

Attending to all \(N \cdot M\) reference patches across multiple images creates substantial computational overhead and introduces distant, irrelevant visual context that degrades fine-grained reconstruction quality. PADFormer addresses this by dynamically selecting the Top-\(k\) most relevant reference patches for each query position. To strike an effective balance between semantic appearance matching and physical locality, the matching score integrates normalized cosine similarity with a Gaussian spatial proximity prior:

\[S^{jl} = (1 - w) \cdot \left\langle \frac{\hat{\mathbf{f}}_\text{q}^j}{\|\hat{\mathbf{f}}_\text{q}^j\|}, \frac{\tilde{\mathbf{f}}_\text{ref}^l}{\|\tilde{\mathbf{f}}_\text{ref}^l\|} \right\rangle + w \cdot \exp\left( -\frac{\|\mathbf{p}_q^j - \mathbf{p}_{\text{ref}}^l\|^2}{2\sigma^2} \right)\]

where \(\mathbf{p}_q^j\) and \(\mathbf{p}_{\text{ref}}^l\) denote the 2D grid coordinates of the query and aligned reference patches, \(w=0.3\) balances feature similarity against spatial locality, and \(\sigma=2.0\) controls the Gaussian bandwidth. For each query token, only the Top-\(k\) (\(k=10\)) candidate patches \(s_j \in \mathbb{R}^{k \times d}\) are admitted to the decoder cross-attention, drastically accelerating computation and filtering out cross-view visual distractors.

3. Cross-Attention Decoder with Anomaly-Agnostic Tokens: Suppressing Defect Leakage

When query regions harbor conspicuous defects, relying solely on self-attention may inadvertently lead the model to faithfully reconstruct the anomalous textures, while sparse reference views might fail to cover the occluded background. To impose a strong normal manifold prior, PADFormer introduces learnable Anomaly-Agnostic Tokens (AAT) within an 8-layer ViT decoder. These tokens function as a non-parametric normal memory bank comprising: (1) 32 global tokens \(T_{\text{global}} \in \mathbb{R}^{32 \times d'}\) shared across all spatial locations to capture macro-level normal appearances and structural primitives; and (2) \(M\) position-indexed local tokens \(T_{\text{local}} \in \mathbb{R}^{M \times d'}\) providing location-aware structural templates. During cross-attention, each query token attends to a composite key/value set formed by concatenating the global tokens, all flattened local tokens, and the retrieved Top-\(k\) aligned reference patches \(s_j\). Because these memory tokens are trained exclusively on anomaly-free samples, they strictly bias the reconstruction toward normal industrial surfaces, preventing defect leakage and ensuring accurate restoration of nominal patterns.

4. Multi-pass Ensemble Inference & CIELAB Filtering: Variance Masking for Artifact Suppression

To guarantee that anomalous pixels are thoroughly scrutinized, PADFormer performs \(I=15\) stochastic inference passes during testing, each utilizing an independent 40% random masking pattern. With a mask ratio of \(r=0.4\), the cumulative probability of any individual patch being masked at least once reaches \(1 - (1-0.4)^{15} \approx 99.95\%\). Reconstructed images are upsampled to the original spatial resolution, and perceptual reconstruction errors are evaluated in CIELAB color space. A Gaussian blur (\(\sigma=1.4\), kernel size 7) is applied to the query channels to suppress high-frequency image acquisition noise before computing squared differences against the unfiltered reconstruction:

\[E_i(x,y) = \sum_{c \in \{L, a, b\}} \left( \mathcal{G}_\sigma(c(Q_\text{test})) - c(\hat{Q}_i) \right)^2\]

Crucially, industrial samples frequently exhibit benign batch variations, minor color casts, or illumination drifts. Because these variations are part of the normal data distribution, the model reconstructs them consistently across all masking patterns, resulting in low variance across passes: \(\text{Var}(x, y) = \frac{1}{I} \sum_{i=1}^I (E_i - \bar{E})^2\). In contrast, genuine structural anomalies lack supportive reference priors and reconstruct erratically under varying visible contexts, producing high variance. PADFormer builds a consistency mask \(\mathcal{M}(x,y) = \mathbb{I}[\text{Var}(x,y) > P_{50}(\text{Var})]\) retaining only regions exceeding the 50th variance percentile. Multiplying the average error by this mask, \(\mathcal{E}(x,y) = \bar{E}(x,y) \mathcal{M}(x,y)\), effectively suppresses systematic reconstruction artifacts and false-positive edge discrepancies.

Loss & Training

PADFormer follows a progressive two-stage training scheme utilizing exclusively anomaly-free images: - Within-view Pretraining (50 epochs): Reference images are synthesized directly from each query image via random rotations and elastic deformations. This forces the model to learn geometric invariance and robust intrinsic feature representations under controlled synthetic transformations. - Cross-view Unified Training (250 epochs): The model is jointly trained across all object categories within a dataset. References are drawn from different viewpoint images of the same object class, fostering category-level representations resilient to 3D orientation shifts. - Objective Function: Both stages optimize a weighted combination of Mean Squared Error and Perceptual Loss:

\[\mathcal{L} = \text{MSE}(Q, \tilde{Q}) + \lambda \text{Perceptual}(Q, \tilde{Q})\]

The framework employs a pretrained MAE ViT backbone, optimized using a Cosine learning rate scheduler with a peak learning rate of \(4 \times 10^{-4}\) and a 2,500-step warmup period.

Key Experimental Results

Main Results

PADFormer is rigorously evaluated against established 3D-based pose-agnostic methods (OmniAD, SplatPose, PIAD) and competitive few-shot anomaly detection models (PromptAD, UniVAD) on the MAD-SIM and PIAD benchmarks. In low-shot, sparse-view configurations, PADFormer achieves decisive performance margins:

Dataset	Setup	Metric	PADFormer (Ours)	Best Baseline	Gain
MAD-SIM	2-shot	Image-AUROC (%)	78.8	57.9 (UniVAD)	+20.9%
MAD-SIM	2-shot	Pixel-AUROC (%)	89.2	82.2 (PIAD)	+7.0%
MAD-SIM	4-shot	Image-AUROC (%)	81.3	60.3 (PromptAD)	+21.0%
MAD-SIM	4-shot	Pixel-AUROC (%)	92.9	86.7 (SplatPose)	+6.2%
MAD-SIM	4-shot	AUPRO (%)	86.7	74.6 (SplatPose)	+12.1%
MAD-SIM	10-shot	Image-AUROC (%)	85.6	58.2 (PIAD)	+27.4%
MAD-SIM	All-shot	Image-AUROC (%)	97.8	97.4 (PIAD)	+0.4%
PIAD	2-shot	Image-AUROC (%)	75.6	54.3 (UniVAD)	+21.3%
PIAD	4-shot	Image-AUROC (%)	80.7	59.2 (UniVAD)	+21.5%
PIAD	4-shot	Pixel-AUROC (%)	91.2	84.4 (PIAD)	+6.8%
PIAD	All-shot	Image-AUROC (%)	95.4	94.7 (PIAD)	+0.7%

Note: In the All-shot setting, competing 3D reconstruction baselines require resource-intensive per-object model fitting with precise pose annotations. In contrast, PADFormer trains a single unified multi-category model without any pose supervision, matching or exceeding 3D baselines.

Ablation Study

Ablation experiments conducted on MAD-SIM under the 4-shot setup systematically demonstrate the efficacy of each architectural design:

Model Configuration (STN / DPS / AAT)	Image-AUROC (%)	Pixel-AUROC (%)	Note
Baseline (CroCo-style cross-attention)	76.8	80.2	Unaligned global attention without memory priors
+ Dynamic Patch Selection (DPS)	78.2	85.4	Top-k patch retrieval filters spatial noise (+5.2% Pixel)
+ STN Feature Alignment + DPS	79.1	89.2	Affine pre-alignment eliminates viewpoint offsets (+3.8% Pixel)
+ STN + DPS + AAT (Full Model)	81.3	92.9	Memory tokens enforce normal texture synthesis (+3.7% Pixel)

Ablation on error evaluation spaces and computational latency:

Error Metric Space	Image-AUROC (%)	Pixel-AUROC (%)	Inference Time (s) ↓	Characteristics & Trade-offs
RGB Space	78.8	86.4	0.6	Fastest, but lacks perceptual uniformity; sensitive to edge noise
CIELAB Space (Default)	81.3	92.9	0.8	Optimal trade-off; perceptually uniform with highest pixel localization
ViT Feature Space	83.5	91.3	2.1	Marginally higher image AUROC, but ~3× slower and lower pixel accuracy

Key Findings

• Collapse of 3D Pipelines under Sparse Views: When constrained to 2-shot or 4-shot references, NeRF- and 3DGS-based models degrade to near-chance image-level AUROC (~48% to 52%), as sparse viewpoints fail to converge dense point geometry or resolve camera trajectories. PADFormer circumvents 3D synthesis entirely, maintaining robust image-level AUROCs above 80%.
• Synergy of Spatial Alignment and Dynamic Selection: Dynamic Patch Selection provides a 5.2% boost in pixel-level AUROC, proving that unconstrained attention over irrelevant reference patches introduces detrimental noise. The STN affine transform provides an additional 3.8% gain by resolving perspective skew prior to similarity calculation.
• Efficacy of Variance-based Masking: Consistency filtering (retaining regions above the 50th variance percentile) yields an absolute 2.4% gain in Image-AUROC by isolating unstable anomaly reconstructions from uniform lighting variations.

Highlights & Insights

• Paradigm Shift from 3D to 2D Feature Synthesis: Prior work maintained that unconstrained viewpoint inspection fundamentally necessitated dense 3D reconstruction and iterative pose fitting. PADFormer overturns this assumption by showing that a 2D masked cross-attention framework with spatial affine alignment can effectively synthesize pose-conditioned, anomaly-free query representations from sparse, unposed references.
• Variance Consistency Filtering: Leveraging the variance across multiple stochastic masked reconstruction passes to differentiate genuine defects from pervasive illumination drift provides an elegant, parameter-free mechanism for false-positive reduction in industrial vision.
• Category-Unified Generalization: Unlike 3D methods that require fitting specialized models per individual object instance, PADFormer demonstrates strong cross-category generalization when trained on diverse industrial categories in a single unified model.

Limitations & Future Work

• Fixed Patch Granularity: Adopting a rigid \(16 \times 16\) patch partition limits localization precision for minuscule, sub-patch hairline cracks and introduces slight boundary ambiguity for expansive, multi-patch defects.
• Inference Latency from Multi-pass Integration: Achieving 99.95% spatial coverage requires 15 forward passes (~0.8s per sample). While significantly faster than 3D test-time optimization, this latency remains challenging for ultra-high-speed production lines (e.g., 60+ FPS). Future investigations could explore deterministic minimal-covering mask patterns to reduce the budget to 3–4 passes.
• Extreme Perspective Extrapolations: When reference viewpoints and the query image have minimal visual overlap (e.g., opposite sides of an opaque object), 2D affine feature warping cannot hallucinate unseen geometry, suggesting the potential benefit of lightweight coarse pose cues for extreme viewpoint shifts.

Related Work & Insights

• vs OmniAD / SplatPose / PIAD: These methods rely on dense 3D radiance fields or Gaussian splats, necessitating 200+ posed training views and test-time gradient-based camera pose optimization. PADFormer operates in 2D image feature space, requires only 2–4 unposed reference views, and avoids test-time pose optimization altogether.
• vs UniVAD / PromptAD (FSAD Baselines): Conventional few-shot anomaly detection models presuppose viewpoint-aligned imagery. When tested under unconstrained 3D orientations, their feature alignment degrades severely. PADFormer bridges the gap between data-efficient FSAD and pose-agnostic PAD.
• vs CroCo / MAE (Cross-view Masked Modeling): While CroCo targets general 3D multi-view correspondence on uncurated scenes, PADFormer tailors cross-view masked reconstruction specifically for anomaly detection by training exclusively on defect-free samples and integrating anomaly-agnostic memory tokens to guarantee defect-free inpainting.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Pioneers a 2D ViT-based masked reconstruction framework for pose-agnostic anomaly detection without 3D reconstruction or pose supervision]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Thoroughly evaluates 2/4/10/All-shot settings across MAD-SIM, PIAD, MVTec-AD, and VisA benchmarks with detailed component ablations]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, well-structured methodology, and crisp analysis of geometric alignment and variance consistency]
• Value: ⭐⭐⭐⭐⭐ [Significantly relaxes data capture and computational constraints for automated industrial inspection under arbitrary viewpoints]

Related Papers

• [ECCV 2026] MATCH: Flow Matching for Multi-View Anomaly Detection
• [ECCV 2026] Efficient RGB-T Object Detection via Sparse Cross-Modality Fusion
• [ICCV 2025] Toward Long-Tailed Online Anomaly Detection through Class-Agnostic Concepts
• [ECCV 2026] Low-latency Event-based Object Detection with Spatially-Sparse Linear Attention
• [ICLR 2026] Long-Context Generalization with Sparse Attention