IMMoE: Incomplete Multi-View Anomaly Detection via Mixture of View Experts Fusion¶

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/HULEI7/IMMoE
Area: Anomaly Detection
Keywords: incomplete multi-view, industrial anomaly detection, mixture of experts, feature reconstruction, area-adaptive loss

TL;DR¶

IMMoE uses local feature dropout to reduce overfitting to masking patterns and view-expert fusion to guide normal-feature reconstruction, achieving 84.7 I-AUROC / 33.1 P-AP on RIMAD with 50% of the product area masked, without fully establishing the recoverability of hidden defects under real occlusion.

Background & Motivation¶

Industrial components often require inspection from multiple angles because one image cannot cover every surface. However, applying the same anomaly detector independently to each view does not mean that it learns complementary relationships between views. When part of a view is occluded, a single-view reconstruction model must rely on the remaining content, potentially losing defect evidence or treating artificial missing-region boundaries as anomalies. Dinomaly and INP-Former provide strong feature-reconstruction baselines, but this paper targets their behavior under locally incomplete inputs rather than redefining industrial anomaly detection as a whole.

Existing fusion methods such as MVAD exchange information across views, yet fusion can also propagate masking artifacts into other views. The problem is therefore not just whether additional cameras exist, but whether each view overfits missing-data patterns and whether training spends too much effort on easy background regions. The authors formulate incomplete multi-view anomaly detection as a setting in which every view is present but some product pixels are unavailable. This differs from complete camera failure, a missing entire view, or an absent sensor modality, and these settings should not be conflated.

The resulting RIMAD benchmark randomly removes local content from product regions in Real-IAD while retaining the original images as reconstruction references. The method first stabilizes individual view features, then learns which view representations to fuse, and finally concentrates training on difficult product regions. Core Idea: reduce reliance on missing-data artifacts through local feature dropout, form shared context with view experts, and feed that context back into normal-feature reconstruction for each view.

Method¶

Overall Architecture¶

The input consists of incomplete views of the same object and corresponding binary missing-region masks; Real-IAD supplies 5 views per object. The mask assigns 0 to missing pixels and 1 to valid pixels; it is not a defect annotation and does not directly indicate anomalies. During training, a shared-weight pretrained encoder extracts features from both incomplete images and original reference images. The reconstruction branch then passes through the Local Anomaly Enhancement Encoder (LAEE), Multi-view Expert Fusion (MVEF), and a reconstruction decoder to restore per-view features. Area Adaptive Loss (AAL) controls gradient weighting during training only; at test time, discrepancies between reference and reconstructed features yield anomaly maps. Although the task definition speaks of reconstructing the original normal image, the implemented supervision operates on features rather than RGB outputs.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Incomplete multi-view images"] --> B["Shared pretrained encoder"]
    B --> C["Local Anomaly Enhancement Encoder<br/>LAEE"]
    C --> D["Multi-view Expert Fusion<br/>MVEF"]
    D --> E["Reconstruction decoder"]
    R["Original images<br/>Shared-encoder reference features"] --> F["Feature discrepancy"]
    E --> F
    F -->|Training| G["Area Adaptive Loss<br/>AAL"]
    M["Missing-region and product masks"] -->|Training region priors| G
    F -->|Testing| H["Anomaly map"]

The original-image reference branch follows Figure 3 and Section 4.1 of the paper, rather than adding an assumption that normal test ground truth is available. The paper does not adequately explain how this reference is obtained under real test-time occlusion, which affects the deployment interpretation discussed below. SAM generates product masks for dataset construction; it is not an online fusion module that must be invoked for every anomaly decision. The generator places random rectangular missing regions inside the product until 50% of its area is masked; the text specifies an area of 8–32 square pixels per rectangle. This note preserves that area unit instead of silently interpreting it as a rectangle side length.

Key Designs¶

1. Local Anomaly Enhancement Encoder: remove dependence on fixed missing-data patterns

Why discard more features when some input pixels are already missing? The authors argue that persistent missing-region boundaries or surviving local cues can encourage the network to fit masking patterns instead of learning normal product structure. LAEE therefore applies random dropout to each view's patch features, with a default rate of 0.1. This drops feature elements; it does not remove an additional 10% of the original image area or generate labeled real defects. Each view has \(N\) patches with \(C\) feature dimensions, and LAEE preserves the basic interpretation of these spatial indices. After dropout, linear attention aggregates context from surviving locations so that damaged local representations can access broader information.

Linear attention is used to avoid the increasing cost of standard self-attention as the number of patches grows. The paper presents a rearranged multiplication of mapped queries and keys, but the cache does not fully specify activation and normalization details, so it is not expanded here into an exact reproducible operator. This module mainly improves within-view robustness and cannot recover semantics available only from another viewpoint on its own. MVEF must introduce the other views afterward; otherwise dropout remains regularization without creating any cross-view information pathway.

2. Multi-view Expert Fusion: build global context and return it to each view

MVEF receives enhanced features from all views, arranged along the view dimension as a \(V\times N\times C\) tensor. A linear layer followed by softmax produces \(V\times N\) importance weights rather than one fixed scalar for each entire image. Each view also has a dedicated expert consisting of two linear layers and a GELU activation to transform its representation. Unlike MoE designs mainly intended to expand capacity through category experts, these experts correspond to views and organize complementary observations. The text does not describe top-k sparse routing, so the MoE name alone does not justify assuming that only a small subset of experts is active. Using consistent notation, the position-wise weighted aggregation in Equation (7) can be expressed as:

\[ F_{\mathrm{fuse}}[j,:]=\sum_{i=1}^{V}g_{i,j}E_i(F_{\mathrm{enh},i})[j,:]. \]

Here \(g_{i,j}\) is the gate weight for view \(i\) at position \(j\), and \(E_i\) is its view expert; the fused representation retains an \(N\times C\) shape. This expression makes clear that global fusion aggregates transformed expert features rather than passing all view tokens unchanged to the decoder. However, matching patch indices in different cameras do not automatically identify the same physical surface, and the text provides no explicit camera calibration or geometric registration step. The operation should therefore be understood as learned semantic fusion rather than established pixel-level geometric correspondence.

Each view then generates queries from its own enhanced features and keys and values from the fused features, retrieving relevant global context through cross-attention. This matters because different viewpoints require different reconstructions: simply copying one global representation into every output would ignore those differences. A Transformer decoder using linear attention subsequently generates the reconstructed features for that view. The authors argue that this aggregate-and-return structure exploits structural evidence from other views under local incompleteness, but do not establish that every masked defect is identifiable elsewhere.

3. Area Adaptive Loss: allocate gradients to difficult product regions

Reconstruction difficulty is uneven: backgrounds are often easy, products are more complex, and missing product regions are harder still. AAL first computes cosine distances between original reference features and reconstructed features, then adjusts gradients according to local error relative to the mean error across the batch. To define the feature discrepancy, the standard cosine distance is given below; this is a metric definition, not a verbatim recovery of the damaged Equation (10) in the cache:

\[ D_{\mathrm{cos}}(a,b)=1-\frac{a^{\top}b}{\lVert a\rVert_2\lVert b\rVert_2}. \]

The ratio of local error to the batch mean is raised to \(\theta=3\), emphasizing relatively difficult patches. Region coefficients then assign 0.4 to missing regions, 0.2 to intact product foreground, and 0 to other regions. These two weighting factors jointly focus training: one tracks where the model currently struggles, while the other uses product and missing-region masks as task priors. Under the textual description of these coefficients, background receives zero region weight, so this should not be described as ordinary uniform full-image reconstruction loss.

Equation (11) in the cache contains an insufficiently explained cg operation, and Equation (12) also has damaged typesetting. It is therefore unclear whether the implementation is exactly equivalent to multiplying a scalar loss by a weight or whether gradients flow through the weight branch itself. This note preserves the authors' description of gradient modulation without inventing stop-gradient placement or an exact total-loss expression.

A Worked Example¶

Consider 5 views of the same normal component, with random rectangles hiding part of the product in one view; this illustrates the workflow rather than adding an experiment. During training, masked images enter the reconstruction branch, while the original normal images enter the shared-encoder reference branch. LAEE randomly drops patch-feature elements at a rate of 0.1 and aggregates the remaining within-view context. MVEF combines view-expert outputs at these positions, after which the current view's queries select relevant global information. Once the decoder restores features, AAL distinguishes missing product regions from visible product regions so that easy background reconstruction does not dominate training. On anomalous test samples, discrepancies between normal-pattern reconstructions and reference features support localization; this workflow does not imply guaranteed detection when a defect is invisible in every available view.

Loss & Training¶

The experiments use ViT-Base/14 with pretrained weights named DINO2-R in the text, citing the work on vision Transformers with registers. Images are resized to \(448\times448\) and center-cropped to \(392\times392\). Training uses StableAdamW with AMSGrad for 50,000 iterations, a learning rate of \(2\times10^{-3}\), and weight decay of \(10^{-4}\). Optimizer coefficients are \((0.9,0.999)\) and are distinct from AAL's region coefficients; all experiments use a single NVIDIA RTX 4090. The text does not fully specify encoder freezing, anomaly-map aggregation into image scores, or whether stochastic dropout remains active at deployment, so these are not supplied as implementation facts.

Key Experimental Results¶

Main Results¶

Real-IAD contains 30 product categories and 5 views per sample; its 99,721 normal images comprise 36,465 training images and 63,256 test images, alongside 51,329 anomalous test images. RIMAD is generated from the same source with a default 50% product-area masking ratio; it is not independently collected real-occlusion data. Metrics are reported as percentages: I-AUROC measures image-level anomaly discrimination, while P-AP measures pixel-level anomaly ranking with greater emphasis on minority anomalous pixels. The following selection comes from Table 1 on page 10 and Table 2 on page 11; the authors state that methods are reproduced under unified experimental settings.

Dataset	Method	I-AUROC	P-AP
RIMAD, 50% masking	Dinomaly	77.7	27.1
RIMAD, 50% masking	INP-Former	82.4	29.6
RIMAD, 50% masking	MVAD	79.3	27.4
RIMAD, 50% masking	IMMoE	84.7	33.1
Real-IAD, complete views	Dinomaly	89.3	42.8
Real-IAD, complete views	INP-Former	89.1	43.7
Real-IAD, complete views	MVAD	86.6	30.3
Real-IAD, complete views	IMMoE	89.7	44.3

On RIMAD, the absolute gains over INP-Former are 2.3 I-AUROC percentage points and 3.5 P-AP percentage points. The abstract's 2.8% and 11.8% are relative improvements obtained by dividing those differences by the baseline, not gains of 2.8 / 11.8 percentage points. Average leadership does not imply leadership in every category: for Bottle Cap in Table 1, IMMoE scores 86.7 / 34.0 versus MambaAD's 92.4 / 37.1.

Ablation Study¶

The following results are from Table 3 on page 12 under the default RIMAD setting; this is a sequential component-addition study rather than a full factorial analysis.

LAEE	MVEF	AAL	I-AUROC	P-AP
No	No	No	76.9	26.2
Yes	No	No	81.3	29.1
Yes	Yes	No	84.0	32.3
Yes	Yes	Yes	84.7	33.1

Adding LAEE first improves I-AUROC by 4.4 percentage points; adding MVEF on top improves it by another 2.7 percentage points. AAL then adds 0.7 I-AUROC percentage points and 0.8 P-AP percentage points, supporting its benefit without isolating every interaction between components. Table 4 on page 13 analyzes masking ratios as follows; each cell reports I-AUROC / P-AP.

Product-area masking ratio	Dinomaly	INP-Former	IMMoE
0%	89.3 / 42.8	89.1 / 43.7	89.7 / 44.3
25%	76.5 / 21.6	77.1 / 22.1	77.9 / 20.0
50%	77.7 / 27.1	82.4 / 29.6	84.7 / 33.1
75%	84.4 / 39.8	83.7 / 34.1	87.3 / 42.0

Key Findings¶

More masking does not produce a monotonic decline. The authors attribute recovery at high masking ratios to greater reliance on cross-view semantics, but provide no separate causal ablation establishing that explanation.
At 25% masking, IMMoE's P-AP is 20.0, below both baselines, so its advantage depends on the masking setting rather than holding at every ratio.
Table 5 on page 13 reports 84.7 / 33.1 with dropout 0.1 and 84.2 / 33.4 with dropout 0.2; the text's claim that 0.1 is optimal holds only for I-AUROC.
Increasing dropout to 0.3 reduces performance to 76.9 / 26.8, showing that excessive feature removal destroys useful information in already incomplete inputs.

Highlights & Insights¶

View fusion operates inside reconstruction rather than merely averaging final anomaly scores. Other cameras can therefore change the predicted normal features of the current view.
Per-view cross-attention after expert aggregation preserves viewpoint-specific requirements. The transferable idea is shared context followed by view-specific retrieval, not simply adding more experts.
Regularization and region weighting address input artifacts and the distribution of training difficulty, respectively. Sequential ablation supports their complementarity with fusion, not the claim that either module alone solves occlusion.

Limitations & Future Work¶

Evaluation-input boundary: Section 4.1 still compares original-image reference features with reconstructed features at test time, without clearly explaining reference availability under real occlusion or pixel-level evaluation treatment of masked regions.
Synthetic-occlusion boundary: Small random rectangles differ from real occluders, reflections, and camera failures. Independent real-occlusion data should distinguish visible defects, hidden defects, and pure occlusion artifacts.
Mechanism and reproducibility boundary: Gate normalization axes, loss-gradient implementation, and inference-score aggregation are insufficiently specified; damaged cached equations further constrain exact reproduction, and this note has not inspected the code.
Evidence strength: The main text provides no multi-seed error bars, parameter counts, or throughput comparison. Small gains on complete data and computational-efficiency claims need statistical and cost evidence.

Vs Dinomaly / INP-Former: IMMoE continues pretrained feature reconstruction, primarily adding cross-view feedback and missing-region adaptation rather than training a classifier with new defect labels.
Vs MVAD: MVAD selects correlated cross-view regions through window attention, whereas IMMoE aggregates view experts and returns the result to each view; winning on default RIMAD does not establish stronger geometric alignment.
Vs MoEAD: MoEAD emphasizes category-related experts and parameter efficiency, while IMMoE ties expert specialization to viewpoints. The expert-selection problems differ, so expert counts alone cannot establish superiority.
Future direction: Compare explicit visibility gates with learned gates and report masked-region and visible-region metrics separately to distinguish cross-view recovery from adaptation to synthetic missing-data patterns; this is a reader proposal, not a reported experiment.

Rating¶

Novelty: 4/5, a task-focused combination of view experts, feature regularization, and region weighting for local incompleteness.
Experimental Thoroughness: 3/5, two data settings and component, masking, and dropout analyses, but no real-occlusion or uncertainty evaluation.
Writing Quality: 3/5, a clear main pipeline with gaps in test-reference availability, loss details, and claims of optimality.
Value: 4/5, useful ideas for missing-data robustness in multi-view industrial inspection, subject to clarifying reference-image requirements before deployment.