A Comprehensive Analysis about Unsupervised Outlier Detection for Images

Conference: ECCV 2026
Paper: Official page (5320) · Paper PDF
Code: https://github.com/zhliu-uod/VUOD
Area: Anomaly Detection / Image Anomaly Detection
Keywords: Unsupervised outlier detection, local separability, global separability, pseudo-labeling, contrastive learning

Title provenance: the heading follows the official acceptance record; the linked PDF is titled VUOD: A Versatile Unsupervised Outlier Detection Framework for Natural, Industrial, Medical Images and Beyond. The local official record and PDF agree on all six authors, the abstract content, and the PDF address, indicating a title-version difference. This note explains the VUOD method in that PDF.

TL;DR

VUOD obtains reliable inlier/outlier pseudo-labels through fine-grained clustering and distance-ranking agreement, learns a lightweight feature mapping, and applies OCSVM to the improved representation, raising aggregated CIFAR-10 AUROC from FlexUOD's 0.942 to 0.972 while extending the same framework to industrial and medical images.

Background & Motivation

Image outlier detection takes an unlabeled collection and ranks images by how anomalous they appear, rather than assigning ordinary semantic class labels. A common approach extracts features with a naturally pretrained ResNet or CLIP encoder and then applies a distance-based, density-based, or learned scoring rule. This can work for semantic mismatches, such as an unrelated object appearing within one category, but subtle manufacturing defects or medical abnormalities may not be prominent in the same representation. When inlier and outlier scores already overlap substantially, a more sophisticated threshold cannot repair missing discriminative information.

The paper therefore targets feature separability rather than designing a separate detector for each application. However, adapting a representation without labels creates a circular problem: unreliable initial predictions can become unreliable supervision for the next model. VUOD addresses this by asking whether local groups can first be separated, and whether their behavior relative to the global distribution can identify trustworthy supervision. Its unsupervised protocol permits contamination in the target collection and adapts to the same collection that it scores; it does not require a previously curated, defect-free training set.

This makes the framework relevant to batch screening, but different from the usual industrial protocol of clean training data followed by independent test images. Medical imaging is one evaluation domain, not the sole task, and the output remains whole-image outlier ranking rather than lesion diagnosis or localization. Core idea: use local clustering and global ranking agreement to select high-confidence pseudo-labels, then reshape the feature space with contrastive learning before applying an outlier scoring head.

Method

Overall Architecture

The input is an unlabeled image collection containing an unknown proportion of outliers, and the output is one outlier score per image. A pretrained encoder supplies whole-image features; VUOD then performs local separability clustering, global separability selection, and contrastive feature enhancement before OCSVM scoring. The described trainable enhancement module is a two-layer MLP operating on extracted features, not an end-to-end fine-tuning procedure for the image encoder.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Unlabeled target images<br/>Pretrained whole-image features"] --> B["Local separability clustering<br/>Fine-grained AP clusters"]
    B --> C["Global separability selection<br/>Refined center and pseudo-labels"]
    C --> D["Contrastive feature enhancement<br/>Two-layer MLP"]
    D --> E["OCSVM<br/>Outlier scores for all images"]

Excluding an uncertain sample from pseudo-label supervision does not remove it from the final evaluation: all input images are eventually scored. Actual filtering for a downstream task additionally requires converting continuous scores into binary decisions, which AUROC itself does not do. For classification and reconstruction, the paper estimates the outlier proportion with FlexUOD and uses the corresponding score quantile as the rejection threshold. That final threshold is distinct from the two thresholds used to select pseudo-labeled clusters.

Key Designs

1. Local separability clustering: find relatively pure groups before labeling individual samples

Affinity Propagation (AP) forms fine-grained clusters using negative squared Euclidean feature distance as similarity. Its alternating responsibility and availability messages select representative samples: responsibility compares a candidate representative against alternatives, while availability aggregates support from other samples. The purpose is not to classify anomalies immediately, but to group locally similar points without requiring a predefined number of clusters. The underlying assumption is that outliers are often sparse, mutually varied, and insufficiently similar to normal examples, allowing some to form small or singleton clusters.

The resulting groups can still be predominantly normal, predominantly anomalous, or mixed. Clustering consequently changes the unit on which supervision is judged, rather than solving the detection problem by itself. Compared with k-Means, AP avoids choosing \(k\) in advance; the paper also observes that k-Means can absorb rare outliers into nearby normal clusters. Nevertheless, AP still assigns points to clusters and should not be described as producing an explicit rejection label. A small cluster is not automatically anomalous, and the usefulness of this stage still depends on local structure surviving in the original representation.

2. Global separability selection: correct a contaminated reference center through ranking agreement

The initial global reference scores each feature by its distance from the mean of the entire target collection. If normal data dominate, that mean can approximate the normal center, making more distant samples plausible outliers. The paper's high-dimensional shell discussion motivates distance concentration, but does not establish universal separation for arbitrary multimodal distributions. As contamination increases, outliers can shift the overall mean and weaken this initial ranking.

For each AP cluster, VUOD computes another ranking using every target sample's distance from that cluster's center. The comparison is therefore between two rankings of the same complete collection, not between cluster sizes, within-cluster distances, or merely the two centers. A cluster compatible with the main normal distribution is expected to produce a ranking closer to the global reference; an incompatible cluster may produce a different ordering. Equation (5) defines the ranking-agreement measure as the absolute Spearman correlation of the rank-index sequences:

\[ \phi_{t,\mathrm{base}}= \left|\operatorname{Spearman}\left( \operatorname{rank\text{-}index}(\{f_t(x_i)\}_{i=1}^{n}), \operatorname{rank\text{-}index}(\{f_{\mathrm{base}}(x_i)\}_{i=1}^{n}) \right)\right|. \]

Here, \(f_t\) scores distance from cluster \(t\)'s center, whereas \(f_{\mathrm{base}}\) scores distance from the full-collection mean. The initial agreement values are min-max normalized, and clusters below 0.9 are excluded when recomputing a refined reference center. This exclusion only concerns reference estimation; their samples remain eligible for final scoring. Each cluster-induced ranking is then compared against the refined global ranking. Clusters above \(\epsilon^+=0.9\) supply inlier pseudo-labels, while those below \(\epsilon^-=0.5\) supply outlier pseudo-labels. The intermediate region supplies neither type of supervision, avoiding forced labels for mixed clusters or unusually varied normal groups. These thresholds govern supervision purity and coverage without requiring the true contamination ratio, but their success remains conditional on the distributional assumptions.

3. Contrastive feature enhancement: teach a lightweight mapping using only trusted supervision

The method samples positive pairs from the predicted inlier set and negative pairs across the predicted inlier and outlier sets. Pairs are resampled each epoch instead of materializing every possible pair, limiting pair-related memory use and improving training stability. A two-layer MLP with ReLU produces \(\ell_2\)-normalized embeddings. The contrastive objective pulls normal pairs together and pushes normal–outlier pairs beyond a margin; similar pairs use \(y=0\), dissimilar pairs use \(y=1\), the margin is 1.0, and Adam updates the mapping parameters.

Importantly, the procedure does not require all outliers to collapse into a common anomaly class. They may arise from unrelated distributions, so tightening normal structure and separating anomalies from normal examples better matches the task than symmetric clustering of two classes. After training, every original feature passes through the mapping, including samples excluded from pseudo-label supervision. OCSVM then models and scores the enhanced features, allowing uncertain samples to be judged in the improved representation rather than permanently classified during the first selection stage. The paper also evaluates Isolation Forest and LOF, with its figures indicating that enhancement benefits these scoring heads too. OCSVM is the default classic detector, not a newly proposed component.

Loss & Training

“Unsupervised” refers to the absence of labels for the target detection task, not the absence of external data used to pretrain the encoder. Pseudo-label generation, MLP fitting, and scoring operate on the same target collection, making this a transductive setting. Natural-image experiments use ResNet-50 and CLIP; industrial and medical experiments use ResNet-18 and Wide ResNet-101. For the latter domains, the original training and test sets are combined, including both normal and anomalous images, rather than preserving a normal-only training split. Some equations are damaged in the local text extraction, so this note retains the verifiable ranking definition and explains the contrastive objective in words instead of reconstructing an unverified full loss equation. The available main text does not specify such details as learning rate, hidden width, or training epoch count; common defaults should not be substituted as reported settings.

Key Experimental Results

Main Results

The following selection comes from Tables 1 and 2; the metric is whole-image AUROC, higher is better, and changes are absolute AUROC differences rather than relative percentages. Natural-image targets contain all inliers from one class and randomly sampled outliers from other classes; Table 1 aggregates experiments across two encoders and six contamination ratios spanning 0.01–0.5. Industrial and medical results use the combined original training and test sets; the selected rows fix ResNet-18 to avoid cross-backbone comparisons.

Dataset and setting	Source	Comparator	Comparator AUROC ↑	VUOD AUROC ↑	Absolute change
CIFAR-10, natural-image aggregate	Table 1	FlexUOD	0.942	0.972	+0.030
MIT-Places, natural-image aggregate	Table 1	FlexUOD	0.928	0.971	+0.043
MVTec-AD, ResNet-18	Table 2	LVAD	0.779	0.911	+0.132
MPDD, ResNet-18	Table 2	RSRAE	0.618	0.906	+0.288
RESC, ResNet-18	Table 2	FlexUOD	0.948	0.901	−0.047

Comparators are strong baselines from the corresponding table columns, not evidence that VUOD wins on every dataset and encoder. RESC with ResNet-18 is an explicit exception: cross-domain usefulness does not mean unconditional superiority. The PaDiM, FRE, and EfficientAD comparisons in Table 3 likewise train and test on the same contaminated target collection, so their numbers should not be read as standard industrial-protocol results.

Ablation Study

Table 6 fixes the OCSVM scoring head and compares raw against enhanced features, averaging over the two encoders used in each domain and retaining the main experiment's target construction. This isolates the overall feature-enhancement contribution, not the individual effects of removing AP, mean refinement, or contrastive learning.

Dataset	Raw-feature AUROC ↑	Enhanced-feature AUROC ↑	Absolute change	Source
CIFAR-10	0.824	0.972	+0.148	Table 6
MVTec-AD	0.754	0.921	+0.167	Table 6
MPDD	0.575	0.916	+0.341	Table 6
LiverCT	0.597	0.825	+0.228	Table 6

MPDD's change from 0.575 to 0.916 corresponds to the paper's approximately 59.3% relative improvement; its absolute AUROC increase is 0.341, not 59.3 percentage points. Table 7 additionally studies pseudo-label thresholds on CIFAR-10: the default \((0.9,0.5)\) obtains 0.972, compared with 0.971 for \((0.9,0.6)\) and 0.970 for \((0.9,0.4)\). This supports local stability near the default, but one dataset does not establish insensitivity across every domain or every threshold choice.

Key Findings

• The main benefit is representation adaptation rather than merely swapping the scoring head. Table 6's fixed-OCSVM comparison makes this distinction particularly clear.
• Whole-image industrial detection remains uneven: Table 2 reports 0.832 on MVTec-LOCO with Wide ResNet-101, contradicting an unqualified reading of the prose claim that industrial results exceed 0.910.
• Downstream classification gains are modest: Table 8 reports Imagenette/ResNet-18 Top-1 accuracy increasing from 90.81% to 91.45%, and Top-5 from 99.11% to 99.39%. These are classification gains after training-set filtering, not outlier-detection AUROC.
• AP's speed advantage is conditional: Table 4 favors AP when k-Means uses the cluster count obtained by AP, but k-Means with a smaller \(k\) is faster. The result does not establish universal AP efficiency.

Highlights & Insights

• Pseudo-labeling need not force a decision for every sample. Group-level confidence allows uncertain images to wait for a better representation before receiving final scores.
• The distinguishing signal is agreement between whole-collection rankings induced by different reference centers. This gives local grouping a concrete connection to global anomaly ordering rather than relying solely on group size.
• Enhancement is decoupled from scoring. Retaining pretrained encoders and classic detectors concentrates new learning in a small mapping and makes its contribution easier to test.

Limitations & Future Work

• Author-stated limitation: the framework targets whole-image ranking and may be less effective for patch- or pixel-level anomaly segmentation. Medical AUROC should not be interpreted as lesion-localization capability.
• Reader assessment: dominant normal structure and locally separable, relatively sparse anomalies are important assumptions. Multimodal normal data or coherent anomaly clusters may undermine center estimation and ranking agreement.
• Reader assessment: AP involves pairwise sample relationships, so favorable finite-scale timing does not remove large-collection memory and computation concerns. Online arrival and unseen-sample settings need separate evaluation.
• Evidence boundary: the local source includes the main paper through conclusions and references but not the separately mentioned appendix. Unavailable clustering visualizations and reconstruction input details are therefore not reconstructed here.
• Reproducibility boundary: the main text omits some training hyperparameters and uncertainty intervals for these aggregate results. All numbers above are author-reported, not independently reproduced.

Related Work & Insights

• Compared with Multi-T / FlexUOD: VUOD moves emphasis toward reliable supervision and representation enhancement rather than only finding boundaries in existing scores. Its downstream binary filtering nevertheless still uses FlexUOD's contamination estimate.
• Compared with OCSVM: OCSVM remains the classic scoring head, not a method VUOD replaces. The central comparison is the same head operating on raw versus enhanced features.
• Compared with specialized industrial detectors: VUOD prioritizes cross-domain whole-image screening with contaminated transductive targets. This comparison does not erase specialized methods' spatial modeling capabilities or their conventional clean-training protocol.

Rating

• Novelty: 3/5. The contribution combines established clustering, ranking agreement, and contrastive learning, with its main distinction in supervision construction and cross-domain framing.
• Experimental Thoroughness: 4/5. Three domains and fourteen datasets provide breadth, with feature and threshold analyses, but individual component isolation and statistical uncertainty remain limited.
• Writing Quality: 3/5. The mechanism is understandable, although some broad claims conflict with table entries and implementation details require additional documentation or code.
• Value: 4/5. A useful reusable baseline for batch image-outlier screening, provided its transductive protocol and whole-image scope remain explicit.