Distribution-Aware Feature Selection for Post-hoc Out-of-Distribution Detection

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/remic-othr/mfs-ood
Area: Medical Imaging
Keywords: Out-of-Distribution Detection, Feature Selection, Wasserstein Distance, Cross-Domain Mixup, Post-hoc Detection

TL;DR

Addressing the issue that post-hoc feature-based out-of-distribution (OOD) detectors treat all high-dimensional feature dimensions uniformly, this paper introduces a distribution-aware marginal feature selection strategy (MFS) via the Wasserstein-1 distance using cross-domain mixup proxy data, consistently enhancing detection performance while drastically slashing inference latency without model retraining.

Background & Motivation

Deep learning systems are increasingly deployed in high-stakes environments such as autonomous driving and clinical healthcare. However, conventional deep classifiers operate under the closed-world assumption that test inputs share identical distributions with training data. When encountering out-of-distribution (OOD) inputs, models frequently yield overconfident erroneous predictions, posing critical safety risks. To equip pre-trained models with the ability to detect unknown inputs, post-hoc feature-based methods have emerged as a dominant paradigm because they avoid costly retraining. Representative detectors such as Mahalanobis Distance (MDS) and deep k-Nearest Neighbors (kNN) extract global representations from penultimate layers. Yet, these dense features often contain numerous dimensions that are either irrelevant to distinguishing in-distribution (ID) from OOD or act as pure noise, severely distorting Euclidean distance and covariance estimates.

Prior feature space adjustment approaches either apply unsupervised subspace projections like Principal Component Analysis (PCA)—disregarding true distributional differences between ID and OOD—or rely on heuristic activation clipping (such as ReAct) and channel masking based on ID intra-class variance, failing to directly measure distributional discrepancy. More fundamentally, in real-world deployment, target OOD samples are strictly unavailable prior to testing, creating a persistent challenge for supervising optimal feature selection.

This work builds on the key empirical finding that, driven by batch normalization decorrelation and ReLU-induced activation sparsity, OOD-discriminative signals in deep representations naturally concentrate along coordinate axes (axis-aligned) and transfer across diverse distribution shifts. Core idea: construct proxy-OOD distributions using cross-domain mixup, compute the one-dimensional Wasserstein-1 distance to efficiently quantify marginal ID/OOD separability per feature dimension, and retain the top-\(k\) most discriminative dimensions to seamlessly boost existing post-hoc detectors.

Method

Overall Architecture

The proposed Marginal Feature Selection (MFS) is an off-the-shelf post-hoc preprocessing framework that requires neither backbone parameter updates nor modifications to detector scoring functions. For a given pre-trained classifier, the pipeline first synthesizes proxy-OOD data by convexly mixing ID inputs with semantically disjoint cross-domain samples. Features are then extracted via the backbone, and one-dimensional empirical feature distributions are established for each dimension. The closed-form Wasserstein-1 distance is calculated across all dimensions to rank their discriminative power, and the retained feature fraction \(\alpha\) is tuned via validation AUROC with an identity fallback. During testing, detectors like kNN or MDS evaluate only the selected subset of dimensions.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["ID Data + Cross-Domain Pool"] --> B["Cross-Domain Mixup Proxy OOD Generation<br/>Convex combination with random ratios"]
    B --> C["Pre-trained Backbone Feature Extraction<br/>Spatial average pooling yields embedding sets"]
    C --> D["Marginal Wasserstein-1 Feature Selection<br/>Integral over empirical CDF discrepancy per dimension"]
    D --> E["Adaptive Fraction Selection & Full-Space Fallback<br/>Grid search on validation set, fallback to full space"]
    E --> F["Downstream Post-hoc Detector Inference<br/>kNN / MDS / MDS++ scoring on reduced subspace"]

Key Designs

1. Cross-Domain Mixup Proxy OOD Generation: Simulating Distribution Shifts Without Real OOD Data

During the post-hoc stage, target deployment shifts are unknown and cannot be collected in advance. Standard mixup performed solely between ID samples yields synthetic features that remain tightly bound to the ID manifold, failing to induce significant separation. On the other hand, gradient-based adversarial perturbations (e.g., FGSM) can be suppressed by models trained with adversarial robustness. To overcome this limitation, MFS introduces an auxiliary pool of cross-domain data that shares zero semantic overlap with ID classes (e.g., natural images for medical models, and histopathology images for natural image classifiers). In each batch, proxy-OOD samples are generated via convex combinations:

\[x_{\text{OOD}, i} = r_i x_i + (1 - r_i) m_j\]

where \(x_i\) is an ID sample, \(m_j\) is a randomly drawn cross-domain sample, and \(r_i\) is sampled uniformly from \(\{0.25, 0.5, 0.75\}\). This design incurs no manual labeling overhead and effectively pulls feature representations away from the ID manifold across multiple shift severities, providing a stable baseline for evaluating feature discriminability.

2. Marginal Wasserstein-1 Feature Selection: Efficient and Parameter-Free Discrepancy Quantification

Evaluating joint multi-dimensional feature subsets is computationally intractable due to combinatorial explosion. Informed by orthogonal rotation experiments confirming axis-aligned discriminability, MFS decouples the problem into \(D\) independent 1D statistical evaluations. Features are first normalized using ID-based Min-Max scaling. For dimension \(d\), given \(N\) ID feature values \(\{a_1^{(d)}, \dots, a_N^{(d)}\}\) and \(M\) proxy-OOD values \(\{b_1^{(d)}, \dots, b_M^{(d)}\}\) with empirical cumulative distribution functions \(F_\rho^{(d)}\) and \(F_\nu^{(d)}\), the Wasserstein-1 distance is computed analytically:

\[W_1^{(d)} = \int_{-\infty}^{\infty} \left| F_\rho^{(d)}(t) - F_\nu^{(d)}(t) \right| dt\]

In practice, this is evaluated by sorting the ID and OOD scalar values independently, running in \(\mathcal{O}(D(N \log N + M \log M))\) time. Unlike MMD which requires quadratic computation or KL divergence which is unstable under disjoint supports, Wasserstein-1 distance is completely parameter-free and captures shifts in mean, spread, and multimodality. The top-\(k\) dimensions (\(k = \lfloor \alpha D \rfloor\)) with the highest distances form the optimal subspace \(\mathcal{I}_k\).

3. Adaptive Fraction Selection & Full-Space Fallback: Balancing Dimensional Compression and Geometry

Because feature manifolds differ significantly across architectures and data modalities, fixed dimension budgets can lead to underfitting or noise inclusion. MFS tunes the retained fraction \(\alpha \in \{0.1, 0.2, \dots, 1.0\}\) via grid search on held-out validation AUROC. Crucially, the selection mechanism incorporates a protective fallback: if severe feature anisotropy or variance dominance prevents any sub-dimensional space from outperforming the full representation, the search selects \(\alpha^* = 1.0\), exactly recovering the original base detector. In all other cases, MFS removes 30% to 99% of uninformative dimensions, mitigating the curse of dimensionality and distance distortion.

A Worked Example

Consider the kNN detector on the MIDOG histopathology mitotic figure benchmark: 1. Feature Extraction: A pre-trained ResNet outputs penultimate features of dimension \(D = 2048\). 2. Proxy Generation & Ranking: Pathology images are interpolated with CIFAR-10 samples at ratios \(r \in \{0.25, 0.5, 0.75\}\) to yield 1000 proxy-OOD samples. The 1D Wasserstein-1 distance is calculated for each of the 2048 dimensions and sorted in descending order. 3. Validation Tuning: Evaluating \(\alpha\) across the validation set reveals that retaining only \(\alpha = 0.01\) (a mere 20 dimensions) achieves peak AUROC. 4. Inference Scoring: For test inputs, only the 20 chosen dimensions are sliced and queried against ID representations. AUROC improves from 70.64 to 76.66 while FPR@95 drops from 67.88 to 60.68, and kNN inference latency is reduced by approximately 97%.

Key Experimental Results

Main Results

Experiments across 3 medical imaging benchmarks (MIDOG, PhaKIR, OASIS-3; 14 datasets) and 2 natural image benchmarks (CIFAR-10, ImageNet-1k; 16 datasets) evaluate average AUROC and FPR@95 across cs-ID, near-OOD, and far-OOD settings:

Method	Category	MIDOG (AUROC/FPR)	PhaKIR (AUROC/FPR)	OASIS-3 (AUROC/FPR)	CIFAR-10 (AUROC/FPR)	ImageNet1k (AUROC/FPR)	Average (AUROC↑ / FPR95↓)
kNN [83]	Distance Baseline	70.64 / 67.88	42.41 / 94.93	98.96 / 5.28	88.02 / 44.15	79.10 / 58.80	75.83 / 54.21
kNN-MFS (Ours)	Marginal Selection	76.66 / 60.68	42.41 / 94.93	99.55 / 2.82	88.41 / 42.93	81.37 / 55.64	77.68 / 51.40
MDS [49]	Density Baseline	70.91 / 69.73	61.33 / 79.65	97.49 / 7.86	84.54 / 52.35	64.60 / 76.11	75.77 / 57.14
MDS-MFS (Ours)	Marginal Selection	76.16 / 58.77	62.74 / 78.91	98.52 / 6.10	84.79 / 51.93	68.78 / 70.61	78.20 / 53.27
MDS++ [64]	Normalized SOTA	72.16 / 65.74	57.64 / 89.76	99.85 / 0.71	86.44 / 52.20	80.18 / 55.16	79.25 / 52.71
MDS++-MFS (Ours)	Marginal Selection	76.78 / 62.35	57.23 / 91.29	99.84 / 0.76	86.82 / 50.56	81.38 / 53.46	80.41 / 51.68
Partial MDS [42]	Subspace DR	71.88 / 67.58	62.18 / 79.49	97.43 / 7.92	82.66 / 56.56	64.01 / 76.74	75.63 / 57.66
Residual [88]	Subspace Projection	72.79 / 66.23	63.81 / 79.08	97.95 / 6.78	82.74 / 56.70	58.80 / 84.93	75.22 / 58.74
ViM [88]	Hybrid Subspace	72.16 / 66.24	50.27 / 94.77	94.47 / 14.52	86.69 / 50.39	80.37 / 54.73	76.79 / 56.13
ASH [19]	Activation Shaping	67.78 / 80.59	66.88 / 71.73	79.57 / 40.70	79.50 / 81.73	83.82 / 50.76	75.51 / 65.10
DICE [82]	Weight Pruning	62.59 / 87.83	43.66 / 91.53	23.95 / 94.32	81.44 / 72.99	79.86 / 62.26	58.30 / 81.79
DDCS [96]	Channel Selection	69.97 / 76.45	49.27 / 81.63	63.81 / 62.92	81.03 / 73.30	80.91 / 57.64	69.00 / 70.39
ActSub [101]	Residual Subspace	69.94 / 70.83	52.15 / 81.98	91.19 / 26.98	80.70 / 64.27	86.12 / 44.76	76.02 / 57.76

Ablation Study

The ablation evaluates different feature-wise discrepancy metrics and proxy data generation strategies across medical and natural imaging benchmarks:

Aspect	Variant	kNN-MFS (Med / Nat)	MDS-MFS (Med / Nat)	MDS++-MFS (Med / Nat)	Description & Characteristics
Discrepancy Metric	Jensen-Shannon (JS)	71.96 / 84.35	78.06 / 75.90	77.73 / 83.77	Symmetric and bounded, but slightly less sensitive to tail differences
	MMD (Linear Kernel)	72.86 / 84.72	79.10 / 76.80	77.16 / 84.02	Mean difference only, low compute but neglects higher-order moments
	MMD (RBF Kernel)	72.90 / 84.73	79.11 / 76.72	77.42 / 83.96	Captures non-linear shifts, requires heuristic bandwidth selection
	Wasserstein-1 (Ours)	72.87 / 84.89	79.14 / 76.79	77.95 / 84.10	Parameter-free, defined for disjoint supports, \(\mathcal{O}(N \log N)\) sorting
Proxy Generation	Adversarial (AD, FGSM)	72.79 / 84.84	76.42 / 77.19	76.85 / 84.01	Requires gradients, suppressed by adversarially robust models
	Mixup (MU, Ours)	72.87 / 84.89	79.14 / 76.79	77.95 / 84.10	Smooth multi-severity shifts without semantic class contamination

Key Findings

• Simultaneous Accuracy Gain and Latency Reduction: MFS improves average AUROC for kNN, MDS, and MDS++ by 1.85, 2.43, and 1.16 percentage points respectively, while reducing FPR@95 by 2.81, 3.87, and 1.03 percentage points. Concurrently, inference latency drops by 58.89% (kNN), 27.03% (MDS), and 42.76% (MDS++). On MIDOG, retaining just 1% of features speeds up kNN by 97% while improving AUROC by 5.74 points.
• PhaKIR Failure Mode & Whitening Rectification: On the laparoscopic video benchmark PhaKIR, kNN-MFS fell back to \(\alpha=1.0\) due to severe feature anisotropy, where dominant high-variance dimensions masked discriminative low-variance signals. Equalizing variances via feature whitening allowed kNN-MFS to identify a compact subset, yielding a remarkable ~14 point AUROC improvement.
• Generalization Across Architectures: Beyond CNNs (ResNet, MobileNetV2, ShuffleNetV2, VGG), MFS consistently improves Swin Transformer performance on CIFAR-10 (+0.21 kNN, +0.25 MDS, +1.71 MDS++) and even enhances latent VAE representations from Stable Diffusion 2.1 despite having only 4 channels.

Highlights & Insights

• Axis-Aligned Structure of OOD Information: Deep representations concentrate discriminative signals along individual coordinate axes due to BN and ReLU activations, rendering 1D marginal evaluation highly effective without complex joint projections.
• Zero Real-OOD Supervision via Cross-Domain Proxies: Simple cross-domain mixup effectively simulates realistic distribution shifts, and the selected features generalize robustly across unseen OOD benchmarks.
• Decoupled Accuracy and Compute Benefits: Unlike existing feature transformations that add runtime projection overhead, MFS operates as a simple index slice during inference, simultaneously enhancing detection accuracy and cutting latency.

Limitations & Future Work

• Author-Admitted Limitations: In strongly anisotropic feature spaces where high-variance directions dominate distance metrics, marginal feature selection without variance equalization may fail to identify informative low-variance components.
• Identified Limitations: Cross-domain mixup assumes access to an external dataset with disjoint semantics; in massive multi-label or fine-grained classification contexts, ensuring complete semantic isolation may require manual verification.
• Improvement Directions: Integrating lightweight adaptive feature whitening prior to selection, or extending marginal scoring to sparse graphical cluster selection.

Related Work & Insights

• vs DICE / ASH / DDCS (Marginal Selection & Shaping): DICE relies on heuristic weight pruning and DDCS uses ID-only variance/similarity scores that ignore true shift behavior. ASH applies coarse activation masking. MFS leverages the principled Wasserstein-1 distance to explicitly capture empirical distribution divergence between ID and proxy OOD.
• vs PCA-MDS / ViM / ActSub (Subspace Projections): ViM and ActSub compute full-matrix eigenvectors and project representations during inference, introducing non-trivial matrix multiplication overhead. MFS preserves original axes and applies zero-cost index slicing, achieving superior runtime efficiency.

Rating

• Novelty: ⭐⭐⭐⭐☆ First to apply Wasserstein-1 distance to post-hoc marginal feature selection and validate axis-aligned transferability.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ 5 major benchmarks, 30 distinct datasets, spanning medical and natural domains across CNNs, Transformers, and VAEs.
• Writing Quality: ⭐⭐⭐⭐⭐ Rigorous narrative with deep empirical validation, clear failure analysis, and sound mathematical grounding.
• Value: ⭐⭐⭐⭐⭐ Retraining-free, computationally efficient, delivering simultaneous boosts in AUROC and inference throughput.

Related Papers

• [ECCV 2026] Ranked Activation Shift for Post-hoc Out-of-Distribution Detection
• [ECCV 2026] Exploiting Local Flatness for Efficient Out-of-Distribution Detection
• [CVPR 2026] RankOOD: Class Ranking-based Out-of-Distribution Detection
• [CVPR 2026] Enhancing Out-of-Distribution Detection with Extended Logit Normalization
• [CVPR 2026] Sparsity as a Key: Unlocking New Insights from Latent Structures for Out-of-Distribution Detection