Unified Multi-Layer Subspace Modeling for Cross-Domain OOD Detection

Conference: ECCV 2026
Paper: ECCV Official Poster
Cache: ../paper_cache/ECCV2026/eccv-3729.txt
Area: Medical Imaging / Transfer Learning
Keywords: Out-of-Distribution Detection, Multi-Layer Feature Fusion, Subspace Modeling, Mahalanobis Distance, Cross-Domain Generalization

TL;DR

PRISM is a tuning-free post-hoc OOD detection method that discards brittle per-layer score aggregation and OOD-calibrated weighting in favor of a "fuse-then-model" principle, constructing a unified shallow-to-deep feature embedding whose ID principal subspace provides both projected class-conditional Mahalanobis distances and orthogonal residual energy scores, achieving a state-of-the-art 93.92% cross-domain average AUROC across natural, medical, and industrial vision benchmarks.

Background & Motivation

Deep neural networks demonstrate remarkable classification accuracy in closed-world setups, yet they consistently output overconfident and erroneous predictions when encountering out-of-distribution (OOD) inputs at deployment. In safety-critical domains such as medical imaging (e.g., endoscopic navigation or histopathological cancer screening) and automated industrial visual inspection (e.g., food contamination or mechanical defect detection), silent misclassifications can trigger severe diagnostic failures or operational hazards. Post-hoc OOD detection has emerged as an appealing solution because it operates directly on pretrained classifiers without retraining or degrading in-distribution (ID) accuracy. Conventional post-hoc detectors typically rely on maximum softmax probabilities, output logits, or penultimate-layer feature representations (such as Mahalanobis distance or k-nearest neighbor embeddings). However, real-world distribution shifts manifest across widely divergent semantic granularities—ranging from low-level sensor artifacts and texture differences to high-level semantic category changes—rendering fixed deep-layer representations insufficient because deep features are heavily specialized for ID classification and often discard informative lower-level cues.

The core tension lies in the fact that the most informative representation depth varies drastically across application domains and shift types, yet existing multi-layer detectors either depend on rigid manual layer selection or calibrate layer aggregation weights using held-out or synthetic OOD validation sets (e.g., logistic regression in MDSEns). Empirical evidence reveals that layer weights learned on one OOD validation set generalize poorly: optimal weights skew toward shallow and intermediate blocks on medical datasets (such as MIDOG and PhaKIR) but shift toward deep blocks on ImageNet-1k; even across different near-OOD splits within the same benchmark, the calibrated weights fluctuate wildly. Consequently, multi-layer aggregation strategies that rely on OOD calibration overfit to specific shift patterns and break down when transferred across domains.

This paper approaches the dilemma from a geometric perspective: rather than scoring each layer independently and aggregating the outputs with fragile weighting heuristics, one can unify multi-layer representations from shallow to deep into a single hierarchical feature space and model its underlying geometric manifold using ID data alone. Core idea: PRISM introduces a unified multi-layer subspace modeling framework that fuses shallow-to-deep pooled features into an ID principal subspace, multiplying the in-subspace projected class-conditional Mahalanobis distance by the off-subspace orthogonal residual energy without any balancing hyperparameter or OOD-dependent calibration.

Method

Overall Architecture

PRISM operates as an ID-only, architecture-agnostic post-hoc pipeline comprising three sequential stages: multi-layer feature construction and normalization, ID principal subspace approximation, and dual-signal anomaly scoring across projected and residual subspaces. Given a test image passed through a frozen backbone, feature maps from uniformly spaced architectural stages are extracted, spatially compressed via global pooling, normalized by their \(\ell_2\) norms, and concatenated into a unified hierarchical representation. A principal component analysis (PCA) basis pre-estimated strictly on ID training data decomposes this high-dimensional vector into an in-manifold projection coordinate and an orthogonal residual vector. Finally, the class-conditional Mahalanobis distance in the projected subspace and the residual energy outside the subspace are computed and multiplied into a single scalar OOD score.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Test Sample x"] --> B["Multi-Layer Feature Construction & Normalization<br/>Uniform stage sampling + Global pooling + L2 normalization"]
    B --> C["ID Principal Subspace Projection Decomposition<br/>PCA basis W splits feature into projection u(x) and residual r(x)"]
    C --> D["Projected Mahalanobis Distance<br/>In-subspace class-conditional distance sproj"]
    C --> E["Orthogonal Residual Energy<br/>Off-subspace unexplained manifold energy sres"]
    D --> F["Parameter-Free Multiplicative Fusion<br/>sPRISM = - sproj * sres"]
    E --> F
    F --> G["Decision Output<br/>Higher score indicates ID, lower indicates OOD"]

Key Designs

1. Multi-Layer Feature Construction and Normalization: Resolving spatial dependence and layer scale disparities To capture both low-level structural context and high-level categorical abstractions, PRISM extracts intermediate activations \(\{\phi^{(l)}(x)\}_{l=1}^L\) from \(L\) uniformly spaced stages of the classifier \(f\). For convolutional backbones with spatial dimensions \(H_l \times W_l\) and \(C_l\) channels, global average pooling (GAP) is applied; for vision transformers with \(T_l\) tokens of dimension \(C_l\), token averaging is performed. This pooling eliminates spatial grid dependencies, ensuring that all downstream scoring operations scale independently of image resolution. Because feature activation magnitudes differ substantially across network depths, raw concatenation would cause deeper or high-variance layers to dominate the geometry. PRISM applies an \(\ell_2\)-normalization to each pooled vector before concatenating them into a unified multi-layer embedding \(z(x) \in \mathbb{R}^D\) where \(D = \sum_{l=1}^L C_l\): $\(\tilde{\phi}^{(l)}(x) = \frac{\mathcal{P}(\phi^{(l)}(x))}{\|\mathcal{P}(\phi^{(l)}(x))\|_2}, \quad z(x) = \left[\tilde{\phi}^{(1)}(x)^\top, \dots, \tilde{\phi}^{(L)}(x)^\top\right]^\top\)$ This per-layer scaling balances shallow and deep representations geometrically, removing the need for manual or learned layer weighting coefficients.

2. ID Principal Subspace Modeling: Decomposing features into dominant modes and orthogonal residuals Neural representations of ID data naturally reside on a low-dimensional manifold embedded in high-dimensional feature space. Fitting class-conditional covariance models directly on the uncompressed concatenated vector \(z(x) \in \mathbb{R}^D\) would trigger numerical instability and noise overfitting. PRISM estimates the global ID training mean \(\mu = \frac{1}{N}\sum_{i=1}^N z(x_i)\) and derives a principal projection matrix \(W \in \mathbb{R}^{D \times d}\) using the top \(d\) eigenvectors of the ID empirical covariance matrix (with \(d=512\) held constant across all experiments). At inference time, any sample vector \(z(x)\) is orthogonally decomposed into its in-subspace projection \(u(x)\) and its orthogonal residual vector \(r(x)\): $\(u(x) = W^\top (z(x) - \mu), \quad r(x) = (I - WW^\top)(z(x) - \mu)\)$ Here, \(u(x)\) measures alignment with dominant training semantic directions, whereas \(r(x)\) isolates feature components orthogonal to the learned ID manifold.

3. Projected Mahalanobis Distance and Residual Energy: Dual geometric anomaly metrics without OOD tuning Distributional deviations manifest along two complementary geometric axes: some OOD samples share global ID support but violate class-conditional clustering (in-manifold semantic drift), while others introduce novel features that fall entirely outside the learned subspace (off-manifold anomalies). Within the \(d\)-dimensional PCA coordinate space, PRISM computes the minimum class-conditional Mahalanobis distance using class means \(\hat{\mu}_c^u\) and a shared pooled covariance matrix \(\hat{\Sigma}_u\): $\(s_{\text{proj}}(x) = \min_{c \in \mathcal{Y}} \left(u(x) - \hat{\mu}_c^u\right)^\top \hat{\Sigma}_u^{-1} \left(u(x) - \hat{\mu}_c^u\right)\)$ Simultaneously, leveraging the orthonormal property \(W^\top W = I\), the residual energy is computed directly via squared vector norms without explicit feature reconstruction: $\(s_{\text{res}}(x) = \|r(x)\|_2^2 = \|z(x) - \mu\|_2^2 - \|u(x)\|_2^2\)$ Both terms quantify non-negative deviation. PRISM fuses them into a single unified score via multiplication: $\(s_{\text{PRISM}}(x) = - s_{\text{proj}}(x) \cdot s_{\text{res}}(x)\)$ The negative product ensures higher values signify ID-likeness. Unlike additive combinations that require tuning linear weighting coefficients on validation OOD data, the multiplicative formulation is completely parameter-free and strongly penalizes anomalies appearing in either representation subspace.

Computational Complexity & Runtime Overhead

PRISM requires only three linear matrix operations during inference: projecting the \(D\)-dimensional fused vector into the \(d\)-dimensional subspace with complexity \(\mathcal{O}(BDd)\), evaluating Mahalanobis distances across \(K\) classes in the reduced subspace with \(\mathcal{O}(BdK)\), and computing residual energy with \(\mathcal{O}(BD)\), where \(B\) is the batch size. Because \(d \ll D\) and spatial resolution is eliminated at the pooling stage, the relative runtime overhead on an ImageNet-1k ResNet-50 benchmark is under 1.0% compared to baseline MSP or optimized MDS++ for batch sizes \(\ge 16\).

Key Experimental Results

Main Results

PRISM was evaluated across three large-scale benchmarks spanning natural images (OpenOOD), medical imaging (OpenMIBOOD), and industrial visual inspection (ICONIC-444) against 22 representative post-hoc baselines using official standard backbones (ResNet-18, ResNet-50, R(2+1)D). The table below summarizes the key comparisons (AUROC % from Table 1):

Method	OpenOOD (IN-1k)	OpenOOD Mean	MIBOOD (MIDOG)	MIBOOD (PhaKIR)	MIBOOD Mean	ICONIC-444 Mean	Overall Average
MSP [26]	77.94	81.49	64.23	39.99	57.82	73.06	70.79
MDS [38]	89.83	80.77	78.14	73.66	83.51	95.31	86.53
MDSEns* [38]	54.41	74.82	96.87	98.49	98.41	93.81	89.01
GRAM [58]	85.61	87.46	89.49	36.46	75.12	97.55	86.71
EBO [43]	41.68	70.06	66.54	32.94	55.10	67.26	64.14
ViM [71]	85.54	85.19	79.12	64.46	81.02	94.54	86.92
KNN [63]	85.64	86.00	76.17	51.12	75.52	92.47	84.66
MDS++ [50]	89.88	87.16	80.72	71.69	84.09	96.12	89.13
PRISM (Ours)	89.96	90.36	91.99	92.60	94.86	96.56	93.92

*Note: MDSEns relies on supervised logistic regression trained on held-out OOD data. While it performs well on medical benchmarks, its performance collapses to 54.41% on ImageNet-1k. PRISM operates in a strictly ID-only regime and achieves the highest overall average AUROC of 93.92%.

Ablation Study

Table 2 in the paper evaluates the individual contributions of the projected Mahalanobis distance \(s_{\text{proj}}\), the residual energy \(s_{\text{res}}\), and the sensitivity to PCA projection dimension \(d\) (AUROC ↑ / FPR95 ↓):

Configuration / Dimension	OpenOOD (IN-1k)	OpenMIBOOD (Medical)	ICONIC-444 (Industrial)	Cross-Domain Average (AUROC / FPR95)	Remarks
\(s_{\text{proj}}\) only	89.94 / 38.62	91.42 / 32.32	95.21 / 16.05	92.19 / 29.00	Dominates on natural and industrial tasks
\(s_{\text{res}}\) only	76.39 / 58.20	95.72 / 17.21	93.82 / 20.30	88.64 / 31.90	Dominates on medical histopathology/endoscopy
Full PRISM (\(d=512\))	89.88 / 35.83	94.86 / 21.10	95.67 / 14.02	93.47 / 23.65	Best balanced cross-domain performance
PCA \(d=128\)	88.94 / 39.81	94.83 / 22.91	95.56 / 14.54	93.11 / 25.75	Stable at lower subspace dimension
PCA \(d=256\)	89.67 / 36.93	95.24 / 21.19	95.59 / 14.46	93.50 / 24.19	Peak cross-domain balance
PCA \(d=768\)	89.97 / 36.09	93.47 / 25.32	95.67 / 13.91	93.04 / 25.11	Minimal variation across dimensions

Furthermore, architectural evaluations in Table 3 verify that PRISM maintains top-tier performance across diverse CNN and Transformer models: on ImageNet-1k, it scores 89.96% on ResNet-50, 90.91% on Swin-T, and 88.76% on ViT-B/16 (averaging 89.88% AUROC / 35.83% FPR95); on ICONIC-444, it delivers 96.56% on ResNet-18, 94.65% on CCT-7/7x2, and 95.79% on ConvNeXt-S (averaging 95.67% AUROC / 14.02% FPR95).

Key Findings

• Cross-domain complementarity of scoring terms: Projected Mahalanobis distance acts as the primary detector when inter-class boundaries are structured (natural images and industrial parts), whereas residual energy drives detection when anomalies stem from unmodeled sensor dynamics, staining protocols, or textural anomalies (medical imaging).
• Fragility of OOD-calibrated ensembles: Aggregating layer-wise scores using weights optimized on an OOD validation set yields catastrophic transfer failures (e.g., MDSEns drops from 98.41% on medical to 54.41% on ImageNet-1k), confirming that true robustness requires ID-only geometric estimation.
• Robustness to subspace dimension: Varying the PCA projection dimension \(d\) between 128 and 768 produces less than 0.5% fluctuation in average AUROC, confirming that PRISM does not require per-dataset hyperparameter tuning.

Highlights & Insights

• The "fuse-then-model" paradigm unifies hierarchical representations: Instead of computing separate scores at each layer and trying to merge them downstream, combining normalized multi-layer features into a single vector preserves inter-layer correlations and enables holistic statistical modeling in one pass.
• Multiplicative coupling avoids arbitrary weighting coefficients: Multiplying the non-negative in-subspace Mahalanobis distance by the off-subspace residual energy creates a joint penalty surface where an anomaly along either axis significantly depresses the final ID confidence score without requiring manual trade-off weights.
• Practical zero-overhead deployment: Because global pooling collapses spatial dimensions early and PCA reduces the feature dimension to a compact subspace (\(d=512\)), PRISM adds less than 1.0% inference latency, making it immediately viable for real-time edge devices and high-throughput diagnostic pipelines.

Limitations & Future Work

• Vulnerability to fine-grained semantic overlap: Qualitative false-positive analysis shows that when an OOD object shares visual sub-attributes with an ID class (or appears as an incidental background element in a complex scene), PRISM can assign it high ID confidence.
• Heuristic uniform stage selection: Intermediate features are currently extracted from predetermined, uniformly spaced network stages; future work could investigate differentiable information-theoretic selection to automatically discover the most discriminative layer combinations.
• Linear subspace approximation limits: Modeling the ID manifold using linear PCA may underfit highly curved nonlinear manifolds; investigating lightweight kernel or local manifold approximations could enhance detection boundaries.

Related Work & Insights

• vs MDS [38] / MDS++ [50]: Standard MDS relies strictly on penultimate-layer features and fails on shallow-sensitive medical anomalies; MDS++ refines covariance normalization but remains restricted to a single depth. PRISM integrates multi-layer cues and orthogonal residual energy to ensure cross-domain stability.
• vs ViM [71]: ViM extracts residual features only from the final layer and pairs them with output logit energy; PRISM models residuals over the full hierarchical multi-layer representation and pairs them with in-subspace Mahalanobis distances derived from the identical geometric manifold.
• vs MDSEns [38] / GRAM [58]: MDSEns relies on supervised logistic regression over OOD validation sets and suffers severe domain shift drops; GRAM captures higher-order Gram matrices but degrades heavily on medical tasks (36.46% on PhaKIR). PRISM is strictly ID-only and exhibits consistent top-tier performance everywhere.

Rating

• Novelty: ⭐⭐⭐⭐ [Replaces fragmented layer-wise score aggregation with a unified multi-layer subspace projection and parameter-free multiplicative scoring]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Rigorous cross-domain validation over OpenOOD, OpenMIBOOD, and ICONIC-444 across 22 post-hoc baselines and diverse architectures]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, self-contained mathematical formulation, and thorough ablation studies]
• Value: ⭐⭐⭐⭐⭐ [Provides a reliable, tuning-free, and computationally efficient out-of-distribution detector ready for safety-critical deployment]