content_hash: 5be6ad0794f58207
Modality-Aware Out-of-Distribution Detection for Multi-Modal Action Recognition¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/LarsDoorenbos/modality-aware-ood
Area: Video Understanding
Keywords: Multimodal Action Recognition, Out-of-Distribution Detection, Post-hoc Detector, Cross-modal Correlation, Manifold Invariants
TL;DR¶
Addressing the limitation that existing multimodal action recognition OOD detection methods only apply regularization during training while discarding cross-modal relations at inference via off-the-shelf unimodal detectors, this paper introduces the first post-hoc modality-aware OOD detector. By capturing linear mapping residuals between unimodal and multimodal predictions, modeling low-variance linear invariants in joint feature space, and fusing them into normalized virtual logits, it establishes a new state of the art on the Multi-OOD benchmark.
Background & Motivation¶
Integrating complementary modalities—such as RGB frames, optical flow, and audio—substantially boosts the accuracy of modern action recognition architectures on closed-set benchmarks. However, when deployed in dynamic, unconstrained environments, these systems remain vulnerable to out-of-distribution (OOD) samples. Lacking proper anomaly rejection mechanisms, models produce overconfident predictions on anomalous inputs, causing silent operational failures. To mitigate this vulnerability, recent studies established the Multi-OOD benchmark and proposed training-time regularization schemes, such as Agree-to-Disagree (A2D), Dynamic Prototype Updating (DPU), and Flow Matching-based outlier synthesis (FM), which optimize feature margins or penalize cross-modal agreement on non-target classes.
Despite these training-phase innovations, a core tension persists: cross-modal dynamics are thoroughly leveraged during training, yet entirely discarded at test time. State-of-the-art multimodal pipelines uniformly rely on off-the-shelf OOD scoring functions originally engineered for unimodal networks (such as MSP, Energy, Mahalanobis distance, or ViM). These inference-time detectors merely evaluate concatenated representations or multimodal classification heads as if they were unimodal features, leaving the rich structural relationships across different modalities during inference unexploited.
Investigating this gap, the authors uncover an intrinsic discrepancy between in-distribution (ID) and OOD behaviors: for ID samples, the multimodal output distribution can be approximated with high precision by a class-wise linear mapping from the unimodal predictions (reaching a Pearson correlation of 0.998 on HMDB51). For OOD inputs, however, because inputs lie off the shared semantic space where unimodal representations converge, this structured linear predictability breaks down. Core idea: construct an inference-time training-free OOD detector that scores cross-modal semantic disruption via unimodal-to-multimodal mapping residuals, captures geometric off-manifold deviations via low-variance linear invariants, and normalizes them through virtual logits.
Method¶
Overall Architecture¶
The proposed method operates as a post-hoc, training-free framework that preserves existing model weights and requires no architecture modifications beyond accessible unimodal classification heads. During test-time inference, multimodal inputs pass through separate modality encoders to produce unimodal logits, while joint representations yield multimodal logits. The framework assesses out-of-distributionness across three complementary channels: first, evaluating the probability residual between unimodal linear aggregation and the true multimodal prediction; second, measuring Mahalanobis distance across the low-variance invariant subspace derived via Ledoit-Wolf covariance shrinkage; and third, scaling both anomaly indicators to match validation logit statistics, injecting them as virtual logits into a normalized softmax formulation that outputs the final OOD posterior.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
IN["Multimodal Test Input<br/>RGB Video / Optical Flow / Audio"] --> ENC["Modality Encoders & Fusion Head"]
ENC --> LOGITS["Unimodal Logits z(m) & Joint Features (e1,...,eM)"]
LOGITS --> S_SCORE["Modality-Aware Prediction Residual Score<br/>Class-conditional linear fit to multimodal probabilities"]
LOGITS --> R_SCORE["Feature-Space Low-Variance Invariants Score<br/>Ledoit-Wolf covariance shrinkage & subspace Mahalanobis distance"]
S_SCORE --> CALIB["Statistic Alignment & Scaling<br/>Calibration against validation max-logit mean & std"]
R_SCORE --> CALIB
CALIB --> VIRTUAL["Virtual Logit Fusion & Multimodal Normalization<br/>Unified mapping to OOD posterior probability g(x)"]
VIRTUAL --> OUT["Output OOD Decision Score"]
Key Designs¶
1. Modality-Aware Prediction Residual Score: Capturing Semantic Disruption via Linear Mapping Residuals
To overcome the disregard of inter-modality relationships at inference time, this design builds on the Platonic Representation Hypothesis that unimodal representations converge toward a shared latent space for in-distribution data. Using a held-out validation set of \(N\) ID samples, the detector solves an independent linear regression with intercept for each class \(c\):
$\(\tilde{\mathbf{w}}_c = \arg\min_{\tilde{\mathbf{w}}_c} \|\tilde{\mathbf{A}}_c \tilde{\mathbf{w}}_c - \mathbf{b}_c\|_2\)$
where \(\tilde{\mathbf{A}}_c = [\mathbf{A}_c, \mathbf{1}] \in \mathbb{R}^{N \times (M+1)}\) stacks the \(M\) unimodal logits for class \(c\) alongside a bias term, and \(\mathbf{b}_c \in \mathbb{R}^N\) collects multimodal logits. At test time, estimated multimodal logits are synthesized via \(\bar{\mathbf{z}}^{(f)} = [\tilde{\mathbf{z}}_1^\top \tilde{\mathbf{w}}_1, \dots, \tilde{\mathbf{z}}_C^\top \tilde{\mathbf{w}}_C]^\top\). Because raw logit norms vary widely between ID and OOD samples, both predicted and observed logits are projected onto the probability simplex via Softmax, defining the modality-aware score as the \(L_2\) probability distance \(s(x) = \|\text{softmax}(\bar{\mathbf{z}}^{(f)}) - \text{softmax}(\mathbf{z}^{(f)})\|_2\). For OOD samples, cross-modal coherence deteriorates, yielding substantially elevated residual distances.
2. Feature-Space Low-Variance Invariants Score: Probing Off-Manifold Samples via Orthogonal Subspaces
To identify samples that violate joint data manifold constraints without depending solely on output layers, the detector mines linear invariants from concatenated multimodal features of dimension \(D\). The method employs the hyperparameter-free Ledoit-Wolf shrinkage estimator to construct a well-conditioned sample covariance matrix \(\mathbf{\Sigma} = \mathbf{Q}\mathbf{\Lambda}\mathbf{Q}^\top\). It selects the \(k\) eigenvectors corresponding to the smallest eigenvalues that jointly explain less than \(p = 0.5\%\) of total feature variance, denoted by \(\mathbf{Q}_k \in \mathbb{R}^{k \times D}\) and \(\mathbf{\Lambda}_k \in \mathbb{R}^{k \times k}\). These tight dimensions capture rigid cross-modal geometric invariants of ID training data. The feature score computes the subspace Mahalanobis distance along these normal directions:
$\(r(x) = \sqrt{(x - \mu)^\top \mathbf{Q}_k \mathbf{\Lambda}_k^{-1} \mathbf{Q}_k^\top (x - \mu)}\)$
where \(\mu\) denotes the training feature mean. Deviations along low-variance directions are heavily penalized, providing high sensitivity to off-manifold multimodal anomalies.
3. Virtual Logit Fusion & Multimodal Normalization: Calibrating Heterogeneous Signals into an OOD Posterior
To fuse the prediction residual \(s(x)\), feature score \(r(x)\), and standard multimodal logits \(\mathbf{z}^{(f)}\), the framework scales both distance measures to match the empirical distribution of maximum logits on validation data:
$\(s'(x) = \gamma_s \left( \frac{\sigma_l}{\sigma_s} s(x) + \mu_l - \frac{\sigma_l}{\sigma_s} \mu_s \right)\)$
with \((\mu_l, \sigma_l)\) and \((\mu_s, \sigma_s)\) denoting means and standard deviations of maximum logits and raw residual scores, respectively, and \(r'(x)\) scaled identically. Hyperparameters \(\gamma_s\) and \(\gamma_r\) balance relative signal weights. Finally, the scaled values act as virtual logits for an auxiliary OOD class, normalized alongside original logits:
$\(g(x) = \frac{e^{s'(x)} + e^{r'(x)}}{\sum_{c=1}^C e^{z_c^{(f)}} + e^{s'(x)} + e^{r'(x)}}\)$
For prototypical ID inputs, prominent class logits dominate the denominator and drive \(g(x) \to 0\). When confronted with OOD samples exhibiting feature manifold divergence or cross-modal mapping breakdowns, the exponential terms in the numerator surge, reliably flagging anomalies.
Loss & Training¶
The detector operates purely post-hoc during inference without modifying network parameters. Base multimodal backbones are pre-trained with \(M+1\) standard cross-entropy losses: $\(\mathcal{L}_{cls}(x, y) = \mathcal{L}_{CE}(z^{(f)}, y) + \sum_{m=1}^M \mathcal{L}_{CE}(z^{(m)}, y)\)$ Default evaluations are conducted on networks trained with DPU (Dynamic Prototype Updating) regularization. Feature covariance and class regression weights are estimated via a single validation pass, requiring negligible computation. Hyperparameters are fixed globally across all benchmarks: \(p = 0.5\%\), \(\gamma_s = 0.4\), and \(\gamma_r = 1.25\).
Key Experimental Results¶
Main Results¶
Experiments adhere to the Multi-OOD benchmark protocols, evaluating Near-OOD and Far-OOD action detection across Kinetics-600, HMDB51, UCF101, HAC, and EPIC-Kitchens. Metrics report False Positive Rate at 95% True Positive Rate (FPR95 \(\downarrow\)) and Area Under the ROC Curve (AUROC \(\uparrow\)).
The following table presents Far-OOD detection performance with Kinetics-600 as the in-distribution dataset using video and optical flow modalities across three seeds:
| Method & Scoring Type | HMDB51 (FPR↓ / AUC↑) | UCF101 (FPR↓ / AUC↑) | HAC (FPR↓ / AUC↑) | EPIC-Kitchen (FPR↓ / AUC↑) | Average FPR↓ | Average AUC↑ | ID Acc (%) |
|---|---|---|---|---|---|---|---|
| A2D (Probability, A2Dp) | 61.7 / 80.6 | 57.0 / 78.0 | 42.0 / 87.0 | 40.1 / 85.1 | 50.2 | 82.7 | 73.7 |
| A2D (Feature, A2Df) | 63.7 / 79.1 | 62.2 / 75.3 | 46.5 / 85.2 | 36.1 / 88.7 | 52.1 | 82.1 | 72.5 |
| A2D (Hybrid, A2Dh) | 63.3 / 74.0 | 66.3 / 74.1 | 53.8 / 81.1 | 34.5 / 87.7 | 54.5 | 79.2 | 73.7 |
| FM (Outlier Synthesis) | 62.9 / 74.3 | 67.7 / 74.4 | 54.9 / 80.6 | 33.5 / 87.6 | 54.8 | 79.2 | 73.7 |
| DPU (Probability, DPUp) | 54.0±2.4 / 83.2±0.4 | 52.6±1.5 / 81.8±0.4 | 46.6±2.4 / 84.8±1.2 | 30.5±0.5 / 89.7±0.9 | 45.9 | 84.9 | 75.2 |
| DPU (Feature, DPUf) | 58.5±6.5 / 84.9±0.7 | 57.7±3.1 / 82.9±1.1 | 52.3±3.5 / 84.6±0.8 | 24.5±0.5 / 91.7±0.9 | 48.3 | 86.0 | 75.2 |
| DPU (Hybrid, DPUh) | 59.7±6.1 / 79.5±3.2 | 59.2±3.6 / 76.7±3.5 | 48.6±2.3 / 82.6±1.3 | 25.6±0.5 / 90.8±0.8 | 48.3 | 82.4 | 75.2 |
| Ours | 49.9±4.2 / 88.5±0.6 | 37.6±2.3 / 90.8±0.8 | 44.2±2.5 / 86.2±1.0 | 27.1±1.2 / 90.9±1.0 | 39.7 | 89.1 | 75.2 |
When evaluated on HMDB51 Far-OOD benchmarks, the proposed detector achieves an average FPR of 10.1 and AUC of 97.0, outperforming DPUh (11.1 FPR) and A2Dp (23.0 FPR). In the challenging multi-modal Near-OOD evaluation, the method establishes top overall performance with an average FPR of 43.1 and AUC of 84.5.
Ablation Study¶
To quantify the individual contributions of the three scoring streams—modality-aware residual \(s(x)\), feature invariant score \(r(x)\), and multimodal probability score \(g(x)\)—ablation experiments were conducted on Kinetics-600 Near-OOD and Kinetics-600:HAC Far-OOD:
| \(s(x)\) (Residual) | \(r(x)\) (Feature) | \(g(x)\) (Logit / Energy) | Near-OOD FPR↓ | Far-OOD (HAC) FPR↓ | Note |
|---|---|---|---|---|---|
| ✓ | ✗ | ✗ | 64.1 | 57.1 | Unimodal mapping residual only |
| ✗ | ✓ | ✗ | 71.6 | 52.7 | Low-variance manifold invariants only |
| ✗ | ✗ | ✓† | 64.8 | 46.8 | Standard multimodal Energy score only |
| ✓ | ✓ | ✗ | 68.4 | 45.9 | Residual + manifold feature invariants |
| ✗ | ✓ | ✓ | 64.8 | 41.7 | Conventional feature + logit hybrid |
| ✓ | ✗ | ✓ | 64.6 | 47.5 | Residual + multimodal logit |
| ✓ | ✓ | ✓ | 64.0 | 41.6 | Full model (three complementary sources) |
Note: (†) indicates standard Energy-based detection using only multimodal logits.
Key Findings¶
- Tri-source Complementarity: Isolating individual scores exposes severe blind spots: feature score \(r(x)\) alone yields a poor 71.6 FPR on Near-OOD, whereas residual score \(s(x)\) alone achieves 57.1 FPR on Far-OOD. Integrating all three signals achieves optimal marks on both benchmarks (64.0 and 41.6 FPR), confirming that robust multimodal OOD detection requires multi-level geometric and prediction constraints.
- Inadequacy of Raw Disagreement: While A2D relies on modality disagreement during training, using unimodal distribution distances directly as an inference score yields inferior performance (74.0 FPR on Near-OOD and 76.0 FPR on Far-OOD). The proposed linear residual \(s(x)\) dramatically reduces FPR by 9.9 and 16.0 points respectively.
- Triple-Modal Scalability: In EPIC-Kitchens:Kinetics experiments evaluating video, optical flow, and audio, audio alone exhibits weaker stand-alone detection (48.5 FPR / 86.7 AUC). However, incorporating audio into video+flow improves overall performance from 12.8 FPR / 97.2 AUC to 10.8 FPR / 98.0 AUC, proving the framework can exploit noisy or weaker auxiliary modalities effectively.
- Extreme Validation Efficiency: Estimating only \(M+1\) parameters per class in \(\tilde{\mathbf{w}}_c\) allows the framework to operate reliably under extreme data sparsity. Subsampling the validation set to 1% (or even 0.5%) causes less than 0.2 points of AUC degradation compared to the full validation split.
Highlights & Insights¶
- Turning Shared Latent Geometry into an OOD Principle: The method repurposes the Platonic convergence of multimodal representations into an elegant post-hoc indicator, showing that linear predictability between unimodal and joint heads shatters when exposed to anomalous distribution shifts.
- Universal Post-Hoc Compatibility: Free from training modifications, the detector functions as a modular drop-in enhancement across plain cross-entropy backbones or specialized training frameworks like A2D, DPU, and FM, as well as early-, mid-, and late-fusion architectures.
- Off-Manifold Invariant Probing: Rather than computing centroid distances across all principal directions, targeting the smallest-variance dimensions via Ledoit-Wolf shrinkage amplifies subtle cross-modal inconsistencies that standard Mahalanobis or kNN distances overlook.
Limitations & Future Work¶
- Requirement for Exposed Unimodal Heads: The framework assumes accessible unimodal classification logits. When applied to tightly coupled black-box foundation models lacking modality-specific heads, lightweight auxiliary linear probes must be attached post-hoc.
- Static Validation Dependency: Linear mappings and covariance statistics require a small, representative in-distribution validation split, which may require adaptation for online streaming or non-stationary data shifts.
- Future Directions: Extending the paradigm to open-vocabulary Vision-Language Models (e.g., CLIP and video LLMs) by tracking cross-modal token alignment residuals, and exploring non-linear manifold invariant modeling.
Related Work & Insights¶
- vs A2D (Dong et al., NeurIPS 2024): A2D pioneered multimodal OOD benchmarks and trained models by maximizing unimodal disagreement on non-ground-truth classes, but defaulted to unimodal detectors at inference. This work shows that inference-time unimodal disagreement fails as an OOD score, whereas cross-modal linear mapping residuals deliver superior detection.
- vs DPU (Li et al., CVPR 2025): DPU improves representation learning via dynamic prototype updates during training. The proposed post-hoc detector acts as an orthogonal downstream booster, cutting average Far-OOD FPR by 6.2 points on top of identical DPU weights.
- vs ViM (Wang et al., CVPR 2022): ViM established virtual logit matching combining feature nullspace projections and classifier logits for unimodal vision. This paper expands the paradigm to multimodal scenarios and introduces inter-modal prediction residuals as an indispensable third dimension.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ First post-hoc, modality-aware OOD detector for multimodal action recognition; elegantly leverages unimodal-multimodal prediction residuals to bridge the inference-time gap.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 5 Multi-OOD datasets, 14 benchmark tasks, bi- and tri-modal setups, multiple fusion mechanisms, and extensive ablation studies.
- Writing Quality: ⭐⭐⭐⭐⭐ Crystal-clear motivation, rigorous mathematical formulation, and thorough diagnostic insights.
- Value: ⭐⭐⭐⭐⭐ Highly practical, plug-and-play post-hoc tool that provides substantial reliability improvements for safety-critical robotic and video understanding systems.