Leveraging Dark Knowledge for Intrinsic Multimodal Out-of-Distribution Detection

Conference: ECCV 2026
Paper: ECCV Official
Area: AI Safety
Keywords: Out-of-distribution detection, Multimodal learning, Knowledge distillation, Dark knowledge, Uncertainty estimation

TL;DR

This paper proposes a Self Multimodal OOD Distillation framework that translates dark knowledge from unimodal experts' logits into likelihood-weighted uncertainty-aware soft targets and supervises a shared joint classification head via decoupled distillation, substantially strengthening intrinsic multimodal OOD detection without retraining heavy backbones.

Background & Motivation

In safety-critical deep learning applications such as autonomous driving, embodied robotics, and healthcare diagnostics, neural networks must not only deliver precise in-distribution (ID) classifications but also reliably identify out-of-distribution (OOD) inputs exhibiting semantic distribution shifts. As modern architectures increasingly operate on multimodal inputs, recent research has focused on multimodal OOD detection. Existing representative methods—such as A2D, AN, DPU, and Feature Mixing—primarily devise sophisticated retraining strategies to sculpt the cross-modal representation space, utilizing contrastive learning to enforce intra-class compactness, maximizing cross-modal disagreement on outlier samples, or synthesizing pseudo-outliers to regularize feature embeddings, followed by a joint classifier supervised with one-hot ground-truth labels.

However, these approaches fail to unleash the full synergy across modalities due to two fundamental tensions. First, prior pipelines treat all modalities equally, completely overlooking the pronounced performance disparities across different modalities—for instance, in action recognition, the video modality typically exhibits substantially stronger OOD detection performance than optical flow, and uniform combination exposes the joint system to noise from underperforming modalities. Second, training models exclusively with one-hot hard labels forces networks to fit both typical and atypical in-distribution samples with uniform, extreme confidence, ignoring inherent sample-level uncertainty and inevitably provoking severe overconfidence on OOD samples. Crucially, the logit-level dark knowledge within unimodal expert networks encodes rich relative uncertainty and inter-class correlations, yet remains entirely unexploited in existing feature-fusion frameworks.

Rather than relying on expensive end-to-end retraining of heavy modality backbones, this work taps directly into the intrinsic dark knowledge of unimodal experts: it treats the collective predictive logit behavior of unimodal models as a Bayesian approximation of the true predictive posterior, adaptively discounts weaker modalities via negative log-likelihood weighting, and transfers this uncertainty via decoupled target and non-target distillation into a lightweight joint classification head. Core idea: leverage uncertainty-aware dark knowledge encoded in unimodal expert logits to construct likelihood-weighted soft targets, guiding a joint classifier in the shared embedding space via decoupled distillation to activate intrinsic multimodal OOD synergy without backbone retraining.

Method

Overall Architecture

The proposed Self Multimodal OOD Distillation framework comprises a set of pre-trained, frozen unimodal expert networks alongside a lightweight joint multimodal classification head. When a multimodal sample is received, each modality stream is first processed by its corresponding feature extractor to produce unimodal embeddings; simultaneously, these embeddings are concatenated and fed into the joint classifier (serving as the Student), while the individual unimodal experts output their respective categorical probability distributions. The framework evaluates the prediction reliability of each unimodal expert on the current training batch using its average cross-entropy negative log-likelihood, dynamically computes normalized modality weights, and constructs a likelihood-weighted soft target distribution (serving as the Teacher). Finally, decoupled knowledge distillation transfers the uncertainty-aware dark knowledge to the joint classifier alongside the standard task cross-entropy loss.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Input Sample<br/>Video / Flow / Audio"] --> B["Multimodal Feature Fusion<br/>Concatenate embeddings z"]
    A --> C["Unimodal Expert Classifiers<br/>Output modality probabilities p_m"]
    C --> D["Likelihood-Weighted Modality Aggregation<br/>Negative log-likelihood weights w_m"]
    D --> E["Uncertainty Soft Target Distribution<br/>Weighted teacher distribution p_T"]
    B --> F["Joint Student Classification Head<br/>Output prediction probability p_S"]
    E --> G["Decoupled Dark Knowledge Distillation<br/>Target TCKD and non-target NCKD"]
    F --> G
    F --> H["Hard Label Task Supervision<br/>Ground-truth cross-entropy L_CE"]
    G --> I["Joint Objective Optimization<br/>10 epochs tuning classifier h only"]
    H --> I
    I --> J["Inference-Time OOD Scoring<br/>Plug-and-play MSP / Energy / GEN"]

Key Designs

1. Likelihood-weighted modality uncertainty aggregation: adaptively suppressing weak modality drag Different modalities possess intrinsically unequal discriminative power for OOD detection; for example, optical flow networks alone demonstrate significantly inferior OOD detection compared to raw video streams. A naive uniform average allows noisy, miscalibrated predictions from underperforming modalities to degrade the joint prediction. Drawing from Bayesian Model Averaging (BMA), the predictive posterior over predictors can be approximated under a uniform model prior as a likelihood-weighted mixture. The framework tracks the cross-entropy loss (negative log-likelihood) of each unimodal expert \(f^m\) on the current mini-batch \(\mathcal{B}\) to determine its dynamic weighting:

\[w_m = \frac{\exp(-\text{CE}(f^m, \mathcal{B}))}{\sum_{j=1}^M \exp(-\text{CE}(f^j, \mathcal{B}))}\]

By rewarding more reliable modalities and penalizing uncalibrated or erroneous experts on the fly, the aggregated teacher distribution \(p^{\text{T}} = \sum_{m=1}^M w_m p^m\) synthesizes the collective uncertainty of all modalities while preventing negative transfer from weaker branches.

2. Decoupled dark knowledge distillation: specialized supervision for near- and far-OOD shifts Conventional KL divergence distillation constrains the student to imitate the entire teacher distribution as an indivisible whole. However, the failure modes of near-OOD and far-OOD detection stem from fundamentally distinct geometric regimes: near-OOD samples lie directly adjacent to the ID decision boundary, demanding sharper sensitivity to ambiguous, hard samples; far-OOD samples reside in low-density feature voids far from ID clusters, where the primary risk is arbitrary overconfidence induced by one-hot cross-entropy supervision. The method decouples the distillation objective into target class knowledge distillation (TCKD) and non-target class knowledge distillation (NCKD):

\[\mathcal{L}_{\text{KD}} = \alpha \, \text{KL}(\boldsymbol{b}^{\text{T}} \parallel \boldsymbol{b}^{\text{S}}) + \beta \, \text{KL}(\hat{\boldsymbol{p}}^{\text{T}} \parallel \hat{\boldsymbol{p}}^{\text{S}})\]

Here, \(\boldsymbol{b} = [p_t, 1-p_t]\) represents the binary probability vector between the target class and all non-target classes, capturing the relative difficulty and margin uncertainty of each training sample. In contrast, \(\hat{\boldsymbol{p}}\) models the normalized conditional probability distribution across non-target classes, characterizing semantic structure in the hypothesis space. For near-OOD tasks, amplifying \(\alpha\) (\(\alpha=0.8, \beta=0.2\)) prioritizes learning boundary ambiguity; for far-OOD tasks, increasing \(\beta\) (\(\alpha=0.1, \beta=0.9\)) enforces relative probability smoothing over non-target logits, effectively dampening spurious spikes in unpopulated feature regions.

3. Self-provided student-teacher supervision in shared embedding space: zero backbone retraining Prior multimodal OOD improvements demand computationally prohibitive retraining of entire backbone encoders or complex multi-task generative outlier synthesis. In this framework, the Student classifier operates directly on the concatenated embedding representation of frozen unimodal encoders, while the Teacher's guidance is self-generated from the logit outputs of the exact same unimodal models. Because no external teacher architecture or additional sensory modules are required, the joint classification head converges in just 10 epochs on the combined features, drastically reducing training overhead while maintaining seamless orthogonality to arbitrary backbone choices.

Loss & Training

The overall training loss for optimizing the joint classifier \(h\) combines the standard multi-class cross-entropy loss \(\mathcal{L}_{\text{CE}}\) with the decoupled distillation objectives:

\[\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{CE}} + \alpha \, \text{KL}(\boldsymbol{b}^{\text{T}} \parallel \boldsymbol{b}^{\text{S}}) + \beta \, \text{KL}(\hat{\boldsymbol{p}}^{\text{T}} \parallel \hat{\boldsymbol{p}}^{\text{S}})\]

Distillation temperature is fixed at natural \(\tau = 1\) to preserve the true diversity and dispersion of the unimodal predictive distributions. The joint classification head is trained using the Adam optimizer with an initial learning rate of \(10^{-4}\) and a cosine annealing scheduler, a batch size of 16, running for only 10 epochs on a single NVIDIA A100 GPU. At inference time, the model incurs zero additional computational overhead, feeding the joint logits directly into standard post-hoc scoring functions such as MSP, Energy, GEN, or VIM.

Key Experimental Results

Main Results

Evaluation spans five action recognition multimodal benchmarks (HMDB51, UCF101, Kinetics-600, EPIC-Kitchens, HAC) across near-OOD and far-OOD tasks. All backbone architectures adopt SlowFast networks pre-trained on Kinetics-600.

Table 1: Multimodal Near-OOD detection results using video and optical flow (averaged over six random runs)

Method	HMDB51 25/26 FPR95(↓)	HMDB51 25/26 AUROC(↑)	UCF101 50/51 FPR95(↓)	UCF101 50/51 AUROC(↑)	EPIC-Kitchens 4/4 FPR95(↓)	Kinetics600 129/100 FPR95(↓)
Base	38.78	88.83	10.10	98.06	75.00	64.61
Base + Ours	34.99	90.12	5.92	98.57	73.84	61.37
A2D	38.34	88.22	7.09	98.19	66.23	63.04
A2D + Ours	37.86	88.78	5.32	98.51	69.40	61.64
AN (A2D+NPMix)	33.77	88.80	7.96	98.24	67.16	62.91
AN + Ours	34.25	90.07	5.51	98.56	65.19	61.80
DPU	34.42	89.15	7.57	98.17	63.81	61.59
DPU + Ours	33.25	89.65	7.84	98.31	64.70	59.46
FM (FeatureMix)	45.10	87.29	8.06	97.92	71.83	64.10
FM + Ours	37.73	88.99	6.06	98.49	72.16	62.75

Table 2: Multimodal Far-OOD detection results using video and optical flow with HMDB51 as ID (averaged over six random runs)

Method	Kinetics-600 FPR95(↓)	UCF101 FPR95(↓)	EPIC-Kitchens FPR95(↓)	HAC FPR95(↓)	Average FPR95(↓)	Average AUROC(↑)	ID ACC(↑)
Base	26.91	31.47	22.12	23.49	26.00	93.18	87.46
Base + Ours	22.17	28.60	16.01	20.75	21.88	94.31	88.12
A2D	20.18	33.87	12.43	15.85	20.58	94.44	87.34
A2D + Ours	17.40	27.55	8.73	16.47	17.54	95.04	87.69
AN	24.29	36.94	7.18	23.15	22.89	93.94	86.66
AN + Ours	19.98	29.35	10.40	19.38	19.78	94.49	86.96
DPU	20.75	28.39	4.33	20.64	18.53	95.41	87.34
DPU + Ours	19.11	25.25	6.50	18.15	17.25	95.16	87.94
FM	20.30	34.89	9.01	19.27	20.87	94.12	86.32
FM + Ours	16.88	27.99	12.03	16.88	18.95	93.92	86.26

Ablation Study

Table 3: Component ablation for Far-OOD detection with HMDB51 as ID (averaged over six random runs)

Config / Variant	UCF101 FPR95(↓)	HAC FPR95(↓)	EPIC-Kitchens FPR95(↓)	Kinetics-600 FPR95(↓)	Average FPR95(↓)	Average AUROC(↑)	ID ACC(↑)
Flow only	50.20	69.92	38.86	64.68	55.92	79.55	75.14
Video only	44.36	36.94	34.21	39.57	38.77	89.31	85.63
Ensemble w/o FF	32.84	28.85	20.64	30.33	28.16	92.80	87.91
Vanilla (\(\alpha=\beta=0\))	31.47	23.49	22.12	26.91	26.00	93.18	87.46
Ensemble w/ FF	32.50	25.88	21.44	28.05	26.97	93.08	88.03
Uniform KD	29.44	22.37	16.62	23.40	22.96	94.08	88.21
Best KD	30.13	20.84	16.90	22.19	22.52	94.17	87.89
Full Model (Ours)	28.60	20.75	16.01	22.17	21.88	94.31	88.12

Key Findings

• Dark knowledge distillation is essential: Naive prediction ensembling (Ensemble w/o FF: 28.16% average FPR95) and simple feature concatenation (Vanilla: 26.00%) consistently underperform our full distillation framework (21.88%), confirming that soft target probability geometries convey crucial uncertainty structures absent in hard one-hot supervision.
• Likelihood weighting protects against weak modalities: Moving from uniform weighting (Uniform KD: 22.96% FPR95) to dynamic likelihood weighting reduces average FPR95 to 21.88%. Under extreme synthetic corruption of the optical flow branch (Table 6), the naive ensemble FPR95 deteriorates to 33.50% on UCF101, whereas our method preserves an FPR95 of 7.38%, demonstrating effective automatic suppression of corrupted modalities.
• Task-specific decoupled distillation synergy: Detailed analysis verifies that near-OOD detection hinges primarily on target-class binary uncertainty (TCKD alone achieves 34.99% FPR95 on HMDB51 25/26), whereas far-OOD separation depends heavily on non-target relational knowledge (NCKD alone reduces EPIC-Kitchens FPR95 from 22.12% down to 16.01%).

Highlights & Insights

• Intrinsically activating latent capability without heavy retraining: Rather than re-engineering representation spaces via costly end-to-end multi-modal retraining, this work unlocks the hidden synergy of existing unimodal models by simply refining the joint classification layer, delivering gains up to 30% while remaining completely model-agnostic.
• Principled formulation of multimodal uncertainty estimation: The method bridges Bayesian model averaging and logit distillation, establishing that the collective output of unimodal classifiers constitutes an effective surrogate for missing sample-level ground-truth uncertainty labels.
• Decoupled supervision tailored to OOD geometric regimes: Clarifying that near-OOD benefits from boundary ambiguity mining via TCKD while far-OOD benefits from void overconfidence suppression via NCKD offers an intuitive, transferable blueprint for post-hoc OOD training.

Limitations & Future Work

• Author-admitted limitation: In open-world deployments where the exact nature of incoming OOD shifts is completely unknown in advance, selecting optimal values for \(\alpha\) and \(\beta\) relies on proxy-OOD validation splits or balanced defaults (\(\alpha=0.5, \beta=0.5\)).
• Broader multimodal validation: Empirical evaluations focus on video action recognition benchmarks involving video, flow, and audio modalities; expanding validation to modern vision-language models (VLMs) and heterogeneous sensory streams (e.g., LiDAR-camera fusion) remains an open area for future exploration.
• Future direction: Developing an adaptive, sample-dependent gating network that dynamically predicts \(\alpha\) and \(\beta\) based on instance-level prediction entropy or cross-modal disagreement could further automate hyperparameter tuning.

Related Work & Insights

• vs Multimodal OOD Baselines (A2D / AN / DPU / Feature Mixing): Previous works focus on complex end-to-end retraining and artificial outlier generation, ignoring intrinsic unimodal competence and modality imbalance; this method freezes backbones and achieves competitive or superior OOD performance purely via logit-level dark knowledge distillation.
• vs Logit Distillation & DKD (Hinton et al. / Zhao et al. / Yang et al.): While decoupled knowledge distillation was originally designed for image classification and single-modal compression, this work adapts the formulation to multimodal OOD detection, uncovering the distinct operational mechanisms of TCKD for near-OOD boundaries and NCKD for far-OOD voids.

Rating

• Novelty: ⭐⭐⭐⭐☆ Unlocks unimodal logit dark knowledge for multimodal OOD detection with an elegant decoupled formulation.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Extensive evaluation spanning five benchmarks, dual OOD tasks, multiple baselines, and robustness under modality corruption.
• Writing Quality: ⭐⭐⭐⭐⭐ Well-structured narrative with rigorous mathematical derivations and clear intuitive visualizations.
• Value: ⭐⭐⭐⭐☆ Highly practical, plug-and-play approach that eliminates the need for expensive backbone retraining.