General Incomplete Multimodal Learning via Dynamic Quality Perception

Conference: ECCV 2026
Paper: ECCV 2026 Poster
Code: https://github.com/Yu-Five/GIML
Area: Multimodal VLM
Keywords: multimodal learning, dynamic quality perception, modality missing, noise-semantic decoupling, incomplete multimodal learning

TL;DR

To tackle the coexistence of complete inter-modality absence and severe intra-modality noise degradation, this paper proposes GIML, a unified framework modeling heterogeneous missing patterns as continuous quality degradation, combining noise-semantic decoupled probabilistic representations with a calibrated noise-aware quality estimator for adaptive fusion.

Background & Motivation

Multimodal learning has achieved remarkable generalization performance across vision-audio analysis, sentiment understanding, and multisensory perception tasks. In practical deployment scenarios, however, hardware sensor faults, communication bandwidth limitations, or harsh weather conditions inevitably lead to incomplete multimodal inputs. Prior work in incomplete multimodal learning predominantly focuses on "inter-modality missing," which assumes that entire modalities are completely missing and relies on cross-modal imputation or joint subspace alignment to reconstruct missing information. In realistic physical environments, however, modality incompleteness frequently manifests as "intra-modality degradation"—where input modalities remain present but suffer from severe corruption, such as Gaussian noise, motion blur, or partial occlusion, yielding extremely low signal-to-noise ratios. Recent attempts like T2DR and TMDC adopt sequential two-stage pipelines to first denoise corrupted inputs and then address missing modalities, but this cascaded structure suffers from optimization objective conflicts and relies on noise-entangled deterministic representations that collapse under unseen corruption patterns.

The fundamental tension lies in the fact that conventional binary indicators (\(0\) for missing and \(1\) for present) cannot describe continuous quality attenuation ranging from slight perturbation to complete absence, while standard unsupervised uncertainty metrics (such as energy scores or feature variance) become overly confident under severe noise and improperly assign non-negligible weights to corrupted modalities. Furthermore, deterministic encoders inherently entangle task-relevant semantic features with degradation patterns, causing models to overfit specific training corruptions and fail under out-of-distribution noise.

This paper's angle of attack is to dissolve the artificial dichotomy between discrete missingness and staged denoising by formalizing modality degradation as a continuous information attenuation process. Core idea: model multimodal incompleteness as a continuous quality degradation spectrum, isolate noise-invariant semantics via a noise-semantic decoupled probabilistic distribution, and calibrate a quality estimator under controlled noise supervision to smoothly decay corrupted modality weights based on dynamic quality perception.

Method

Overall Architecture

The pipeline of GIML comprises four coherent stages: continuous degradation simulation, probabilistic feature decoupling, noise intensity perception, and quality-aware dynamic fusion. Given multimodal inputs, a continuous noise injection function simulates varying degradation intensities (ranging from clean signals to complete masking). Each modality encoder extracts preliminary representations, which are processed by the Noise-Semantic Decoupled (NSD) module into Gaussian distributions where the mean captures task-relevant semantics and the variance captures degradation-induced uncertainty. A lightweight Noise-aware Quality Estimator (NQE) then maps the predicted variance to the true noise level under gradient-detached supervision. Finally, the fusion module adaptively aggregates semantic features using normalized inverse-variance weights derived from the perceived quality scores.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Inputs & Continuous Degradation Injection<br/>x_i = f_n(x̃_i, η_i)"] --> B["Modality Encoders Feature Extraction<br/>z_i = f_e(x_i)"]
    B --> C["Noise-Semantic Decoupled Module<br/>Gaussian parameterization & prior regularization"]
    C --> D["Noise-aware Quality Estimator<br/>Lightweight mapping predicting degradation severity"]
    D --> E["Quality-Aware Dynamic Fusion<br/>Smooth exponential decay weighting & adaptive aggregation"]
    E --> F["Downstream Classification & Multi-Objective Joint Optimization"]

Key Designs

1. Noise-Semantic Decoupled Module: disentangling task semantics from corruption interference

Standard deterministic embeddings entangle task-relevant semantics with degradation artifacts, leading to severe overfitting to specific noise profiles. To address this limitation, the NSD module parameterizes each modality embedding \(z_i^{(v)}\) as a multivariate Gaussian distribution, predicting a semantic mean \(\mu_i^{(v)} = f_\mu^{(v)}(z_i^{(v)})\) and a log-variance \(\log \sigma_i^{(v)} = f_\sigma^{(v)}(z_i^{(v)})\). Semantic information is explicitly captured by the mean, while degradation uncertainty is modeled by the variance. Using the reparameterization trick, stochastic representations are sampled as \(s_i^{(v)} = \mu_i^{(v)} + \sigma_i^{(v)} \odot \epsilon\) with \(\epsilon \sim \mathcal{N}(0, \mathbf{I})\).

To enforce rigorous separation between semantic content and corruption uncertainty, two complementary constraints are introduced: first, a unimodal cross-entropy classification loss \(\mathcal{L}_{cls}^{(v)}\) is applied to sampled feature \(s_i^{(v)}\) to ensure that \(\mu_i^{(v)}\) retains class-discriminative information under stochastic perturbation; second, a degradation-aware prior regularization term is established:

\[\mathcal{L}_{reg}^{(v)} = \frac{1}{N} \sum_{i=1}^N \mathrm{KL}\Big(\mathcal{N}(\mu_i^{(v)}, (\sigma_i^{(v)})^2 \mathbf{I}) \parallel \mathcal{N}(0, (\eta_i^{(v)})^2 \mathbf{I})\Big)\]

By setting the prior mean to zero, semantic bias is avoided while aligning the predicted feature variance \(\sigma_i^{(v)}\) with the ground-truth degradation intensity \(\eta_i^{(v)}\). Consequently, semantic features and degradation noise are isolated into orthogonal statistical dimensions, enabling semantic representations to remain invariant to diverse and unseen noise distributions.

2. Noise-aware Quality Estimator: calibrating uncertainty across the full degradation spectrum

Unsupervised quality proxies derived from feature dispersion or energy scores often fail when noise is either very weak or extremely severe, yielding uncalibrated estimates that overweight severely degraded inputs. The lightweight estimator \(f_t^{(v)}\) resolves this by learning a direct mapping from the decoupled uncertainty variance \(\sigma_i^{(v)}\) to the intrinsic data degradation intensity \(\hat{\eta}_i^{(v)}\):

\[\hat{\eta}_i^{(v)} = f_t^{(v)}(\sigma_i^{(v)})\]

Unlike implicit estimation methods, NQE is directly supervised by the injected degradation scalar \(\eta_i^{(v)}\) using mean squared error:

\[\mathcal{L}_{mse}^{(v)} = \frac{1}{N} \sum_{i=1}^N \left(\hat{\eta}_i^{(v)} - \eta_i^{(v)}\right)^2\]

A gradient detach operation is applied between the estimator and the representation backbone during backpropagation, preventing regression gradients from perturbing representation learning. This explicit calibration guarantees that quality perception remains monotonic and accurate across the entire degradation continuum from \(\eta=0\) (clean) to \(\eta=1\) (fully absent).

3. Quality-Aware Dynamic Fusion: smooth adaptive weighting via the inverse-variance principle

Once the calibrated degradation intensity \(\hat{\eta}_i^{(v)}\) is estimated, it must be translated into adaptive fusion weights \(w_i^{(v)} \in [0, 1]\). Following the inverse-variance principle in statistical estimation, the reliability of a modality decays exponentially as degradation escalates, defined as:

\[w_i^{(v)} = \frac{\exp\left(-(\hat{\eta}_i^{(v)})^2\right)}{\sum_{k=1}^V \exp\left(-(\hat{\eta}_i^{(k)})^2\right)}\]

When a modality is completely missing, its estimated degradation approaches infinity relative to clean inputs, smoothly driving its weight \(w_i^{(v)} \to 0\). In this manner, GIML seamlessly recovers conventional missing-modality behavior while dynamically downweighting corrupted features during partial degradation, preventing noisy modalities from contaminating the joint representation.

Loss & Training

The overall training objective of GIML combines the multimodal classification task loss, unimodal classification losses, distribution regularization, and noise intensity regression:

\[\mathcal{L} = \mathcal{L}_{cls}^f + \sum_{k=1}^V \mathcal{L}_{cls}^{(k)} + \mathcal{L}_{reg} + \mathcal{L}_{mse}\]

Here \(\mathcal{L}_{cls}^f\) represents the cross-entropy loss evaluated on the fused representation \(z_{\tau i} = f_f(\sum_{v=1}^V w_i^{(v)} s_i^{(v)})\), while \(\mathcal{L}_{cls}^{(k)}\) provides auxiliary unimodal semantic supervision. The framework is trained end-to-end using SGD (momentum 0.9, weight decay 1e-4, initial learning rate 1e-3). Training batches contain a 50/50 mix of clean samples and samples with uniformly sampled mask rates from \(0.0\) to \(1.0\) at intervals of \(0.1\), allowing the model to master both clean fusion and degraded scenarios simultaneously.

Key Experimental Results

Main Results

On audio-visual emotion recognition (CREMA-D) and action recognition (Kinetics-Sounds), GIML is evaluated against state-of-the-art two-stage denoising and imputation baselines TMDC and T2DR under varying intra-modality mask ratios \((r_a, r_v)\) and inter-modality missing configurations.

Dataset	Modality Availability (inter)	Intra Degradation Ratio \((r_a, r_v)\)	TMDC Acc (%)	T2DR Acc (%)	GIML (Ours) Acc (%)	Gain over Prev. SOTA
CREMA-D	AV (full modalities)	(0.0, 0.0)	66.99	67.93	73.94	+6.01%
CREMA-D	AV (moderate degradation)	(0.5, 0.5)	61.69	62.69	67.80	+5.11%
CREMA-D	AV (imbalanced degradation)	(0.1, 0.5)	61.85	63.82	68.18	+4.36%
CREMA-D	AV (severe degradation)	(0.3, 0.7)	57.23	59.89	62.93	+3.04%
CREMA-D	A (audio only)	(0.0, –)	52.66	53.92	55.57	+1.65%
CREMA-D	V (visual only)	(–, 0.0)	46.80	48.95	60.38	+11.43%
CREMA-D	V (visual degraded)	(–, 0.5)	41.68	42.63	55.30	+12.67%
KS	AV (full modalities)	(0.0, 0.0)	60.40	62.73	69.51	+6.78%
KS	AV (moderate degradation)	(0.5, 0.5)	58.08	60.31	63.54	+3.23%
KS	AV (imbalanced degradation)	(0.1, 0.5)	57.67	59.62	65.20	+5.58%
KS	AV (severe degradation)	(0.3, 0.7)	56.75	59.03	61.68	+2.65%
KS	V (visual degraded)	(–, 0.5)	43.04	40.85	50.67	+7.63%

Ablation Study

To investigate the individual contributions of the loss formulations and module designs, ablation experiments on CREMA-D analyze accuracy alongside Spearman rank correlations between variance/estimated noise and actual feature distortion \(\Delta f\).

Configuration Variant	Accuracy Acc (%)	\(\rho(\mathrm{var}, \Delta f)\) (A / V)	\(\rho(\hat{\eta}, \Delta f)\) (A / V)	Note & Mechanism Analysis
GIML Full Model	67.41	0.927 / 1.000	0.991 / 0.991	Full model: variance aligns with degradation, estimation is highly calibrated
w/o \(\mathcal{L}_{reg}\) (no prior reg.)	65.83	0.336 / 0.273	1.000 / 1.000	Variance fails to reflect noise level, destroying uncertainty interpretability
w/o \(\mathcal{L}_{mse}\) (no regression)	64.01	1.000 / 1.000	0.018 / 0.527	Quality estimator uncalibrated, fusion weights degenerate toward random
GIML-NQE (\(w^{(v)} = 1\))	63.39	-	-	Disables dynamic weighting; accuracy drops by 3.95% under asymmetric noise
GIML-NSD (raw features)	48.95	-	-	Replaces decoupled Gaussian with \(z^{(v)}\); accuracy collapses by 5.13% under Gaussian noise

Key Findings

• Robustness under Asymmetric Corruption: Under imbalanced corruption conditions like \((0.1, 0.5)\), prior methods suffer significant performance degradation because corrupted features mislead the fusion process. GIML dynamically downweights the degraded channel, preserving more than 4.3% accuracy advantages.
• Superior Generalization across Unseen Noise Types: Models trained solely on Mask corruption generalize remarkably well to unseen continuous intensities and novel noise distributions (zero-mean Gaussian noise, rain, fog, snow). Under severe Gaussian distribution shift \((2, 5)\), GIML attains 61.82% accuracy on CREMA-D, vastly outperforming T2DR (14.11%) and TMDC (45.83%).
• Distinct Roles of Dual Regularization: \(\mathcal{L}_{reg}\) guarantees that the latent variance specifically models data corruption rather than semantic content, whereas \(\mathcal{L}_{mse}\) anchors this uncertainty to physical noise severity.

Highlights & Insights

• Unified Continuous Degradation Formulation: By reinterpreting binary missingness as the asymptotic limit of continuous degradation, GIML eliminates the need for separate architectures or disjoint training stages for noise removal versus missing data imputation.
• Supervised Quality Calibration: Rather than relying on unreliable heuristics like predictive entropy, GIML introduces controlled perturbations to directly supervise the quality perception network, producing monotonic, physically grounded fusion coefficients.
• Probabilistic Separation against Noise Overfitting: Decoupling representations into semantic means and uncertainty variances prevents classifiers from latching onto noise-specific patterns, providing strong out-of-distribution robustness.

Limitations & Future Work

• Reliance on Artificial Corruption Priors: Supervision during training requires synthetic degradation functions (such as random masking); discrepancies may emerge when encountering highly complex non-linear sensor failures or physical lens distortions in real deployment.
• Balance of Unimodal vs. Multimodal Objectives: Hyperparameter \(\beta_1\) governing unimodal classification supervision requires tuning to prevent over-constraining multimodal fusion interactions.
• Scaling to Multimodal LLMs: Currently verified on CNN and Transformer backbones; exploring continuous quality-aware dynamic token pruning within large vision-language models represents an impactful future direction.

Related Work & Insights

• vs TMDC / T2DR: Earlier methods divide intra-modality denoising and inter-modality completion into two sequential stages, creating optimization conflicts and high inference overhead; GIML unifies both phenomena into a single-stage, end-to-end continuous formulation.
• vs Classical Probabilistic Embeddings: Prior probabilistic representations lack ground-truth noise calibration, often yielding distorted variance estimates under extreme noise; GIML enforces ground-truth MSE calibration to guarantee reliable uncertainty estimation across the entire degradation spectrum.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Formulates modality missing and intra-modality corruption as a unified continuous spectrum with explicit supervised calibration.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Extensive evaluations across five multimodal benchmarks spanning audio-visual, text-vision, and depth modalities under unseen intensities and noise types.
• Writing Quality: ⭐⭐⭐⭐⭐ Rigorous methodology presentation, clear mathematical formulation, and consistent conceptual framing.
• Value: ⭐⭐⭐⭐⭐ Offers a robust, practically actionable framework for multimodal sensor fusion in real-world noisy and incomplete environments.

Related Papers

• [AAAI 2026] MCMoE: Completing Missing Modalities with Mixture of Experts for Incomplete Multimodal Action Quality Assessment
• [CVPR 2026] Anchor-Guided Gradient Alignment for Incomplete Multimodal Learning
• [ECCV 2026] GKDT: General Keypoint Detection Transformer
• [CVPR 2026] UI-Lens: Assessing General MLLMs' Potential to Automate UI Display Quality Assurance
• [NeurIPS 2025] CyIN: Cyclic Informative Latent Space for Bridging Complete and Incomplete Multimodal Learning