Progressive Representation Learning for Multimodal Sentiment Analysis with Incomplete Modalities

Conference: ECCV 2026
Paper: ECCV page
Area: Audio & Speech
Keywords: multimodal sentiment analysis, missing modalities, Fisher information, progressive interaction, cross-modal alignment

TL;DR

PRLF decomposes "which modality is trustworthy right now" into two complementary signals — classification confidence and Fisher information — to pick a per-sample dominant modality, then lets the auxiliary modalities self-refine first and align to the dominant one through projection and residual denoising over four iterations, reaching average F1 of 77.02 / 76.24 / 81.19 on CMU-MOSI / CMU-MOSEI / SIMS under inter- and intra-modality missingness.

Background & Motivation

Multimodal sentiment analysis (MSA) combines language, acoustic, and visual signals to infer a speaker's emotional polarity, and the mainstream line of work keeps refining the fusion mechanism itself: MISA splits representations into modality-invariant and modality-specific parts, MMIM constrains fusion with hierarchical mutual information maximization, Self-MM attaches self-supervised auxiliary tasks to every modality, and TETFN relies on text-dominant cross-modal mapping. All of them assume that the three modalities are available at inference time. In practice, environmental noise, hardware failure, transmission loss, and privacy restrictions make a modality disappear entirely or partially, so the input is often incomplete or corrupted. Two families of methods target this gap: generative approaches such as DiCMoR, which uses class-conditional normalizing flows to align the distribution of recovered features with the true one (MRAN aligns and reconstructs modalities as well), and distillation-based approaches such as UMDF and CorrKD, where a teacher trained on complete modalities transfers cross-modal knowledge to a student that must cope with missing inputs.

Neither family answers a question the data itself raises: when a modality goes missing, the value of each remaining modality differs from sample to sample, yet these methods keep the traditional fusion paradigm and drop missing and complete modalities into the same attention operator without distinction. The authors use two measurements to show what that blindness costs. The first is the Fisher Information Matrix (FIM): discarding critical data in one modality collapses its information content, while discarding non-critical data barely matters — modality importance is sample-dependent, so no global modality-weight table can work. The second is feature phase: across 300 samples, the angular deviation between features extracted at different missing rates and those extracted from complete modalities grows with the missing rate, which explains why naive fusion backfires — the missing modality's feature direction has already drifted, and fusing it drags the well-learned representations of intact modalities off course.

Worse, the model itself usually does not notice that a modality became unreliable. The evidence (Fig. 3 in the paper) is that the visual modality receives complete input during epochs 10–14 and loses key frames at epoch 15, yet its classification confidence stays high — apparently memorizing facial features rather than relying on the actual input — while the trace of the FIM drops sharply. The common practice of picking the dominant modality by confidence therefore breaks down under missingness and needs a signal that reflects how much effective information the input still carries. Core idea: rather than completing or distilling, estimate per-sample modality reliability, select a dominant modality, and let the remaining modalities align to the dominant one's feature direction step by step — early iterations do only intra-modal refinement to avoid noise, and cross-modal interaction is released only in later iterations, so fusion strength rises progressively with training. ⚠️ The paper defines this "progressive" property strictly over iteration steps (interaction depth grows with the step index), not as a curriculum over the missing rate — the missing rate stays constant during training.

Method

Overall Architecture

PRLF takes a video segment's three modality sequences \(S=[X^V, X^A, X^L]\) as input, where \(X^V, X^A, X^L\) are the visual, acoustic, and language sequences. The pipeline splits into two parts: AMRE (Adaptive Modality Reliability Estimator) answers "which modality is most trustworthy for this sample," and ProgInteract (Progressive Interaction) uses that answer to progressively align the auxiliary modalities. Each modality first passes through a dedicated encoder \(f_m=\mathcal{E}_m(X^m)\); the resulting unimodal features feed both AMRE (to score reliability) and ProgInteract (to interact). AMRE fuses classification confidence with Fisher information into a per-modality importance \(\mu\); the largest \(\mu\) marks the dominant modality and the rest are auxiliary. ProgInteract then runs \(T=4\) iterations. Each iteration does intra-modal self-refinement, then a cross-modal interaction weighted by \(\mu\), and blends the two by a time-dependent coefficient \(\lambda_t\) (favoring self-refinement early, cross-modal features late). The blended dominant and auxiliary features enter a Decomposer, which projects the dominant modality into the auxiliary space, imposes an orthogonality (phase) constraint on the residual, and denoises it; the refined auxiliary features feed the next iteration. After the last iteration the three final representations are concatenated and classified with cross-entropy.

The missingness setup deserves to be stated up front. The paper distinguishes inter-modality missingness (a whole modality disappears; evaluation covers \(\{l\},\{a\},\{v\},\{l,a\},\{l,v\},\{a,v\}\)) from intra-modality missingness (some frame-level features inside a modality sequence are randomly dropped, controlled by a rate \(p\in\{0,0.1,\dots,0.9\}\)). Missing entries are filled with zero vectors and the model performs no reconstruction at all — it relies on reliability estimation and direction alignment to resist noise. During training the missing pattern is resampled every epoch (the seed changes per epoch, so the same sample sees a different pattern each round), the missing rate stays constant, and missingness is applied at whole-modality granularity.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Three modality inputs<br/>language / acoustic / visual"] --> B["Unimodal encoders<br/>f_L, f_A, f_V"]
    B --> C["Modality reliability estimation<br/>confidence + Fisher information"]
    C --> D["Progressive interaction<br/>λ_t from unimodal to cross-modal"]
    D --> E["Decomposer alignment<br/>projection + residual denoising"]
    E -->|t < 4, loop back| D
    E -->|t = 4| F["Concatenate three representations<br/>cross-entropy prediction"]

Key Designs

1. Modality reliability estimation: patching the spot where confidence lies with Fisher information

A router network assigns each modality its own classification head \(\hbar_m\); for a given sample the confidence of modality \(m\) on the correct class is \(\alpha^{(i)}_m\), and the three confidences are concatenated and L1-normalized into \(\hat{\alpha}^{(i)}\). This step comes with a per-modality cross-entropy loss \(\mathcal{L}_{uni}\) that writes "each modality must also classify correctly on its own" into the training objective. But the paper's evidence shows that confidence alone misjudges: with key frames missing the visual head still reports high confidence, which suggests it leans on memorization rather than on the input. Hence the second signal, the trace of the Fisher Information Matrix. For a unimodal encoder together with its classification head, the FIM trace can be written directly as the expected squared gradient norm:

\[\mathrm{Tr}(F_m^{(i)}) = \mathbb{E}\left[g_m g_m^\top\right] = \left\|g_m\right\|_2^2\]

It measures how sensitive the output is to parameter perturbations, i.e. roughly how much effective information the modality's input contributes. The paper also gives a theoretical account: viewing modality \(m\) as 16 frames, the gradient with respect to the parameters decomposes approximately into per-frame contributions; with a binary mask \(\delta_t\) for missing frames it becomes \(\sum_t \delta_t g_t\). Key frames carry a much larger gradient norm than non-key frames (\(|g_t^{key}|_2^2 \gg |g_t^{non}|_2^2\)), so masking them out removes the dominant contribution and \(\mathrm{Tr}(F_m)\) drops sharply — exactly what Fig. 3b shows. Normalizing the three FIM traces gives \(\hat{\beta}^{(i)}\).

The two signals fail in complementary regimes. Early in training gradients are weak and every modality has low Fisher information, so FIM cannot rank anything; at that stage a relatively higher confidence at least indicates that the modality's feature space contains more separable patterns. Later, FIM becomes reliable and is more accurate than a confidence that can deceive itself. The method therefore uses the relative growth of the FIM trace between consecutive epochs, \(\Delta^{(t,i)}_m\), to set the mixing weight dynamically, so that \(\mu\) leans toward the confidence term early and toward the Fisher term once the FIM rises. ⚠️ Equation (11) (the exact form of the weight \(w^{(t,i)}\)) is corrupted in the cached text; it is only readable as a sigmoid over \(\Delta\), so the precise expression should be checked against the original paper.

\[\mu^{(t,i)} = (1-w^{(t,i)})\,\hat{\alpha}^{(i)} + w^{(t,i)}\,\hat{\beta}^{(i)}\]

The resulting \(\mu\) serves both as the criterion for choosing the dominant modality and as the weighting coefficient of cross-modal interaction — one quantity answers "whom to trust" and "toward whom to align" at once, with no extra gating network to train.

2. Progressive interaction: moving fusion from "listen only to yourself" to "listen only to others"

Since missingness drifts the auxiliary modalities' feature directions, full-strength cross-modal fusion from the first step would contaminate the intact representations. ProgInteract therefore does not fuse once but iterates \(T\) steps, each with three layers. The first is intra-modal self-refinement, \(f_m^{self} = f_m + \mathrm{Dropout}(\cdot)\), i.e. a further pass inside the modality to extract discriminative information (⚠️ only the residual structure and the Dropout are legible in Eq. (13) of the cached text; check the original for the exact form). The second is cross-modal interaction, where modality importance \(\mu\) takes effect explicitly — attention over the auxiliary modalities is modulated by \(\mu\), so the dominant modality (largest \(\mu\)) naturally receives more weight; the two cross-modal features are concatenated and passed through another softmax attention to form that step's \(f_m^{cross}\). The third is the time-dependent mixing coefficient \(\lambda_t\): the step output linearly mixes self-refined and cross-modal features, and \(\lambda_t\) decreases monotonically with the iteration index, approaching 1 early (almost only intra-modal features) and 0 late (almost entirely cross-modal). ⚠️ The expression for \(\lambda_t\) is lost in the cached layout; what can be confirmed is a linear decay with \(t\) that equals 1 at \(t=1\) and 0 at \(t=T\) — check the original for the precise form.

This schedule is where "progressive" actually lives. Early in training the model has not converged and the cross-modal signal contributed by missing modalities is dominated by noise, so pinning \(\lambda_t\) to the self-refinement side acts as a gradual-unfreezing gate on fusion; as iterations proceed and intra-modal representations stabilize, \(\lambda_t\) falls and cross-modal interaction is released, letting auxiliary modalities approach the dominant one. Compared with fixed fusion weights, it buys a "stabilize first, fuse later" optimization path for two cheap hyper-parameters (one linear schedule and the iteration count).

3. Decomposer alignment: projecting the dominant modality into the auxiliary ones, denoising the residual and feeding it back

Attention weighting alone cannot guarantee that an auxiliary modality truly turns toward the dominant one, because attention tunes magnitude rather than direction. The Decomposer instead decomposes the dominant feature explicitly into a part the auxiliary modality can explain and a part it cannot: the dominant and auxiliary features are concatenated and fed to a gating network that outputs a projection weight \(g^t_{aux}\) of the same dimension as the features; the projection is \(\mathrm{proj}=g^t_{aux}\odot f^{fuse,t}_{dom}\), and the residual \(\mathrm{res}=f^{fuse,t}_{aux}-\mathrm{proj}\) captures what the auxiliary modality holds that the dominant one does not cover.

To keep both alignment and complementarity alive, the paper constrains the projection and the residual to be orthogonal, which serves as the phase consistency loss:

\[\mathcal{L}_{phase} = \frac{1}{N}\sum_{t,n}\left[(\mathrm{proj}_n^t)^\top \mathrm{res}_n^t\right]^2\]

Minimizing it pushes the projection term toward the dominant modality's direction while forcing the residual to carry independent, complementary content, so the two do not entangle — the paper calls this "moderate phase convergence." The residual may still carry noise, so a denoising network estimates a noise component from it (ReLU followed by Dropout), and the auxiliary feature is updated as projection plus the denoised residual before flowing into the next iteration; the coefficient \(\gamma\) (0.8 in the experiments) controls how much residual information is admitted. The dominant feature stays unchanged within the step and acts as the anchor of the iteration. Each round therefore makes the auxiliary modalities strike a constrained compromise between being pulled toward the dominant modality and keeping their own information — a compromise enforced by the loss rather than by a hand-designed fusion formula.

A Worked Example

Take a sample with all three modalities present in which language is judged dominant (\(T=4\)). AMRE produces \(\mu_L>\mu_A>\mu_V\) (say roughly 0.52 / 0.28 / 0.20 after normalization), so \(f_L\) is dominant and \(f_A, f_V\) are auxiliary. At iteration 1, \(\lambda_1=1\), so \(\lambda_1 f^{self}\) fully dominates the cross-modal term and the model essentially performs intra-modal self-refinement on all three pathways; the Decomposer still projects \(f_L\) into the spaces of \(f_A\) and \(f_V\) and produces residuals, but with unstable features those residuals are mostly noise and the denoiser removes a large share. At iterations 2 and 3, \(\lambda_t\) drops to roughly 0.67 and 0.33, cross-modal attention takes over, the projection component grows round by round while the residual becomes cleaner, and \(\mathcal{L}_{phase}\) drives projection and residual toward orthogonality. At iteration 4, \(\lambda_4=0\): that step's output is decided entirely by cross-modal features, and the auxiliary modalities have by then been pulled close to the dominant modality's feature direction; the three final representations are concatenated and classified into a sentiment prediction. ⚠️ These \(\lambda\) values depend on the linear schedule of Eq. (16), whose layout is corrupted in the cache — verify against the original. Compared with a one-shot fusion baseline, the same model parameters see a completely different fusion depth at step 1 and step 4, which is precisely what the progressive design produces.

Loss & Training

The total objective has three terms: cross-entropy for sentiment classification \(\mathcal{L}_{task}\) on the final concatenated representation, the per-modality head cross-entropy \(\mathcal{L}_{uni}\) from Eq. (2) (which keeps each modality's unimodal representation discriminative), and the Decomposer's phase consistency loss \(\mathcal{L}_{phase}\):

\[\mathcal{L}_{total} = \mathcal{L}_{task} + \eta_1 \mathcal{L}_{uni} + \eta_2 \mathcal{L}_{phase}\]

Key hyper-parameters: balance coefficient \(\gamma=0.8\), loss weights \(\eta_1=0.5\) and \(\eta_2=0.1\), iteration count \(T=4\). Language uses 300-d GloVe embeddings and 768-d BERT-base-uncased hidden states, visual uses 35-d Facet facial action units, and acoustic uses 74-d COVAREP descriptors. Missing entries are zero-filled; during training the seed changes every epoch so missing patterns are resampled while the missing rate stays constant; testing follows the CorrKD setup and all results are averaged over five random seeds. ⚠️ The loss ablation section calls \(\mathcal{L}_{uni}\) a "feature uniformity loss," whereas Eq. (2) defines it as per-modality cross-entropy; the two descriptions are inconsistent, so this note follows Eq. (2) and the precise meaning should be checked against the original.

Key Experimental Results

Main Results

F1 under inter-modality missingness (whole modality absent) on CMU-MOSI / CMU-MOSEI / SIMS:

Dataset	Setting	Prev. SOTA	PRLF	Gain
MOSI	average over six missing combinations	HRLF 76.74	77.02 (77.11 in the ablation tables ⚠️)	+0.28
MOSI	full modality	EMOE 85.40	85.78	+0.38
MOSEI	average over six missing combinations	HRLF 75.63	76.24	+0.61
MOSEI	full modality	EMOE 85.30	85.44	+0.14
SIMS	average over six missing combinations	LNLN 80.86	81.19	+0.33

Not every single-modality and bimodal cell is a win, although the paper's prose claims consistent superiority across all settings — the tables should be read carefully. On MOSI, vision-only {v} has HRLF at 64.59 versus PRLF at 64.05 (−0.54), and {l,a} has UMDF at 85.63 versus PRLF at 84.98 (−0.65); on MOSEI, vision-only {v} has HRLF at 64.90 versus PRLF at 63.67 (−1.23); on SIMS, acoustic-only {a} (81.74 vs LNLN 81.91) and vision-only {v} (81.63 vs 81.91) are marginally lower as well. PRLF wins the remaining cells and takes both the average and the full-modality settings, so its advantage comes mainly from the missing-modality-plus-collaboration regime rather than from a stronger unimodal encoder.

Ablation Study

Module, loss, and iteration-step ablations on MOSI under inter-modality missingness (average F1, reference PRLF at \(T=4\) = 77.11):

Config	Average F1	Change
PRLF (Step = 4)	77.11	—
w/o CMI (Fisher information only)	76.12	−0.99
w/o FIMI (confidence only)	76.09	−1.02
w/o AMRE (neither signal)	74.96	−2.15
w/o ProgInteract (PI)	71.32	−5.79
w/o \(\mathcal{L}_{uni}\)	75.73	−1.38
w/o \(\mathcal{L}_{phase}\)	76.63	−0.48
Step = 1	72.29	−4.82
Step = 2	74.72	−2.39
Step = 3	76.49	−0.62
Step = 5	75.80	−1.31

Iteration-step ablation under intra-modality missingness (random frame-level feature dropping at rate \(p\)), F1 on MOSI:

Iteration	p=0	p=0.3	p=0.5	p=0.7	p=0.9
Step = 1	81.14	72.25	69.58	60.73	55.37
Step = 2	83.08	73.88	70.65	62.06	56.39
Step = 3	85.21	75.28	71.97	64.62	57.59
Step = 4	85.78	75.89	73.05	65.31	58.64
Step = 5	84.49	75.19	71.24	64.16	57.48

Key Findings

• ProgInteract carries the method: removing it costs 5.79 average F1 and drops the hardest {a,v} setting from 76.03 to 68.49 (−7.54), whereas removing all of AMRE costs only 2.15. Progressive cross-modal alignment delivers the bulk of the gain; reliability estimation is better understood as the precondition that makes alignment happen against the right target.
• CMI and FIMI are complementary, with FIMI slightly ahead: removing FIMI alone (−1.02) hurts a little more than removing CMI alone (−0.99), and removing both (−2.15) costs more than half the sum of the two individual drops, matching the "confidence early, Fisher information late" design.
• The iteration count has a clear optimum: both the inter- and intra-modality curves peak at \(T=4\) and fall back at \(T=5\) (77.11 → 75.80 on MOSI). Too few steps (\(T=1\), only 72.29) throws away the schedule; too many over-interact and hurt generalization.
• The advantage grows with the missing rate: sweeping \(p\) from 0 to 0.9 degrades every method, but PRLF degrades slowest and still reaches F1 60 on MOSI and 70 on MOSEI at \(p=0.9\). Averaged over \(p\), PRLF / EMOE / LNLN score 72.48 / 71.21 / 71.25 on MOSI, 78.82 / 74.60 / 75.65 on MOSEI, and 75.04 / 71.25 / 73.13 on SIMS. Purely fusion-oriented methods like EMOE do well at low rates but collapse quickly, while missing-data methods like LNLN are steadier yet still behind PRLF.
• Feature distributions are tighter: on t-SNE the clustering variance is 1.205 with AMRE versus 1.367 without it, and dropping PI visibly blurs the class boundaries — AMRE suppresses noise from unreliable modalities while PI strengthens cross-modal semantic alignment.

Highlights & Insights

• Using the FIM trace as a probe for "how much effective information this modality still carries", backed by a gradient-decomposition argument for why losing key frames collapses \(\mathrm{Tr}(F)\). It is per-sample and forward-computable in approximation, and it is complementary to confidence precisely in the regimes where confidence fails, so a single dynamic weight stitches the two failure intervals together.
• "Progressive" is defined over fusion depth rather than data difficulty: no curriculum, no change to the missing rate — just a linear decay of \(\lambda_t\) over iterations that keeps early training away from noise in missing modalities. This trick transfers directly to any multimodal fusion with possibly corrupted inputs, at essentially zero cost.
• Turning "align, but keep the complement" into an optimizable objective via orthogonality: the projection carries direction, the residual carries unique content, and \(\mathcal{L}_{phase}\) penalizes the squared inner product. Instead of hoping attention learns alignment implicitly, this makes the degree of alignment a controllable knob — reusable for cross-modal retrieval or multi-view clustering.
• Resampling missing patterns every epoch while keeping the missing rate fixed amounts to free data augmentation over missing-modality configurations, which plausibly explains why PRLF holds up across all six missing combinations and the full-modality setting.

Limitations & Future Work

• No reconstruction means information cannot be recovered. Missing entries are zero-filled, so the model can only mitigate damage within the range of "weight the available evidence + align directions"; when key frames genuinely vanish, the ceiling is set by the surviving modalities. Generative completion and PRLF are not mutually exclusive — attaching a completion module ahead of AMRE is an obvious next step.
• The per-sample FIM overhead is unreported. Computing squared gradient norms per sample needs extra backward passes, yet the paper gives no training-time or memory comparison against baselines, which directly affects practical value.
• Key equations are unreadable in the paper. The fusion weight \(w^{(t,i)}=\sigma(\cdot[\Delta])\), the time coefficient \(\lambda_t\), and the self-refinement equation (13) are corrupted in the cached text; this note reconstructs them from context and flags them, so reproduction requires the original.
• The "progressive" semantics are limited: the missing rate is constant during training, \(\lambda_t\) is a hand-set linear decay, and \(T\) is a grid-searched constant (\(T=5\) already degrades). There is no adaptive stopping criterion and no coupling between missing severity and interaction strength (e.g. heavier missingness might warrant releasing cross-modal interaction later).
• Evaluation caveats: the main table reports only F1, with ACC-7 / ACC-5 / ACC-2 / MAE relegated to the supplementary material; the test protocol follows CorrKD, so cross-paper comparisons must check whether the missing-pattern sampling matches. Also, Table 1 writes the MOSI average as 77.02 while Tables 4/5/6 write 77.11 for identical per-setting numbers (arithmetic on those numbers gives 77.11), which looks like a typo in the main table — verify against the original before quoting.

Related Work & Insights

• vs generative methods (DiCMoR / MRAN): they reconstruct the features or semantics of missing modalities from the observed ones, hoping to recover the ground truth; PRLF reconstructs nothing and instead weights modality value at fusion time and aligns feature directions. Reconstruction can recover information when the missing rate is moderate, but its own error gets amplified downstream; PRLF takes on no reconstruction risk but genuinely loses information. Combining the two — complete first, then let AMRE judge how trustworthy the completion is — is a natural next step.
• vs distillation-based methods (UMDF / CorrKD / HRLF): they steer a missing-modality student with a teacher trained on complete modalities, so the knowledge comes from an external model; PRLF is a single model that estimates reliability online from confidence plus Fisher information and uses "who is reliable" directly as the interaction weight. The relative weakness is the absence of a teacher's complete-modality prior, which shows up in single-modality settings such as vision-only, where HRLF is ahead (MOSI −0.54, MOSEI −1.23).
• vs traditional fusion methods (MISA / MMIM / Self-MM / TETFN): these assume complete modalities and degrade badly under missingness (TETFN scores only 39.32 in the MOSI acoustic-only setting). PRLF differs not in the fusion operator itself but in making a "whom to trust" decision before fusing.
• vs EMOE: also a dynamic-expert fusion idea, EMOE is strong at low missing rates and degrades quickly as the rate rises, whereas PRLF's degradation curve is the flattest — because PRLF writes reliability estimation explicitly into both the interaction weights and the iteration schedule rather than leaving adaptation implicit.

Rating

• Novelty: ⭐⭐⭐⭐ [Using Fisher information and classification confidence as complementary modality-reliability signals, plus a linear schedule that turns fusion depth into gradual unfreezing, is an uncommon angle; the overall framework is still a combination of reliability weighting and cross-modal alignment]
• Experimental Thoroughness: ⭐⭐⭐⭐ [Three benchmarks × inter-/intra-modality missingness × missing-rate sweeps, plus module, loss, and step ablations and t-SNE and importance visualizations; the deductions are the F1-only main table, the missing FIM overhead analysis, and one average-value typo]
• Writing Quality: ⭐⭐⭐ [The motivation chain (confidence fails → introduce FIM → fuse the two complementarily) is clear and Fig. 3 is solid evidence, but many equations are corrupted, the loss naming is inconsistent, and the prose claim of consistent superiority conflicts with the tables]
• Value: ⭐⭐⭐⭐ [For the real problem of missing modalities contaminating intact ones, the solution needs neither reconstruction nor a teacher model, keeps training cost manageable, and both the progressive schedule and the orthogonality alignment transfer easily to other multimodal tasks]