Skip to content

Learning Invariant Modality Representation for Robust Multimodal Learning from a Causal Inference Perspective

Conference: ACL 2026 arXiv: 2604.18460 Code: GitHub Area: Audio & Speech Keywords: causal invariant representation, multimodal sentiment analysis, out-of-distribution generalization, feature disentanglement, virtual environments

TL;DR

This paper proposes CmIR (Causal modality Invariant Representation learning), which leverages causal inference theory to explicitly disentangle each modality into a causal invariant representation and an environment-specific spurious representation. Through an elegant objective combining invariance constraints, mutual information constraints, and reconstruction constraints, the framework ensures that invariant representations maintain stable predictive relationships across environments. CmIR achieves state-of-the-art performance on multimodal sentiment, humor, and sarcasm detection, with particularly strong results under OOD and noisy conditions.

Background & Motivation

Background: Multimodal affective computing predicts sentiment by integrating linguistic, acoustic, and visual modalities. Existing methods perform well on in-distribution test sets but tend to exploit spurious cross-modal correlations present in training data.

Limitations of Prior Work: (1) Models may over-rely on a speaker's habitual smile (a spurious visual feature) rather than semantic content; (2) Noisy modalities (e.g., background noise, low-resolution video) further corrupt spurious correlations, widening the generalization gap; (3) Existing causal approaches either lack theoretical guarantees or target specific biases (e.g., speaker bias) and are not generalizable.

Key Challenge: A general framework is needed to distinguish causal from spurious features—one that does not rely on prior assumptions about bias types or require predefined bias labels.

Goal: To establish a theoretically grounded general framework based on causal inference that disentangles each modality into causal invariant and environment-specific spurious components.

Key Insight: The defining property of causal invariant representations is predictive stability across environments—if \(P(Y|Z_m^{\text{inv}}, E=e_1) = P(Y|Z_m^{\text{inv}}, E=e_2)\), then \(Z_m^{\text{inv}}\) contains only causal features.

Core Idea: Disentanglement is achieved through a three-constraint optimization: an invariance constraint ensures consistent predictions across environments, a mutual information constraint enforces independence between the two components, and a reconstruction constraint prevents information loss. In the absence of explicit environment labels, virtual environments are simulated by injecting Gaussian noise of varying intensities into the original features.

Method

Overall Architecture

Each modality \(X_m\) is disentangled by encoder \(g_m\) into \((Z_m^{\text{inv}}, Z_m^{\text{spu}})\). Prediction is performed solely using the concatenation of invariant representations \(\{Z_m^{\text{inv}}\}_{m=1}^M\). During training, the model jointly optimizes a prediction loss and three constraint terms. Decoder \(r_m\) reconstructs the original input from both components to prevent information loss.

Key Designs

  1. Virtual Environment Construction + Invariance Constraint:

    • Function: Enables cross-environment invariance training in the absence of explicit environment labels.
    • Mechanism: Each sample is randomly assigned a virtual environment label \(e \in \{1,...,K\}\), and additive Gaussian noise of intensity \(\alpha^{(e)} = \alpha^{(1)} \cdot e\) is injected. Invariant representations are extracted from the perturbed features, and an L1-norm constraint enforces consistency across environments: \(\mathcal{R}_{\text{inv}}^{(m)} = \sum_{e_1 \neq e_2} \|Z_m^{\text{inv},(e_1)} - Z_m^{\text{inv},(e_2)}\|_1\)
    • Design Motivation: This constraint is strictly stronger than KL divergence (identical inputs necessarily yield identical output distributions), applies to both classification and regression tasks, and requires no additional unimodal predictors.

  2. Mutual Information Minimization via Orthogonality Approximation:

    • Function: Ensures statistical independence between the invariant and spurious components.
    • Mechanism: A normalized cross-correlation matrix \(\bm{C}^m = \text{Nor}(\bm{Z}_m^{\text{inv}}) \cdot \text{Nor}(\bm{Z}_m^{\text{spu}})^\top\) is computed over the batch and penalized using a weighted Frobenius norm: diagonal entries (within-sample orthogonality) are weighted by 1, and off-diagonal entries by \(\alpha < 1\).
    • Design Motivation: Direct mutual information computation is intractable; orthogonality serves as a necessary condition for independence. When used together with the invariance and reconstruction constraints, it ensures semantic separation.

  3. Reconstruction Constraint to Prevent Degenerate Solutions:

    • Function: Ensures that the two disentangled components collectively retain all information from the original input.
    • Mechanism: Decoder \(r_m\) reconstructs the original features from \((Z_m^{\text{inv}}, Z_m^{\text{spu}})\): \(\mathcal{R}_{\text{rec}}^{(m)} = \|X_m - r_m(Z_m^{\text{inv}}, Z_m^{\text{spu}})\|_2^2\)
    • Design Motivation: Without the reconstruction constraint, the model may converge to degenerate solutions in which the invariant component captures all information while the spurious component is vacuous, or vice versa.

Loss & Training

The overall objective is: \(\mathcal{L} = \mathcal{L}_{\text{pred}} + \sum_{m=1}^{M} \lambda_1 \mathcal{R}_{\text{inv}}^{(m)} + \lambda_2 \mathcal{R}_{\text{dec}}^{(m)} + \lambda_3 \mathcal{R}_{\text{rec}}^{(m)}\). Full proofs are provided for three theorems: the existence, extractability, and OOD risk advantage of invariant representations.

Key Experimental Results

Main Results

Evaluation is conducted on CMU-MOSI/MOSEI/CH-SIMS-v2 (sentiment) and UR-FUNNY (humor) and MUStARD (sarcasm). CmIR achieves state-of-the-art performance under both standard and OOD settings.

Key Findings

  • Under OOD conditions (CMU-MOSI OOD), CmIR's margin of improvement is more pronounced, confirming the generalization advantage of causal invariant representations.
  • Under noisy modality testing, CmIR degrades far less than baselines, demonstrating that isolating the spurious component renders the model more robust to noise.
  • Ablation studies confirm that all three constraints are indispensable—removing any single constraint leads to a measurable performance drop.

Highlights & Insights

  • The three-constraint framework is elegantly designed: the invariance constraint ensures causality, the orthogonality constraint ensures purity, and the reconstruction constraint ensures completeness—none can be omitted.
  • Virtual environment construction offers a practical compromise: while less precise than ground-truth environment labels, it provides a feasible solution given that most datasets lack such annotations.
  • The theoretical underpinning (three formal theorems) provides a rigorous foundation for the entire framework.

Limitations & Future Work

  • Virtual environment construction relies on an additive Gaussian noise assumption, which may not fully capture real-world distribution shifts.
  • Several hyperparameters (number of environments \(K\), noise coefficient \(\alpha\), and the three \(\lambda\) values) require careful tuning.
  • Both the encoder and decoder are simple MLPs; stronger architectures may yield further improvements.
  • vs. IRM: Invariant Risk Minimization targets single-modal settings; CmIR extends the principle to multimodal disentanglement.
  • vs. Existing Multimodal Causal Methods: Prior causal approaches target specific biases (e.g., speaker or modality bias), whereas CmIR is a general framework that operates without any assumptions about bias type.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ First systematic integration of causal invariant representation learning with feature disentanglement in multimodal affective computing.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Six datasets, standard/OOD/noisy settings, comprehensive ablation, and formal theoretical proofs.
  • Writing Quality: ⭐⭐⭐⭐⭐ Rigorous theoretical derivations and thorough experimental coverage.
  • Value: ⭐⭐⭐⭐⭐ A paradigm-level contribution to multimodal robustness research.