Decoding Multimodal Causality: End-to-End Multimodal Mediation Pathways Inference

Conference: ECCV2026
Official Paper: 3479
PDF: Full Paper
Area: Medical Imaging
Keywords: Causal mediation analysis, multimodal learning, probabilistic causal graphs, path stability, uncertainty quantification

TL;DR

Rather than fixing one causal graph, MM-SIGMA learns an ensemble in multimodal latent space, selects stable mediation pathways, and aggregates effects and variances, achieving 0.74 pathway recovery and 0.94 coverage for nominal 95% intervals in the paper's end-to-end synthetic evaluation; its clinical pathways remain observational findings contingent on identification assumptions.

Background & Motivation

Multimodal medical prediction usually asks which information helps predict risk, whereas causal mediation analysis asks through which mechanisms an exposure changes an outcome. Sleep measurements, retinal vascular phenotypes, body fat, and cardiovascular measurements come from different physiological systems. Better prediction from their joint use does not establish a directed mediation chain. Traditional mediation analysis often specifies pathways before estimating direct and indirect effects, limiting its usefulness when the relevant cross-system connections are not yet known.

Concatenating image features with tabular variables does not resolve this problem. A conventional encoder may retain reconstruction information while losing conditional dependencies needed for mediation analysis. A structure learner that returns only its best graph then treats uncertain edges as known facts. Narrow downstream confidence intervals may reflect omitted structural uncertainty rather than stronger evidence. This work connects representation learning, structure discovery, pathway filtering, and effect estimation, with a primary application to HPP medical data containing fundus images.

Core Idea: learn an ensemble of plausible causal graphs from latent representations that retain variable correspondence, analyze only complete pathways that recur across the ensemble, and include both within-graph estimation error and between-graph effect disagreement in the final uncertainty.

Method

Overall Architecture

Inputs include exposures, outcomes, candidate mediators, confounding covariates, and tabular and image observations. MM-SIGMA proceeds through Causally Constrained Representation, Probabilistic Structure Discovery, Path Stability Filtering, and Effect Estimation and Aggregation, returning candidate cross-modal pathways with effects and intervals; it is not a prediction network that turns images directly into clinical causal conclusions.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Tables, images,<br/>exposures and outcomes"] --> B["Causally Constrained Representation"]
    B --> C["Probabilistic Structure Discovery"]
    C --> D["Path Stability Filtering"]
    D --> E["Effect Estimation and Aggregation"]
    E --> F["Candidate mediation pathways<br/>effects and intervals"]

The estimand must first be specified. The natural direct effect compares outcomes under different exposures while holding mediators at their reference-exposure state. The natural indirect effect holds exposure fixed in the outcome model and measures transmission through a change in the mediator distribution. Writing the structured and image mediators jointly as \(M\), the paper's natural indirect effect can be expressed compactly as:

\[ \mathrm{CM\!\!\!-NIE}(a,a^*)=\mathbb{E}\left[Y(a,M(a))-Y(a,M(a^*))\right]. \]

Here \(a\) is the target exposure, \(a^*\) is the reference, and \(M(a)\) denotes the potential mediators under the corresponding exposure. Computing a model-based counterfactual difference does not automatically make it causal: identification requires consistency, ignorability, positivity, and sufficiency of the image latent representation for relevant outcome information. Structure discovery also relies on causal Markov and faithfulness assumptions. These are prerequisites, not properties automatically obtained after training.

Key Designs

1. Causally Constrained Representation: retain variable identity and dependencies during compression

Each structured variable has a separate encoder rather than the whole table being compressed into an opaque vector; images use a CNN encoder. A structured latent variable therefore still corresponds to an original sleep or body-fat measurement, giving graph edges traceable endpoints. Image-side interpretation is restricted to relevant features captured by the encoder: the image cannot be uniquely inverted, and an image latent node does not establish a specific pixel-level pathological mechanism.

Beyond reconstruction and KL terms, training includes cross-modal mutual-information alignment and conditional-independence constraints. The latter penalize empirical conditional mutual information for specified variable triplets to retain the intended conditional independencies in latent space. Triplets come from biomedical knowledge and kernel conditional-independence tests on a held-out split; the paper uses a Bonferroni-corrected \(p>0.1\) selection rule. Importantly, failure to reject the absence of dependence does not prove independence. The paper's complete structure-preservation claim additionally requires a sufficient separating set for the true graph; that condition cannot be omitted.

2. Probabilistic Structure Discovery: retain plausible graphs instead of committing too early

Flow-SEM models each latent variable with a conditional mean and noise scale determined by its parents, using independent Gaussian noise. Training combines data likelihood, a continuous acyclicity constraint, and sparsity regularization. It therefore searches for directed acyclic dependencies rather than symmetric similarities. A sigmoid transforms continuous edge weights into inclusion scores for Bernoulli sampling, followed by greedy cycle removal to obtain candidate directed acyclic graphs (DAGs).

The ensemble matters because a clinical pathway may exist under some plausible structures but disappear under others; displaying only the best graph hides that disagreement. However, the edge scores are variational approximations, and cycle removal changes the sampling distribution: this is not an exact Bayesian posterior. The paper provides conditional consistency arguments involving model identifiability and bounded in-degree. Finite-sample directional certainty still requires empirical assessment and is not guaranteed by calling the model probabilistic.

3. Path Stability Filtering: check whether a complete chain holds jointly

The method enumerates simple directed exposure-to-outcome paths in valid DAGs and computes the Cross-Modal Path Stability Score, CM-PSS. This is the fraction of valid candidate graphs containing all edges of a pathway simultaneously, rather than a chain assembled from individually confident edges. Following the paper's normalization by valid graphs, it can be written as:

\[ \mathrm{CM\!\!\!-PSS}(\pi)=\frac{1}{N_{\mathrm{valid}}}\sum_{G\in\mathcal{G}_{\mathrm{valid}}}\mathbf{1}\{\pi\subseteq G\}. \]

Here \(\mathcal{G}_{\mathrm{valid}}\) is the valid candidate-graph ensemble and \(N_{\mathrm{valid}}\) its size. Only paths meeting threshold \(\tau\) are retained and mapped to original variables through encoder correspondence. This score measures pathway stability conditional on the model and candidate-graph distribution. It is not effect magnitude, clinical credibility, or a direct guarantee of frequentist false-discovery-rate control.

4. Effect Estimation and Aggregation: distinguish estimands before combining within- and between-graph error

For each graph supporting a pathway, the method uses its parent sets to fit exposure propensity, mediator distributions, and outcome regression. Cross-fitting reduces bias from training and evaluation on the same observations. Three-node exposure–single-mediator–outcome paths use extended efficient influence functions (EIFs) to estimate natural effects. Longer paths switch to plug-in estimation and an interventional indirect effect to avoid accumulating cross-world independence assumptions. Consequently, although the CM-NIE name is retained for long paths, its estimand is not identical to the single-mediator natural indirect effect.

The final aggregation uses equally weighted supporting graphs. The point estimate averages graph-specific effects; the variance retains uncertainty within each graph and adds dispersion among graph-specific point estimates:

\[ \widehat V_{\mathrm{total}}=\widehat V_{\mathrm{within}}+\widehat V_{\mathrm{between}}. \]

The first term averages supporting-graph estimation variances. The second averages squared deviations of graph-specific effects from the aggregate mean. Intervals then use a normal approximation. This BMA implementation uses equal weights rather than computing a marginal likelihood for every graph. The explicit formula accounts for within-graph estimation and between-graph differences; the term “end-to-end” does not establish exact calibration of every finite-sample error from representation learning or pathway selection.

Loss & Training

Representation learning combines reconstruction, KL, causal constraints, and cross-modal alignment; structure learning combines negative log-likelihood, acyclicity, and sparsity. The source mixes log-likelihood and “loss” terminology in its representation objective, leaving the sign convention unclear. Reproduction should verify the optimization convention; this note does not transcribe the ambiguous total objective as a directly executable minimization formula.

Synthetic data embed structured variables in images through rotation, brightness, contrast, noise, and label association, with parallel, serial, and hybrid pathways. Experiments generate 10 independent datasets per configuration and report 50 Monte Carlo repetitions; hyperparameters use grid search and 5-fold cross-validation. Baselines share the multimodal encoder input layer, which does not establish identical internal representations or complete training procedures across methods.

Key Experimental Results

Main Results

Structure-discovery metrics include adjacency F1, orientation F1, and cross-modal edge F1, all higher-is-better. Pathway recovery PR is the number of recovered true pathways divided by the total number of true pathways; it does not directly penalize additional false-positive paths. Lower bias is better, whereas 95% interval coverage should approach 0.95 rather than be maximized. The table separates structure evaluation from source Table 2 and end-to-end mediation evaluation from Table 5; these are not the same task.

Evaluation setting	Metric and direction	MM-SIGMA	Comparator	Comparator value
Synthetic structure discovery, Table 2	Adjacency F1 ↑	0.487 ± 0.028	MM-DECI	0.412 ± 0.035
Synthetic structure discovery, Table 2	Orientation F1 ↑	0.421 ± 0.032	MM-DECI	0.348 ± 0.039
Synthetic structure discovery, Table 2	Cross-modal edge F1 ↑	0.398 ± 0.035	MM-DECI	0.315 ± 0.048
End-to-end mediation, Table 5	Pathway recovery ↑	0.74 ± 0.04	MM-DECI+EIF	0.64 ± 0.05
End-to-end mediation, Table 5	NDE bias ↓	0.012 ± 0.008	MM-DECI+EIF	0.039 ± 0.019
End-to-end mediation, Table 5	NIE bias ↓	0.015 ± 0.009	MM-DECI+EIF	0.037 ± 0.020
End-to-end mediation, Table 5	95% interval coverage, target 0.95	0.94 ± 0.02	MM-DECI+EIF	0.90 ± 0.03

Cross-modal edge F1 improves by an absolute 0.083, while end-to-end pathway recovery increases by 0.10, or 10 percentage points. The readable source captions do not clearly and consistently define the statistical meaning of the “±” terms. They are retained without relabeling them as standard deviations or confidence-interval half-widths.

HPP provides real medical observational data: sleep monitoring covers 6,366 adults and 16,812 nights, while fundus images come from 8,467 individuals. These are modality-specific collection sizes, not the final number of fully paired analysis participants. Analyses adjust for age, sex, BMI, and smoking status, with MICE for missing values. The established sleep HRV→PWV→carotid IMT pathway has reported NDE −0.089 ± 0.028 and NIE −0.156 ± 0.034; the candidate retinal features→body fat→blood pressure→IMT pathway has indirect effect −0.067 ± 0.022. Signs depend on exposure and outcome coding, so these observational estimates cannot directly be interpreted as treatment benefits.

Ablation Study

All values below come from Table 5, with default threshold \(\tau=0.5\). Recovery, NIE bias, and interval coverage respectively measure how many true pathways are recovered, how accurate effect estimation is, and how well uncertainty is calibrated.

Config	Pathway recovery ↑	NIE bias ↓	95% interval coverage, target 0.95
Full model	0.74 ± 0.04	0.015 ± 0.009	0.94 ± 0.02
Without causal constraints	0.61 ± 0.06	0.034 ± 0.015	0.90 ± 0.03
Without cross-modal alignment	0.65 ± 0.05	0.029 ± 0.013	0.91 ± 0.03
Without BMA	0.71 ± 0.05	0.021 ± 0.011	0.87 ± 0.04
Threshold 0.3	0.79 ± 0.05	0.027 ± 0.012	0.91 ± 0.03
Threshold 0.7	0.68 ± 0.05	0.018 ± 0.010	0.93 ± 0.02

Key Findings

• Removing causal constraints reduces recovery from 0.74 to 0.61, a loss of 13 percentage points; removing BMA reduces coverage from 0.94 to 0.87, a loss of 7 percentage points. These components more directly affect pathway retention and interval calibration, respectively.
• Lowering the threshold to 0.3 improves recovery but increases NIE bias. Recovery alone favors permissive filtering and must be interpreted alongside bias and coverage.
• Source Table 3 reports recovery of 0.487 for medium samples, 0.512 for large samples, and 0.412 under strong noise. These belong to a separate condition analysis and should neither be pooled directly with Table 5's 0.74 nor automatically treated as contradictory results.

Highlights & Insights

• Pathways are a more relevant uncertainty unit than individual edges for mediation analysis. Individually stable edges need not jointly form a chain in the same candidate graph.
• The evaluation separates mechanism recovery from reliability of effect intervals. The BMA ablation shows that ignoring structural disagreement can substantially reduce coverage even when pathway recovery remains reasonable.
• Variable-wise encoding gives the structured side explicit interpretive targets. On the image side, the interpretation remains a summary of relevant features rather than a unique reconstruction of an image-level causal mechanism.

Limitations & Future Work

• Identification relies on ignorability, latent sufficiency, and other conditions that observational data cannot fully verify. Clinical association studies motivate the analysis but do not establish those conditions or validate discovered pathways.
• Longer paths change the indirect-effect estimand and use plug-in estimation. Reproduction and application must distinguish natural and interventional effects instead of retaining only the shared CM-NIE label.
• Candidate graphs use approximate sampling and greedy cycle removal, followed by equal-weight averaging; actual coverage of nominal 95% intervals remains 0.94. The paper also notes potential underestimation of between-graph variance when few graphs support a pathway.
• Synthetic images use controlled transformations, while HPP lacks an observable ground-truth causal graph, so recovery of real high-dimensional image mechanisms is not established. Final paired sample sizes, complete hyperparameters, and definitions of some reported uncertainty terms require clearer reporting.

Related Work & Insights

• vs MM-DECI+EIF: The two-stage pipeline learns structure before estimating effects, whereas this method uses candidate structures in pathway filtering and effect aggregation. Table 5 supports gains in recovery and coverage, but they should not all be attributed to a single component.
• vs MM-NOTEARS+DeepMed: Continuous acyclic structure learning itself is not new here. The distinction is its combination with cross-modal constrained representations, complete-path stability, and propagation of structural uncertainty.
• vs traditional mediation analysis and CausalMixNet: This work emphasizes discovering pathways without fixing them in advance. Automated search expands the candidate space, but a stability threshold does not remove the need for causal identification, post-selection inference, or external validation.

Rating

• Novelty: 4/5. Unifies multimodal pathway discovery and effect uncertainty, with innovation primarily in the combination and problem formulation.
• Experimental Thoroughness: 3/5. Includes structure, effect, and ablation evaluations, but real data lack causal ground truth and reporting details remain incomplete.
• Writing Quality: 3/5. The pipeline is clear, but natural versus interventional effect naming, objective signs, and statistical uncertainty reporting need greater precision.
• Value: 4/5. A useful methodological reference for observational mechanism analysis involving images, not yet a clinically validated causal decision tool.