Skip to content

Towards Reliable Medical Large Vision-Language Models via Counterfactual Preference Optimization

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/zxgapollo/CoMedPO
Area: Multimodal VLM
Keywords: Medical LVLMs, Counterfactual Preference Optimization, Modality Bias, Causal Inference, Hallucination Mitigation

TL;DR

To mitigate clinical hallucinations caused by non-diagnostic background shortcuts and disease co-occurrence bias in Medical Large Vision-Language Models (Med-LVLMs), this paper proposes Counterfactual Medical Preference Optimization (CoMedPO), which uses causal mediation analysis to eliminate harmful background direct effects while preserving essential anatomical indirect effects without incurring any inference overhead.

Background & Motivation

Medical Large Vision-Language Models (Med-LVLMs) hold great promise for clinical visual question answering (Med-VQA), automated radiology reporting, and multimodal decision support. However, factual reliability remains a critical bottleneck in high-stakes clinical workflows. Unlike natural images, medical pathology often resides in subtle, highly localized lesion areas that are easily overwhelmed by coarse cross-modal alignment. Consequently, models frequently succumb to modality bias, exploiting spurious statistical shortcuts—such as patient demographics, imaging hardware artifacts, and background chest contours—rather than grounding their decisions on genuine lesion evidence.

Recent efforts have attempted to address hallucination by adopting Direct Preference Optimization (DPO) to align post-SFT models using paired clinical feedback. Nevertheless, standard DPO frameworks inherit dataset biases from biased reference policies, failing to decouple true clinical causality from dataset artifacts. Furthermore, models suffer from pathology co-occurrence bias; for example, frequent co-occurrence of pleural effusion and infiltration often prompts models to hallucinate one condition merely upon detecting the other. While recent methods like MMedPO attempt to simulate interventions by injecting local noise into lesion regions, this unstructured perturbation breaks the beneficial joint interplay between lesions and their surrounding anatomical structures. In medical imaging, background context (such as pleural line continuity and cardiac silhouette symmetry) provides indispensable spatial priors for differential diagnosis.

To resolve this conflict, the key angle of attack is to ground multimodal preference alignment within a structural causal model (SCM) to analytically separate causal pathways. Core idea: formulate multimodal alignment as a counterfactual mediation contrast that decouples the harmful natural direct effect (NDE) transmitted through background shortcuts from the beneficial total indirect effect (TIE) of lesion-background dependencies, guaranteeing convergence toward a causal-optimal policy even with a biased reference model.

Method

Overall Architecture

CoMedPO aims to train an unbiased diagnostic policy from an inherently biased SFT reference model \(\pi_{\text{ref}}\). The pipeline begins by extracting lesion regions from the input image using a lightweight localization model (Med-SAM), creating a background-only image by zeroing out lesion masks and a perturbed image via Gaussian noise injection. These images are concatenated into factual and counterfactual composite pairs under a unified comparative prompt to perform controlled interventions. Within a Structural Causal Model, rewards are decomposed into direct and indirect effects, yielding a Total Indirect Effect (TIE) causal reward. The counterfactual DPO loss (\(\mathcal{L}_{\text{cDPO}}\)) is derived using policy-reward duality and optimized jointly with standard DPO (\(\mathcal{L}_{\text{DPO}}\)). At test time, the model operates strictly on standard single-image inputs, eliminating all preprocessing and multi-image overhead.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Medical Input<br/>Original medical image I and clinical prompt T"] --> B["Causal Composite Images & Contrastive Query<br/>Segment lesion via Med-SAM to yield background Ib & perturbed Inull<br/>Construct factual pair (Ib⊕I) and counterfactual pair (Ib⊕Inull)"]
    B --> C["Causal Mediation & Total Indirect Effect<br/>Model paths I→B→R and B/L→E→R via SCM<br/>Define causal reward as TIE: rc(x,y) = Rb,e - Rb,e*"]
    C --> D["Symmetric Counterfactual Preference Optimization<br/>Derive symmetric log-ratio contrasts in Bradley-Terry model<br/>Jointly optimize L_DPO and L_cDPO"]
    D --> E["Unbiased Multimodal Medical Policy<br/>Standard single-image inference / zero background shortcut reliance"]

Key Designs

1. Causal Composite Images & Contrastive Query: executing controlled lesion interventions

Randomized controlled trials (RCTs) that independently vary lesion and background factors are clinically and ethically infeasible. To simulate counterfactual intervention, CoMedPO employs Med-SAM to detect and segment lesion regions from image \(I\). Masked lesion pixels are zero-filled to generate a background-only image \(I_b\) (representing background variable \(B=b\)), while lesion regions are perturbed with Gaussian noise to yield counterfactual lesion image \(I_{\text{null}}\). These are combined into factual composite image \((I_b \oplus I)\) and counterfactual composite image \((I_b \oplus I_{\text{null}})\). A standardized clinical query \(T'\) is prepended ("Both images belong to the same patient. Please analyze each and then provide a joint clinical conclusion"), compelling the model to contrast outcomes under invariant anatomical background context.

2. Causal Mediation & Total Indirect Effect: isolating direct bias while preserving anatomical context

Existing models map visual inputs directly to rewards, allowing the shortcut link \(I \to B \to R\) to transmit imaging device signatures and non-diagnostic background priors directly into the policy. Meanwhile, the desirable link \(B/L \to E \to R\) carrying anatomical symmetry and landmark constraints is confounded. CoMedPO formulates an SCM comprising image \(I\), background \(B\), lesion \(L\), joint multimodal representation \(E\), and reward \(R\). Defining Total Effect as \(\mathrm{TE} = R_{b,e}(I) - R_{b^*,e^*}(I)\) and the Natural Direct Effect as \(\mathrm{NDE} = R_{b,e^*}(I) - R_{b^*,e^*}(I)\), subtracting NDE from TE isolates the Total Indirect Effect (TIE): $\(r_c(x, y) = \mathrm{TIE} = R_{b,e}(x, y) - R_{b,e^*}(x, y)\)$ This formulation cancels the direct background shortcut while preserving the clinically indispensable joint representation of lesion morphology conditioned on anatomical background.

3. Symmetric Counterfactual Preference Optimization: debiasing reference policies with theoretical guarantees

Substituting the causal TIE reward into the Bradley-Terry preference model reveals a symmetric contrast structure across response pairs \((y_w, y_l)\): $\(r_c(x, y_w) - r_c(x, y_l) = \left[R_{b,e}(x, y_w) - R_{b,e}(x, y_l)\right] - \left[R_{b,e^*}(x, y_w) - R_{b,e^*}(x, y_l)\right]\)$ Symmetry is crucial: correcting only preferred response \(y_w\) leaves residual background bias in \(y_l\), contaminating pairwise gradients. Using policy-reward duality to eliminate explicit reward models, the counterfactual DPO loss is defined over factual input \(x_c = (I_b \oplus I, T')\) and counterfactual input \(x_c^* = (I_b \oplus I_{\text{null}}, T')\): $\(\mathcal{L}_{\text{cDPO}} = -\mathbb{E}_{(x_c, x_c^*, y_w, y_l) \sim \mathcal{D}_f}\left[\log \sigma\left(\eta^{-1} \Lambda_{\text{cDPO}}\right)\right]\)$ where $\(\Lambda_{\text{cDPO}} = \left[\log \frac{\pi_\theta(y_w \mid x_c)}{\pi_{\text{ref}}(y_w \mid x_c)} - \log \frac{\pi_\theta(y_w \mid x_c^*)}{\pi_{\text{ref}}(y_w \mid x_c^*)}\right] - \left[\log \frac{\pi_\theta(y_l \mid x_c)}{\pi_{\text{ref}}(y_l \mid x_c)} - \log \frac{\pi_\theta(y_l \mid x_c^*)}{\pi_{\text{ref}}(y_l \mid x_c^*)}\right]\)$ The authors prove that under bounded bias \(\delta(x, y)\) in \(\pi_{\text{ref}}\), optimizing \(\mathcal{L}_{\text{cDPO}}\) guarantees monotonic convergence toward the causal-optimal policy \(\pi_c^\star(y \mid x) \propto \pi_{\text{ref}}(y \mid x) \exp(\eta r_c(x, y))\) in KL divergence.

A Worked Example

Consider an illustrative chest radiograph (CXR): 1. Clinical Case: An image \(I\) showing faint, localized opacity at the left lung base (mild pleural effusion), accompanied by bilateral hyperlucency characteristic of chronic emphysema. 2. Baseline Failure: Standard Med-LVLMs exploit the prominent hyperlucency background shortcut, hallucinating extensive pulmonary infiltration and predicting positive findings even on lesion-free background patches. 3. CoMedPO Intervention: - Med-SAM isolates the small basal effusion, generating zeroed background \(I_b\) and Gaussian-noised counterfactual \(I_{\text{null}}\). - Composite pairs \((I_b \oplus I)\) and \((I_b \oplus I_{\text{null}})\) are presented under prompt \(T'\). - If the model assigns high probability to "pulmonary infiltration" on the counterfactual pair \((I_b \oplus I_{\text{null}})\) where the true lesion is absent, this log-probability is subtracted during \(\Lambda_{\text{cDPO}}\) calculation, severely penalizing reliance on emphysema background cues. 4. Inference Execution: Deployed with standard single-image input \(I\) and direct prompt, the debiased policy ignores emphysema context and accurately reports isolated left pleural effusion without hallucinated infiltration.

Loss & Training

To preserve standard instruction-following and descriptive ability on unaltered clinical inputs, the total training objective combines standard DPO and counterfactual cDPO: $\(\mathcal{L}_{\text{CoMedPO}} = \mathcal{L}_{\text{DPO}} + \lambda \mathcal{L}_{\text{cDPO}}\)$ where \(\lambda = 1.0\) is chosen based on empirical ablations. Training is implemented via LoRA (rank=128, scaling \(\alpha=256\), dropout=0.05) on 8 NVIDIA A100 GPUs for 3 epochs. The learning rate is set to 1e-6 with cosine decay and a 0.03 warmup ratio, using per-device batch size 2.

Key Experimental Results

Main Results

Evaluations were conducted across four medical benchmarks: Med-VQA (SLAKE, VQA-RAD) and radiology report generation (IU-XRay, MIMIC-CXR), using LLaVA-Med-1.5-mistral-7B as the base model under both standard alignment and SFT-enhanced regimes.

Model / Pipeline SLAKE Open SLAKE Closed VQA-RAD Open VQA-RAD Closed IU-XRay BLEU IU-XRay ROUGE-L IU-XRay METEOR MIMIC-CXR BLEU MIMIC-CXR ROUGE-L MIMIC-CXR METEOR
LLaVA-Med v1.5 47.54 52.99 29.50 64.54 14.56 10.31 10.95 10.25 9.38 7.71
+ DPO 49.84 53.59 32.00 66.53 16.08 12.95 17.13 11.19 9.45 7.80
+ mDPO 51.20 60.29 35.00 68.13 18.43 22.38 26.52 11.55 9.62 7.90
+ SPPO 52.62 67.58 38.50 68.93 21.66 27.30 32.10 12.32 10.19 8.83
+ MMedPO 51.67 58.25 36.50 67.33 23.49 29.52 34.16 12.85 11.13 10.03
+ CoMedPO 53.11 71.17 40.00 71.31 23.78 32.68 35.26 12.65 11.61 10.37
+ SFT Baseline 52.62 58.01 31.50 64.14 22.75 28.86 33.66 12.39 10.21 8.75
+ SFT + DPO 53.49 64.23 33.00 64.94 23.82 29.97 34.89 12.37 10.38 9.10
+ SFT + mDPO 53.10 63.88 34.50 67.33 23.57 28.35 33.13 11.77 9.45 8.91
+ SFT + SPPO 58.57 66.99 41.00 73.31 23.78 30.05 34.78 12.87 12.32 9.45
+ SFT + MMedPO 56.60 67.82 37.00 67.73 24.00 30.13 35.17 13.28 13.22 10.20
+ SFT + CoMedPO 62.17 74.40 42.50 74.90 26.23 34.13 38.25 14.91 14.58 12.22

Ablation Study

Empirical causal validations and component ablations were conducted on VQA-RAD and IU-XRay using the SFT-enhanced model.

Experimental Variant VQA-RAD Open VQA-RAD Closed IU-XRay BLEU IU-XRay ROUGE-L IU-XRay METEOR Note
Full CoMedPO (SFT) 42.50 74.90 26.23 34.13 38.25 Standard clean inference
I. Background Shortcut Sensitivity (Lower is better)
DPO (Background-Only Input) 34.50 63.74 21.56 27.22 32.39 Relies heavily on background priors
MMedPO (Background-Only Input) 26.50 65.34 15.77 10.36 10.05 Still susceptible to background cues
CoMedPO (Background-Only Input) 22.00 53.78 13.29 9.45 10.58 Sharp accuracy drop confirms shortcut removal
II. Cross-Dataset Distribution Shift
DPO (SLAKE→RAD / MIMIC→IU) 32.50 (-0.50) 60.96 (-3.98) 19.82 (-4.01) 23.97 (-6.00) 26.89 (-8.00) Vulnerable to site distribution shift
MMedPO Shift 33.50 (-3.50) 62.15 (-5.58) 21.17 (-2.83) 27.26 (-2.87) 31.84 (-3.33) Noticeable transfer performance drop
CoMedPO Shift 40.50 (-2.00) 72.51 (-2.39) 25.32 (-0.91) 32.43 (-1.70) 36.55 (-1.70) Smallest degradation; robust generalizability
III. Loss Formulation Variants
w/o Standard DPO (\(\mathcal{L}_{\text{DPO}}\)) 29.50 57.77 14.11 10.65 10.37 Loses factual grounding anchor
Causal-weighted DPO (\(\mathcal{L}_{\text{DPO}}\)) 37.50 70.52 24.25 28.40 34.64 Sample reweighting underperforms explicit contrast
w/o Counterfactual Query 42.00 72.91 26.56 34.02 36.68 Robust to minor prompt variations
IV. Composite Input & Control Analysis
Composite Input Only (Mismatched Prompt) 37.50 69.32 22.35 28.69 32.57 Visual-prompt mismatch degrades reasoning
Composite + Matched Prompt 43.43 73.00 25.67 33.62 35.05 High score when formatting is fully aligned
Random Composite Control 37.00 67.33 21.24 27.67 31.82 Random patches fail to instantiate causal mediation

Key Findings

  • Elimination of Spurious Background Reliance: When presented solely with lesion-free backgrounds, MMedPO retains 65.34% closed-question accuracy on VQA-RAD by guessing from context, whereas CoMedPO drops sharply to 53.78% (and 9.45% ROUGE-L on IU-XRay). Lower performance under lesion-free ablation directly validates the elimination of spurious direct background effects.
  • Superior Cross-Hospital Domain Transfer: On cross-dataset transfer tests, CoMedPO shows minimal performance degradation (-2.39% on VQA-RAD Closed vs. -5.58% for MMedPO; -1.70% on IU-XRay ROUGE-L vs. -6.00% for DPO), demonstrating that mitigating site-specific background shortcuts substantially enhances domain generalization.
  • Necessity of Dual-Objective Optimization: Discarding \(\mathcal{L}_{\text{DPO}}\) causes VQA-RAD open recall to plummet from 42.50% to 29.50%, confirming that counterfactual regularization must be anchored by factual preference supervision to prevent policy collapse.

Highlights & Insights

  • Bridging Structural Causal Models and DPO: By decomposing preference rewards into Natural Direct Effects (harmful shortcuts) and Total Indirect Effects (beneficial anatomical context), CoMedPO provides a rigorous mathematical foundation for debiasing Med-LVLMs.
  • Symmetric Factual-Counterfactual Contrast: Proves that uncorrected negative responses cause background bias leakage, introducing a symmetric preference contrast formulation that purges bias from both preferred and dispreferred pairs simultaneously.
  • Zero-Cost Inference Deployment: Causal interventions and multi-image stitching are restricted entirely to training, enabling the final aligned model to run on standard single-image inputs without additional inference latency or memory overhead.

Limitations & Future Work

  • Reliance on Upstream Segmentation Quality: Generating accurate background images \(I_b\) depends on Med-SAM localization. Subtle or ill-defined lesions that escape detection can leave residual pathology in \(I_b\), slightly compromising causal contrast purity.
  • Heuristic Image-Space Interventions: Zero-filling and Gaussian noise injection are relatively simplistic intervention operators that may introduce unnatural boundary artifacts compared to realistic counterfactual generative diffusion models.
  • Future Directions: End-to-end joint optimization of lesion grounding and preference alignment, alongside generative pathology synthesis for high-fidelity counterfactual generation.
  • vs MMedPO: MMedPO relies on computationally intensive multi-agent scoring and unstructured lesion noise that disrupts anatomical context. CoMedPO formalizes an explicit causal graph, isolating harmful background shortcuts via TIE while retaining vital anatomical priors with lower computational overhead.
  • vs Standard DPO / mDPO / SPPO: Conventional preference optimization implicitly assumes unbiased datasets and inherits spurious shortcuts from the SFT reference policy. CoMedPO establishes theoretical convergence to the causal-optimal policy despite reference bias, serving as an effective, model-agnostic plug-in across diverse preference learning algorithms.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Formulates a mathematically sound causal mediation framework for multimodal medical preference optimization with rigorous convergence proofs.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Extensive validation across four clinical benchmarks, featuring dedicated background sensitivity and cross-dataset distribution shift analyses.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear causal formulation, structured methodology, and compelling empirical analysis.
  • Value: ⭐⭐⭐⭐⭐ Delivers an essential paradigm for factual grounding and trustworthy AI deployment in high-stakes clinical diagnostic domains.