Proximity-Constrained Counterfactual Decoding for Hallucination-Robust Medical VQA

Conference: ECCV 2026
Paper: ECCV Official
Full Text Cache: /Users/zy/workspace/paper_cache/ECCV2026/eccv-4924.txt
Area: AI Safety
Keywords: Medical VLMs, Hallucination Robustness, Contrastive Decoding, Counterfactual Reasoning, Proximity Constraints

TL;DR

To prevent conventional global image perturbations in contrastive decoding from destroying faint diagnostic cues in medical VLMs, this paper proposes Proximity-Constrained Counterfactual Decoding (PCCD), which enforces dual global/object representation-space bounds, decouples existence and attribute interventions, and guarantees safe greedy fallback without any model retraining.

Background & Motivation

Large multimodal vision-language models (VLMs) have demonstrated substantial utility across radiological interpretation and medical visual question answering (VQA). However, driven by stubborn language priors and spurious pathology co-occurrences in training corpora, these foundation models frequently suffer from visual hallucinations. They fabricate absent pathologies, misassign anatomical laterality or severity, and mischaracterize tissue structures, posing immediate risks to diagnostic reliability and patient safety. Existing training-free contrastive decoding paradigms, such as Visual Contrastive Decoding (VCD), mitigate text priors by contrasting token prediction logits against globally corrupted visual representations (e.g., Gaussian blur or random noise).

Crucially, clinical imaging operates under an entirely different evidentiary regime than natural imagery. Decisive diagnostic markers frequently manifest as faint, localized anatomical disruptions—such as subtle apical pleural lines indicating pneumothorax, faint mucosal textures, or fine catheter tip placements. Empirical measurements reveal that conventional VCD blur degrades macro CheXpert diagnostic label agreement to 83.1% on held-out chest radiographs (CXRs), dropping pneumothorax agreement down to 0.70. This implies that nearly one in five clinical hallmarks is unintentionally eliminated by the perturbation itself. Furthermore, unconstrained attention reweighting approaches drift into diagnostic label alterations in 14.2% of evaluation cases. Reliable mitigation thus requires three essential conditions: (C1) strict preservation of localized lesion evidence, (C2) decoupled handling of heterogeneous error drivers (co-occurrence priors behind existence errors versus style biases behind attribute errors), and (C3) explicit semantic trust-region bounding on counterfactual views.

To bridge this fundamental gap, this paper formulates contrastive view admissibility via representation-space trust regions. Core idea: admit counterfactual views into contrastive decoding only when they satisfy dual global and lesion-masked proximity bounds, deploying targeted lesion-preserving context edits (OA-VCD) and latent style edits (LD-VCD) while safely falling back to greedy decoding if bounds fail.

Method

Overall Architecture

PCCD operates as a modular, training-free test-time intervention framework founded on the principle of semantic admissibility: perturbed counterfactual visual representations only influence generation logits when certified to lie within a bounded semantic distance from the original image. To tackle heterogeneous hallucination origins, PCCD constructs two distinct, complementary counterfactual branches. The Object-Aware branch (OA-VCD) utilizes cross-attention Grad-CAM heatmaps to isolate pathology regions, perturbing only the surrounding contextual tissue to suppress co-occurrence priors. The Latent-Disentangled branch (LD-VCD) performs channel-wise photometric editing in a frozen VQGAN latent space to neutralize style and illumination biases while freezing geometric structures. During autoregressive token generation, an adaptive router evaluates per-token context dependency and attribute lexicon membership to select the optimal admissible branch. When neither counterfactual passes the dual proximity gate, PCCD reverts unconditionally to greedy decoding, ensuring that mitigation never degrades below the unmitigated baseline.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Medical Image and Query Prefix"] --> B["Feature & Latent Extraction<br/>Vision Encoder & VQGAN Latent"]
    B --> C["Dual-Branch Counterfactual View Generation"]
    subgraph SG1 ["Dual-Branch Construction"]
        C --> D["Object-Aware Branch (OA-VCD)<br/>Grad-CAM Lesion Mask & Context Perturbation"]
        C --> E["Latent-Disentangled Branch (LD-VCD)<br/>Photometric Sensitivity Disentanglement & Latent Jitter"]
    end
    D --> F["Dual Proximity Gate Verification<br/>Global Bound & Object-Masked Bound"]
    E --> F
    F -->|At least one branch admissible| G["Adaptive Branch Selection & Contrastive Decoding<br/>Lexicon Matching & Contrast Margin Maximization"]
    F -->|Both branches fail gate| H["Safe Fallback Guarantee<br/>Revert to Baseline Greedy Decoding"]
    G --> I["Faithful Diagnostic Response Generation"]
    H --> I

Key Designs

1. Dual Proximity Constraints: Defining Clinical Trust Regions

To prevent unconstrained visual perturbations from corrupting critical diagnostic signals, PCCD requires any candidate counterfactual view \(v'\) to pass two geometric similarity tests in the vision encoder pooled feature space \(\phi(v)\). The global similarity constraint enforces overall clinical consistency: $\(s(v, v') = \frac{\langle \phi(v), \phi(v') \rangle}{\|\phi(v)\|_2 \|\phi(v')\|_2} \ge \omega\)$ To prevent contextual perturbations from leaking into pathology sites via receptive-field overlap, an object-masked similarity constraint is simultaneously imposed over the localized lesion mask \(M\): $\(s_{\text{obj}}(v, v') = \frac{\langle \phi(M \odot v), \phi(M \odot v') \rangle}{\|\phi(M \odot v)\|_2 \|\phi(M \odot v')\|_2} \ge \omega_{\text{obj}}\)$ With default thresholds \(\omega = \omega_{\text{obj}} = 0.995\), this dual gate guarantees under first-order logit decomposition that diagnostic drift is bounded while the contrastive margin for faithful tokens is systematically amplified. Candidate views that violate either boundary are immediately rejected.

2. Object-Aware Branch (OA-VCD): Context Prior Attenuation with Lesion Preservation

This branch targets existence hallucinations (such as falsely predicting pneumothorax or consolidation driven by anatomical background co-occurrences). The method localizes predictive evidence using Grad-CAM heatmaps across cross-attention layers, applying Gaussian smoothing (\(\sigma = 3\text{ px}\)) and temperature scaling to produce a calibrated lesion mask \(M\). The counterfactual view is synthesized as: $\(v' = M \odot v + (1 - M) \odot \Pi(v)\)$ where \(\Pi\) introduces semantic-preserving background transformations (low-frequency blur, illumination shifts, or subtle color jitter). Because pathological regions remain untouched, this intervention specifically attenuates background correlations without erasing the visual evidence required for clinical diagnosis.

3. Latent-Disentangled Branch (LD-VCD): Structure-Preserving Attribute Debias

Targeting attribute errors (such as severity grading, laterality confusion, or texture misinterpretations caused by sensor variations and windowing differences), this design manipulates a frozen VQGAN latent space \(z = E(v) \in \mathbb{R}^{C \times H' \times W'}\). Latent channels are systematically profiled along two axes: photometric sensitivity \(p_c\) measured via jitter variance, and structural necessity \(o_c\) measured via SSIM degradation under channel dropout. Ranking channels by \(q_c = p_c(1 - o_c)\), the top candidates form a style set \(C_{\text{style}}\) while leaving morphological content channels intact: $\(v' = D(z_{\text{content}}, z_{\text{style}} + \delta)\)$ This surgical style edit reduces spurious attribute logits while strictly maintaining organ boundaries and pathology geometry, with perturbation amplitudes bounded by line search under the dual proximity gate.

4. Adaptive Branch Selection and Safe Fallback Guarantee

At decoding step \(t\), the decoder measures attention concentration on background regions \(\beta_t\) to dynamically scale the contrastive weight \(\alpha_t = \alpha_{\text{min}} + \lambda \beta_t\). The routing mechanism checks whether top-\(k\) token candidates belong to an attribute lexicon \(\mathcal{L}_{\text{attr}}\); if so, admissible LD-VCD edits are favored. When both branches satisfy the proximity constraints, the router selects the view that yields the higher contrastive margin: $\(m(v') = (1 + \alpha_t) \ell(y^* \mid v) - \alpha_t \ell(y^* \mid v') - \max_{y \neq y^*} \left[(1 + \alpha_t) \ell(y \mid v) - \alpha_t \ell(y \mid v')\right]\)$ Crucially, if localization masks fail or visual perturbations cause both branches to violate proximity bounds, PCCD sets \(\alpha_t = 0\) and reverts immediately to greedy decoding. This unconditional fallback breaks the vulnerability chain of prior decoders, guaranteeing that the model never degrades below unmitigated greedy performance.

Loss & Training

PCCD is completely training-free and operates strictly at test-time inference. All weights—including the vision encoder, cross-attention projections, VQGAN autoencoder, and language model backbones—remain frozen. Per image, the spatial mask \(M\), visual embedding \(\phi(v)\), and VQGAN latent code \(z\) are cached once. During token generation, only a single auxiliary forward pass is required per step in standard operation (with an optional second pass over plausible token set \(|V_t| \le 8\) when resolving competitive dual-branch routing), maintaining an asymptotic computational complexity comparable to standard VCD (~2× greedy runtime).

Key Experimental Results

Main Results

Evaluation spans multimodal radiology VQA (MIMIC-Diff-VQA, VQA-RAD), existence probing (Med-POPE), gastrointestinal endoscopy description (Gut-VLM), and full radiology report generation (MIMIC-CXR). All experiments report mean performance across three seeds with patient-level disjoint splits.

Dataset / Benchmark	Backbone	Metric	Ours (PCCD)	Prev. SOTA (VASparse)	Gain
MIMIC-Diff-VQA	CheXagent-3B	GREEN AUC (%)	89.59	87.82	+1.77 pp
MIMIC-Diff-VQA	CheXagent-3B	AUG (%)	64.06	62.08	+1.98 pp
MIMIC-Diff-VQA	LLaVA-Med-7B	GREEN AUC (%)	77.70	76.15	+1.55 pp
VQA-RAD	CheXagent-3B	GREEN AUC (%)	78.60	76.80	+1.80 pp
VQA-RAD	LLaVA-Med-7B	GREEN AUC (%)	68.90	67.00	+1.90 pp
Med-POPE (Adversarial)	CheXagent-3B	F1 Score (%)	79.7	77.4	+2.3 pp
Gut-VLM (GI Endoscopy)	LLaVA-1.6-7Bft	QAAS (%)	92.70	91.35	+1.35 pp
Gut-VLM (GI Endoscopy)	LLaVA-1.6-7Bft	R-Sim	4.19	4.09	+0.10
MIMIC-CXR (Report Gen)	CheXagent-3B	RadGraph F1 (%)	23.1	21.8	+1.3 pp
MIMIC-CXR (Report Gen)	CheXagent-3B	GREEN (%)	30.8	29.4	+1.4 pp

Ablation Study

Component ablations conducted on MIMIC-Diff-VQA (existence-heavy) with CheXagent-3B and Gut-VLM (attribute-heavy) with LLaVA-1.6-7Bft, reporting gains \(\Delta\) relative to baseline greedy decoding:

Config	MIMIC-Diff AUC (%)	\(\Delta\) AUC	Gut-VLM QAAS (%)	\(\Delta\) QAAS	Note
PCCD (Full model)	89.59	+3.39	92.50	+1.61	Full dual-branch model with proximity constraints
OA-only	88.55	+2.35	91.60	+0.71	Specialized for existence, limited on style attributes
LD-only	87.95	+1.75	91.20	+0.31	Specialized for style attributes, weak on co-occurrence
w/o all proximity (\(s, s_{\text{obj}}\))	87.50	+1.30	90.95	+0.06	Severe diagnostic drift without trust regions
w/o object proximity (\(s_{\text{obj}}\))	88.10	+1.90	91.25	+0.36	Context leakage across encoder receptive fields
Fixed \(\alpha\) (no adaptive scaling)	88.35	+2.15	91.70	+0.81	Static contrastive weight ignores context reliance
w/o \(\mathcal{L}_{\text{attr}}\) routing	88.05	+1.85	91.10	+0.21	Attribute tokens fail to route to LD-VCD
Greedy baseline	86.20	-	90.89	-	Unmitigated baseline decoding

Key Findings

• Dual proximity gating is the primary driver of robustness: Removing both proximity guards incurs the single largest performance drop (-2.09 pp AUC on MIMIC-Diff-VQA and -1.55 pp QAAS on Gut-VLM). Furthermore, unconstrained perturbation drops pneumothorax label agreement to ~70%, whereas proximity-guarded counterfactuals preserve over 98.8% label consistency.
• Heterogeneous branches offer complementary specialization: OA-VCD provides the bulk of gains on existence queries (+2.35 pp AUC), while LD-VCD drives the highest R-Sim gain (4.08) in endoscopy attributes, validating the necessity of separating existence and style interventions.
• Safe fallback guarantees empirical non-degradation: Under poor attribution localization (Grad-CAM IoU < 0.2), unguided decoders like AGLA suffer severe degradation (-1.9 pp below greedy). In contrast, PCCD triggers greedy fallback in 82% of steps, ensuring that performance never drops below the unmitigated baseline (\(\Delta \text{AUC} \ge 0\)).

Highlights & Insights

• Representation Trust Region for Decoding: Unlike existing contrastive decoders that perturb images indiscriminately, PCCD introduces formal semantic proximity constraints in feature space, balancing prior suppression against pathology preservation.
• Decoupled Architecture for Heterogeneous Errors: Concurrently tackles co-occurrence existence hallucinations via lesion-preserved context perturbation and attribute hallucinations via disentangled VQGAN style editing.
• Safe Fallback with Zero Retraining: Operates out-of-the-box on frozen models with guaranteed non-degradation via greedy fallback, and naturally stacks with post-hoc fine-tuning methods (yielding +2.5 AUC over aligned models like MedHallTune).

Limitations & Future Work

• Author-Admitted Limitations: First, PCCD incurs roughly 2× wall-clock overhead relative to greedy decoding due to auxiliary forward passes during generation, which could be optimized for real-time interactive reporting. Second, under massive anatomical distortion (e.g., severe complete opacification), latent channel disentanglement in VQGAN can become less precise.
• Identified Limitations: The spatial precision of Grad-CAM is constrained by the downsampling factor of deep vision feature maps, potentially resulting in slightly dilated masks around sub-centimeter pulmonary nodules.
• Improvement Directions: Combining PCCD with visual token sparsification (e.g., VASparse) can lower runtime overhead to ~1.6× greedy, and integrating fine-grained patch-level attention priors can enhance localization around tiny lesions.

Related Work & Insights

• vs VCD (Visual Contrastive Decoding): VCD applies global Gaussian blur, causing up to 17% clinical label flips in subtle findings; PCCD prevents diagnostic degradation via localized object masking and dual proximity bounding.
• vs AGLA / VASparse: Attention-reweighted methods like AGLA lack rejection mechanisms and degrade below greedy baselines when localization is noisy; PCCD integrates explicit semantic trust regions and safe fallback.
• vs Post-Hoc Detectors (VASE, RadFlag): Post-hoc detectors only assess already generated text, while PCCD actively steers the autoregressive token distribution at test time without requiring training data.

Rating

• Novelty: ⭐⭐⭐⭐⭐ (Pioneering representation-space proximity trust regions and decoupled multi-branch counterfactuals for medical VQA)
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Evaluated across 4 diverse clinical modalities, 4 VLM backbones, multiple negative regimes in Med-POPE, and comprehensive ablations)
• Writing Quality: ⭐⭐⭐⭐⭐ (Rigorous mathematical formulation, clear theoretical motivation, and clinically grounded argumentation)
• Value: ⭐⭐⭐⭐⭐ (Training-free, highly safe, and easily deployable for real-world clinical decision support systems)