MedQA-MM: Shortcuts Behind Medical Visual Reasoning¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/benluwang/MedQA-MM-EMNLP2026
Area: Vision-Language Reasoning
Keywords: Medical Multimodal MCQ, Shortcut Learning, Reasoning Inflation, Matched Counterfactual Repairs, Cross-Modal Benchmark Diagnostics
TL;DR¶
Addressing the "reasoning inflation" crisis where medical multimodal multiple-choice benchmarks conflate surface accuracy with clinical visual reasoning, this paper establishes an evidence-ladder auditing framework with matched minimal repairs across six benchmarks, and releases MEDQA-MM, a 1,000-item shortcut-mitigated evaluation benchmark.
Background & Motivation¶
Medical vision-language models (VLMs) have demonstrated surging benchmark accuracy across multimodal multiple-choice question (MCQ) suites, leading researchers and clinicians to infer that these systems possess sophisticated clinical acumen capable of integrating high-resolution medical imaging with nuanced patient presentations. However, standard MCQ evaluation protocols record only the final discrete choice selection, remaining fundamentally agnostic to the underlying reasoning route that yielded the answer. When an expert diagnostician examines a clinical vignette, the decisive "aha!" moment is triggered by an image-grounded findingโsuch as identifying a focal air-space consolidation in the right lower lobeโand connecting it with differential pathology. In contrast, multimodal LLMs frequently arrive at the identical choice through an unintended shortcut route, capitalizing on option length asymmetries, clinical heuristics in the text prompt, burned-in DICOM annotations, or visible surgical hardware.
This divergence between observed performance and latent competence manifests as severe "reasoning inflation": benchmark accuracy is reflexively celebrated as evidence of genuine multimodal reasoning even when decisions are driven by non-visual heuristics. Medical multimodal MCQs uniquely inherit multiple shortcut modalities that have historically been investigated in isolation: educational measurement flaws (such as conspicuous length discrepancies, absolute exclusionary phrasing, and grammatical cues), NLP dataset artifacts (such as hypothesis-only heuristics in NLI), visual question answering language priors, and radiology-specific artifacts (such as arrows, ruler calipers, chest tubes, and orthopedic fixation plates). Naively deleting all contaminated questions risks decimating scarce medical datasets and indiscriminately purging clinically necessary patient context.
To resolve this measurement dilemma, the authors introduce a data-centric audit-and-repair framework governed by a systematic evidence ladder. Rather than conflating the static presence of a surface cue with harmful model dependence, the approach combines prompt-visible field normalization, four-condition modality ablations, non-visual diagnostic guesser baselines, and counterfactual matched repairs that perturb suspected surface cues while strictly preserving medical truth conditions and one-best-answer structures. The core idea is to formalize a "two-aha" route-disentanglement framework across option form, non-visual clinical text, and image artifacts, deploying matched minimal repairs and clinician validation to isolate true visual reasoning from superficial heuristics and establish the shortcut-mitigated MEDQA-MM benchmark.
Method¶
Overall Architecture¶
The audit-and-repair methodology encompasses an end-to-end pipeline: canonicalizing items across six medical multimodal MCQ datasets, enforcing strict segregation between provenance metadata and prompt-visible fields, executing multi-condition modality ablations, applying constrained counterfactual edits, and subjecting repairs to double-blind clinician review. Evaluated across Full Input, Text-only (no image), Options-only (no stem or image), and Image+Options (no question stem) settings, the framework exposes the non-visual solvability of benchmark items and quantifies causal cue dependence via targeted destination analysis.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multimodal Medical MCQ<br/>(Image + Stem + Options)"] --> B["1. Evidence Ladder Protocol<br/>4-condition ablation & route separation"]
B --> C["2. Option-Form Matched Repairs<br/>Length-gap / Spatial / Absolute edits"]
C --> D["3. Image Hint & Natural Cue Neutralization<br/>Masked inpainting & SDOH contextual control"]
D --> E["4. Clinical Semantic & Risk Adjudication<br/>Dual-expert panel validation"]
E --> F["MEDQA-MM Benchmark<br/>(1,000-item shortcut-mitigated suite)"]
Key Designs¶
1. Evidence Ladder Protocol: Pain Point -> Mechanism -> Efficacy Standard evaluation treats aggregate accuracy on complete multimodal inputs as a monolithic indicator of visual diagnostic reasoning, ignoring that models frequently succeed via non-visual routes. To eliminate this confounding, the authors formalize an evidence ladder spanning four distinct evidential tiers: screened candidate cues identified by surface heuristics; ablation-supported cues exposed through partial-input probes (Options-only, Text-only, Image+Options) and dedicated diagnostic guessers; repair-supported cues validated via counterfactual behavioral shifts; and expert-validated cues verified by clinician panels. By decoupling static cue detection from empirical model reliance, this protocol prevents legitimate, clinically indispensable findings (such as chest tubes indicating pneumothorax management) from being indiscriminately discarded as spurious shortcuts.
2. Option-Form Matched Repairs: Pain Point -> Mechanism -> Efficacy Surface-level option formatting frequently leaks the correct answer without requiring image examination, as exam writers inadvertently make correct options disproportionately longer, uniquely enriched with spatial prepositions, or contrasted against overly extreme distractors. The authors construct three matched repair cohorts totaling 508 original-edited pairs: length-gap repairs (306 pairs), spatial/prepositional repairs (101 pairs), and absolute/conspicuous distractor repairs (101 pairs). Under strict minimal-edit constraints, an advanced LLM minimally rewrites 1โ2 distractorsโexpanding distractor character length by at least 5 characters, mirroring anatomical/spatial terminology, or softening absolute terms like "never" or "always"โwhile keeping the question stem, correct answer key, and option order strictly intact. Evaluating models across paired items via repeated sampling and destination analysis reveals marked accuracy drops (\(\Delta_{\text{text}} = \text{Acc}_{\text{edited}} - \text{Acc}_{\text{original}}\)) accompanied by precise probability mass redirection toward the manipulated distractors.
3. Image Hint & Natural Cue Neutralization: Pain Point -> Mechanism -> Efficacy Diagnostic medical figures frequently feature embedded physician annotations (arrows, circles, bounding boxes, or callout text) that pinpoint lesions, while clinical vignettes incorporate sociodemographic and social-determinants-of-health (SDOH) attributes (such as race, occupation, or housing) that trigger memorized illness scripts. For artificial visual cues, 200 high-confidence candidate images undergo transparent-mask localization and conservative inpainting to erase annotation strokes while preserving underlying pathological morphology. For natural clinical context, 302 paired items undergo controlled context neutralization to remove nonessential demographic signals without altering clinical care standards. To rigorously assess whether models genuinely suffer from cue removal rather than harmlessly reporting it, the authors define the bad/cue ratioโthe proportion of instances where a model used the cue in its original correct rationale but failed once the cue was neutralizedโrevealing that while larger frontier models remain robust, compact models exhibit substantial harmful reliance (exceeding 15% to 20%).
4. Clinical Semantic & Risk Adjudication: Pain Point -> Mechanism -> Efficacy Unsupervised automated benchmark rewriting risks introducing medical inaccuracies, creating ambiguous clinical choices, or generating accidental multiple-correct options that corrupt evaluation validity. To guarantee strict truth-value preservation, a five-member clinical review panelโincluding a senior board-certified radiologist and an academic surgeonโconducts double-blind evaluations covering cue plausibility, medical correctness, one-best-answer retention, and artifact reduction. Human validation confirmed 94.7% (142/150) medical validity in option-form repairs and 97.8% (45/46) validity in visual hint repairs. Crucially, where clinical artifacts such as surgical fixation plates or endovascular embolization coils cannot be erased without destroying anatomical textures (48 borderline and 9 confirmed cases), the protocol enforces full item exclusion rather than risky inpainting.
Shortcut Attribution and Cross-Modal Diagnostic Protocol¶
To guarantee that prospective items genuinely demand visual interpretation, the protocol institutes an adversarial stress test utilizing diagnostic guesser models. An Options-only guesser (Qwen3.5-4B LoRA) and a Question+Options guesser (Qwen3.5-9B LoRA) are fine-tuned strictly without visual input to probe dataset-level statistical regularities and illness script memorization. Items solvable with high confidence by non-visual guessers are flagged as construct-compromised. Under this diagnostic gating, items qualify for the high-assurance benchmark only when text-only and options-only performance drops to baseline chance levels, ensuring that credited scores reflect true image-grounded diagnostic deduction.
Key Experimental Results¶
Main Results¶
The canonical audit evaluates 7,706 items across six medical multimodal benchmarks (AMBOSS, JAMA Clinical Challenge, MMMU Health/Medicine, MedThinkVQA, MedXpertQA-MM, NEJM Image Challenge). Results across a 13-configuration open-model panel and specialized non-visual diagnostic guessers demonstrate pervasive non-visual solvability.
| Input Setting / Model Condition | AMBOSS | JAMA | MedThinkVQA | MedXpertQA-MM | MMMU-HM | NEJM | Overall Mean / Panel Average |
|---|---|---|---|---|---|---|---|
| Random Chance Baseline (\(1/K\)) | 17.24% | 24.95% | 20.00% | 20.00% | 25.95% | 20.00% | 22.13% |
| Options-only Guesser (No Stem / No Image) | 32.96% | 51.66% | 57.78% | 26.46% | 33.93% | 34.49% | 38.12% (+15.99 pp) |
| Question+Options Guesser (No Image) | 87.78% | 70.16% | 54.72% | 34.36% | 47.67% | 56.12% | 53.75% |
| Open-Model Multimodal Panel (13 Configs): | |||||||
| - Options-only Setting | โ | โ | โ | โ | โ | โ | 29.71% |
| - Image+Options Setting | โ | โ | โ | โ | โ | โ | 43.84% |
| - Text-only Setting (No Image) | โ | โ | โ | โ | โ | โ | 53.96% |
| - Full Input (Image + Stem + Options) | โ | โ | โ | โ | โ | โ | 62.63% |
Ablation Study¶
The ablation evaluation benchmarks causal impact across the 508 matched option-form repair pairs (with 20 repeats per condition and 95% confidence intervals) and measures cross-modal resilience on the newly constructed 1,000-item MEDQA-MM subset.
| Intervention Cohort / Evaluation Setting | Sample Size (\(N\)) | Metric Definition | Accuracy Change (\(\Delta\)) | Selection Shift: Edited Distractor | Selection Shift: Untouched Distractor |
|---|---|---|---|---|---|
| Length-Gap Repair Cohort | 306 pairs | \(\Delta_{\text{text}}\) (edited - original) | -6.58 pp [-7.60, -5.59] | +9.47 pp [8.26, 10.75] | -2.87 pp [-3.66, -2.11] |
| Spatial/Prepositional Repair Cohort | 101 pairs | \(\Delta_{\text{text}}\) (edited - original) | -4.77 pp [-5.86, -3.70] | +2.63 pp [1.28, 4.05] | -0.69 pp [-1.85, 0.40] |
| Absolute/Conspicuous Repair Cohort | 101 pairs | \(\Delta_{\text{text}}\) (edited - original) | -3.50 pp [-5.18, -1.79] | +4.07 pp [1.60, 6.35] | -1.46 pp [-2.84, 0.00] |
| MEDQA-MM Subset Evaluation: | |||||
| - Options-only Probe | 1,000 items | Absolute Accuracy | 12.33% (well below chance) | โ | โ |
| - Text-only Probe (No Image) | 1,000 items | Absolute Accuracy | 5.21% (drastic non-visual collapse) | โ | โ |
| - Image+Options Probe | 1,000 items | Absolute Accuracy | 28.47% (+1.88 pp over Full Input) | โ | โ |
| - Full Multimodal Input | 1,000 items | Absolute Accuracy | 26.59% (authentic high-difficulty regime) | โ | โ |
Key Findings¶
- Severe Text-Only Shortcut Permeability: Across 13 open multimodal configurations, Text-only accuracy averages 53.96%, trailing Full Input (62.63%) by a mere 8.67 pp. The Q+O guesser achieves 87.78% on AMBOSS without ever seeing pixels, proving that existing benchmarks overwhelmingly reward textual illness script memorization rather than visual acumen.
- Direct Behavioral Sensitivity to Option Geometry: Neutralizing length-gap shortcuts reduces model accuracy by 6.58 pp, causing an immediate +9.47 pp migration of predictions directly toward the lengthened distractors while untouched distractors drop by 2.87 pp. In clean spatial subsets where length confounds are rigorously excluded, target distractor selection still climbs by +2.70 pp, confirming that option surface syntax causally steers model attention.
- Elimination of Non-Visual Routes in MEDQA-MM: On the curated 1,000-item MEDQA-MM benchmark, Text-only accuracy collapses to 5.21% and Options-only drops to 12.33%, completely shutting down language-shortcut exploits. Concurrently, Image+Options (28.47%) exceeds Full Input (26.59%), where expert error analysis reveals that question stems actively introduce distraction ("stem interference") in 10.7% of errors by anchoring models onto misleading textual differentials.
Highlights & Insights¶
- Formulates the "Two-Aha" conceptual framework that unifies psychometric item-writing principles, NLP dataset artifacts, and medical computer vision biases under a unified construct validity lens, demonstrating why raw accuracy scores induce reasoning inflation.
- Pioneers the counterfactual matched repair methodology for multimodal benchmarks, moving beyond observational correlations to deliver rock-solid causal proof via destination analysis that models systematically lean on superficial option formatting.
- Replaces naive claims of "shortcut-free" benchmarks with a realistic, transparent "shortcut-mitigated" reporting paradigm that mandates accompanied releases of ablation scores, repair provenance, and clinician-reviewed residual risks.
Limitations & Future Work¶
- Human Clinical Audit Bandwidth: While automated editing scales efficiently, clinical validation requires exhaustive expert labor. Only representative subsets underwent full double-blind clinician verification, leaving open the need for community-wide verification at scale.
- Technical Bounds on Inpainting Medical Hardware: Post-treatment hardware (implants, surgical clips, pacemaker leads) often signals clinical stage but cannot be inpainted without generating unnatural blurring or visual hallucinations, currently forcing outright item exclusion.
- Resolving Stem Interference Dynamics: The counter-intuitive finding that omitting clinical vignettes improves multimodal accuracy highlights an unresolved architectural vulnerability: VLMs lack adaptive cross-modal confidence weighting to ignore uninformative text when image signs are definitive.
Related Work & Insights¶
- vs MMMU-Pro / HallusionBench: While general diagnostic suites rely on artificial image perturbations, random rotation, or text injection to uncover hallucinations, MedQA-MM investigates genuine clinical exam flaws, naturally occurring demographic heuristics, and medical annotation cues through conservative matched counterfactuals.
- vs Med-Flamingo / LLaVA-Med: Prior medical foundation models derived their headline competence claims from high accuracy on existing multimodal exam benchmarks; this work demonstrates that substantial proportions of those gains stem from non-visual shortcuts, providing an essential correction for future clinical evaluation.
Rating¶
- Novelty: โญโญโญโญโญ Establishes a rigorous psychometric and causal foundation for auditing and repairing shortcuts in multimodal medical reasoning.
- Experimental Thoroughness: โญโญโญโญโญ Spans 7,706 items across six major benchmarks, 19 model configurations, 508 matched repair pairs with repeated sampling, destination tracking, and multi-clinician adjudication.
- Writing Quality: โญโญโญโญโญ Exceptionally lucid narrative connecting complex measurement theory with concrete empirical protocols.
- Value: โญโญโญโญโญ Crucial benchmark correction preventing false reassurance regarding the autonomous diagnostic capabilities of medical AI.