Skip to content

Med-R2: An Adversarial Benchmark for Evidence-Grounded Reasoning in Medical VLMs

Conference: EMNLP 2026
Paper: ACL Anthology
Area: Multimodal VLM
Keywords: Medical VQA, Evidence-grounded Reasoning, Adversarial Robustness, Clinical Workflow, Visual Grounding

TL;DR

Med-R2 proposes the first four-stage adversarial evaluation benchmark strictly aligned with clinical diagnostic workflows (spanning 42,432 images, 31 tasks, and 110,406 QA pairs), utilizing triad adversarial perturbations and joint reasoning chains to expose that current medical VLMs rely heavily on superficial textual shortcuts rather than robust visual evidence grounding.

Background & Motivation

Vision-Language Models (VLMs) are catalyzing a profound paradigm shift in automated medical diagnostics, evolving rapidly from narrow-domain classifiers toward multi-modal diagnostic assistants capable of synthesizing findings into descriptive natural language rationales. However, their opaque black-box nature represents a formidable barrier to deployment in high-stakes clinical practice. Physicians must verify whether model predictions stem from verifiable pathology visible in radiological scans or whether they merely reflect spurious correlations and textual statistical priors memorized during pre-training.

Conventional Medical Visual Question Answering (Med-VQA) benchmarks exhibit a critical disconnect from real-world clinical demands. Legacy datasets such as VQA-RAD, SLAKE, and recent large-scale suites evaluate models primarily by matching isolated final diagnostic answers. This ignores the multi-tiered cognitive trajectory of practicing radiologists, which progresses systematically from evaluating scan quality and noise, localizing relevant anatomical structures, characterizing lesion morphology, to integrating cross-modal evidence into a structured diagnostic impression. Crucially, existing benchmarks lack systematic adversarial perturbation schemes, allowing models to achieve inflated benchmark scores by leaning on suggestive prompt biases while crumbling when exposed to clinical artifacts, distractor bounding boxes, or misleading diagnostic queries.

To bridge this fundamental evaluation gap, Med-R2 introduces a dual-dimensional evaluation system evaluating both evidence-driven reasoning chains and adversarial reasoning robustness. Integrating 15 international radiological datasets into a large-scale CT/MRI repository, the benchmark mirrors the clinical workflow across four hierarchical stages tested under both independent and joint inquiry paradigms. Core idea: deconstruct the clinical diagnostic trajectory into a four-stage evidence-grounded hierarchy, inject symmetric positive, neutral, and negative adversarial perturbations at every stage, and leverage dual independent-and-joint multi-hop QA to rigorously measure whether models reason from visual evidence or cling to superficial prompt biases.

Method

Overall Architecture

The Med-R2 evaluation framework rigorously formalizes radiological reasoning into a four-stage diagnostic pipeline combined with adversarial stress testing. The benchmark operates over representative 2D slices extracted from 3D CT/MRI volumes, driven by structured clinical queries configured across three complementary modules: Module 1 (Visual Comprehension, 110,406 QA pairs spanning all four clinical stages), Module 2 (Inference Evaluation, featuring 61,876 independent QA pairs and 3,088 multi-hop joint QA chains), and Module 3 (Open-ended Evaluation, comprising 42,354 report generation instances and unconstrained clinical queries). The end-to-end framework measures perceptual grounding, contextual reasoning stability, and resilience against adversarial interference.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Medical Input<br/>CT/MRI slices + clinical prompt queries"] --> B["Four-Stage Clinical Diagnostic Cascade<br/>Image Quality โ†’ Anatomy โ†’ Lesion โ†’ Report"]
    B --> C["Triad Adversarial Perturbation Framework<br/>Positive guidance / Neutral unbiased / Negative distraction"]
    C --> D["Independent and Joint Reasoning Paradigms<br/>Single-step verification vs. multi-hop sequential reasoning"]
    D --> E["Pixel-Level Grounded Verification & Stepwise Tuning<br/>BBox/Segmentation grounding + Hulu-Med full-parameter SFT"]
    E --> F["Multimodal Evaluation Output<br/>Forced-choice ACC + DeepSeek feature ROUGE-1/BERTScore"]

Key Designs

1. Four-Stage Clinical Diagnostic Cascade: Disconnected shallow vision and deep clinical inference -> Formulating four hierarchical diagnostic stages -> Systematically exposing progressive reasoning breakdown

Standard benchmarks treat clinical visual inference as a flat, single-hop classification task, concealing where intermediate reasoning chains break down. Med-R2 models real-world radiological workflows by establishing four sequential reasoning tiers: Stage 1 evaluates Image Quality through simulated Gaussian noise across three PSNR tiers (20 dB, 25 dB, 35 dB) and patient motion blur/ghosting artifacts; Stage 2 examines Anatomy Level understanding across 193 organ/tissue classes with location, boundary, and color variation tests; Stage 3 inspects Lesion Level perception across 212 pathological classes via abnormality detection, lesion recognition, and color box selection; Stage 4 benchmarks Clinical Report generation using authentic clinical impression corpora from CT-RATE and RadGenome-ChestCT alongside 170 disease classifications. This hierarchical breakdown permits granular isolation of model failure points along the diagnostic progression.

2. Triad Adversarial Perturbation Framework: Over-reliance on textual statistical shortcuts over visual grounding -> Injecting symmetric positive, neutral, and negative prompt perturbations -> Stress-testing reasoning robustness and unmasking superficial matching

To expose models that exploit superficial textual patterns without truly parsing visual evidence, Med-R2 constructs an adversarial triad across all evaluation tasks. Positive QA variants provide clear, helpful prompts or precise region-of-interest cues to guide the model along correct reasoning pathways; Neutral QA variants offer uninformative, objective prompts without leading cues to capture pure visual comprehension; Negative QA variants deliberately inject misleading clinical premises, confounding distractors, or conflicting colored bounding boxes to mislead the model toward incorrect reasoning paths. Equal 1:1:1 distributions across all three variants neutralize guessing artifacts and quantify the vulnerability of VLMs when prompt cues diverge from visual truth.

3. Independent and Joint Reasoning Paradigms: Isolated VQA queries fail to reflect cumulative clinical decision pathways -> Chaining multi-level clinical inquiries into sequential dependency loops -> Verifying cross-level logical coherence and intermediate visual anchoring

To overcome the inability of isolated VQA pairs to measure sequential decision-making consistency, Module 2 deploys dual evaluation paradigms. Under Independent QA (61,876 pairs), sub-questions across different reasoning levels are presented separately to evaluate discrete visual feature extraction. Under Joint QA (3,088 multi-step trajectories), queries are linked into an end-to-end prompt: "Modality Identification -> Abnormal Organ Localization -> Visible Lesion Finding -> Final Diagnosis." The model must output all four sequential step predictions simultaneously (e.g., comma-separated letters), requiring later clinical deductions to strictly anchor on intermediate visual deductions and exposing multi-hop logical drift.

4. Pixel-Level Grounded Verification & Stepwise Tuning: Lack of explicit spatial correspondence impedes interpretability -> Curating 15 grounded public datasets and benchmarking stepwise fine-tuning -> Demonstrating spatial alignment deficits and establishing data-driven robustness gains

To ground clinical reasoning in verifiable visual evidence, Med-R2 aggregates 42,432 CT and MRI images across 15 recognized medical imaging repositories (e.g., SA-Med2D-20M, AMOS 2022, DeepLesion, BraTS 2021). All anatomical structures and pathological lesions are paired with pixel-level masks or bounding box coordinates. To evaluate whether this structured grounding data can actively rectify model reasoning vulnerabilities, the authors execute full-parameter stepwise fine-tuning on Hulu-Med models. This validates that supervising models with grounded intermediate clinical stages substantially elevates adversarial resilience on negative distractor samples.

Benchmark & Evaluation Protocol

The benchmark evaluates 14 contemporary vision-language models, encompassing 7 medical-specialized models (Hulu-Med-4B/7B/14B, Linshu-7B, Medgemma-4B, HuatuoGPT-Vision-7B/34B), 4 general-purpose open-source models (Janus-Pro-7B, Intern3-VL-8B, Qwen3-VL-8B, Qwen2.5-VL-7B), and 3 frontier proprietary models (GPT-4o, GPT-5.2-thinking, Qwen3-VL-235B). - Multiple-Choice Tasks: Formulated under educational/research system prompts using forced formatting constraints to require a single uppercase letter output (Aโ€“D), evaluated via standard Accuracy (ACC %). - Joint QA Tasks: Requires comma-separated sequential predictions corresponding to the four chained stages within a unified prompt context. - Open-Ended Clinical Reports: Evaluated by using DeepSeek-V3 to extract clinically relevant visual findings (both normal and pathological) from both candidate reports and references, followed by computing lexical ROUGE-1 and semantic BERTScore over the extracted feature sets.

Key Experimental Results

Main Results

Comprehensive benchmark performance across 14 VLMs on Med-R2 is summarized below (ACC % reported for categorical stages and Joint QA; ROUGE-1 / BERTScore reported for Clinical Reports):

Model Group Model Image Quality (ACC %) Anatomy Level (ACC %) Lesion Level (ACC %) Joint QA (ACC %) Overall Avg (ACC %) Clinical Report (ROUGE-1 / BERTScore)
Medical-Specific Hulu-Med-4B 41.4 60.3 57.9 58.4 54.5 13.7 / 83.3
Medical-Specific Hulu-Med-7B 36.5 60.7 58.8 48.9 51.2 14.6 / 83.6
Medical-Specific Hulu-Med-14B 47.4 68.1 68.6 63.1 61.8 14.5 / 86.3
Medical-Specific Linshu-7B 67.3 56.0 41.6 60.2 56.3 12.7 / 82.0
Medical-Specific Medgemma-4B 24.2 36.4 32.5 45.3 34.6 10.9 / 81.1
Medical-Specific HuatuoGPT-Vision-7B 41.9 58.7 50.5 51.0 50.5 8.5 / 81.5
Medical-Specific HuatuoGPT-Vision-34B 48.0 65.9 53.5 54.8 55.6 12.5 / 83.1
Open-Source Janus-Pro-7B 74.6 51.3 49.5 32.7 52.0 10.3 / 81.9
Open-Source Intern3-VL-8B 58.3 61.0 54.0 73.8 61.8 10.3 / 82.6
Open-Source Qwen3-VL-8B 53.3 63.3 52.0 75.3 61.0 8.6 / 81.6
Open-Source Qwen2.5-VL-7B 49.3 51.5 55.1 57.3 53.3 7.8 / 80.1
Proprietary GPT-4o 53.0 56.1 47.0 65.5 55.4 8.6 / 82.3
Proprietary GPT-5.2-thinking 63.7 65.6 49.9 90.2 67.4 9.9 / 81.9
Proprietary Qwen3-VL-235B 54.2 64.2 51.3 72.5 60.6 7.9 / 81.2

Detailed breakdown on the critical Lesion Level under adversarial environments highlights vulnerability to misleading prompts:

Model Abnormality Pos Abnormality Neu Abnormality Neg Lesion Rec Pos Lesion Rec Neu Lesion Rec Neg Color Box Pos Color Box Neg Lesion Avg (ACC %)
Hulu-Med-4B 77.3 73.6 43.5 86.4 54.9 20.7 62.35 44.2 57.9
Hulu-Med-7B 84.0 78.9 33.3 75.3 64.5 35.1 65.28 33.9 58.8
Hulu-Med-14B 96.1 92.9 65.7 87.8 60.2 27.9 68.9 48.9 68.6
Linshu-7B 51.5 23.4 11.3 69.6 57.9 41.8 45.8 31.6 41.6
Medgemma-4B 45.2 32.8 22.9 41.2 32.7 28.9 39.7 16.4 32.5
HuatuoGPT-Vision-7B 50.7 55.6 54.8 73.2 60.7 32.5 40.3 36.4 50.5
HuatuoGPT-Vision-34B 87.5 83.2 21.5 57.1 54.5 51.9 39.7 31.3 53.3
Janus-Pro-7B 90.7 56.8 12.4 73.2 51.9 40.4 37.1 33.6 49.5
Intern3-VL-8B 71.0 68.8 33.9 70.1 62.5 55.3 36.1 33.9 54.0
Qwen3-VL-8B 75.6 76.4 25.3 86.1 51.3 35.2 33.9 31.8 52.0
Qwen2.5-VL-7B 80.7 68.9 16.4 71.4 67.4 71.4 35.9 28.9 55.1
GPT-4o 41.0 55.1 51.0 62.9 54.8 44.1 35.1 31.9 47.0
GPT-5.2-thinking 88.4 44.2 27.1 82.7 59.2 33.1 34.1 30.2 49.9
Qwen3-VL-235B 76.3 72.2 35.7 90.8 44.1 27.2 34.2 29.5 51.3

Ablation Study

Full-parameter fine-tuning of Hulu-Med-7B on Med-R2 demonstrates dramatic calibration and adversarial defense gains:

Evaluation Dimension & Task State Positive QA Neutral QA Negative QA Task Average (Avg)
Image Quality (Noise) Zero-shot 61.9 42.2 5.1 41.1 (Overall IQ Avg)
Finetuned 97.7 (โ†‘35.8) 98.1 (โ†‘55.9) 90.3 (โ†‘85.2) 95.2 (โ†‘54.1)
Image Quality (Artifacts) Zero-shot 93.3 37.2 8.7 -
Finetuned 97.9 (โ†‘4.6) 95.3 (โ†‘58.1) 92.1 (โ†‘83.4) -
Anatomy Level (Location) Zero-shot 72.1 46.9 63.4 60.3 (Overall Anatomy Avg)
Finetuned 90.0 (โ†‘17.9) 81.8 (โ†‘34.9) 63.3 (โ†‘-0.1) 86.7 (โ†‘26.4)
Anatomy Level (Recognition) Zero-shot 81.6 51.4 46.5 -
Finetuned 97.7 (โ†‘16.1) 94.5 (โ†‘43.1) 95.1 (โ†‘48.6) -
Lesion Level (Recognition) Zero-shot 86.4 54.9 20.7 57.9 (Overall Lesion Avg)
Finetuned 88.4 (โ†‘2.0) 87.0 (โ†‘32.1) 51.9 (โ†‘31.2) 82.5 (โ†‘24.6)
Lesion Level (Color Box) Zero-shot 62.4 (Pos) - 44.2 (Neg) -
Finetuned 79.6 (โ†‘17.2) - 74.1 (โ†‘29.9) -
Clinical Report (Interference) Zero-shot 15.4 / 83.4 (Box Pos) - 10.8 / 83.2 (Box Neg) 13.7 / 83.3 (Report Avg)
Finetuned 57.4 / 90.9 (โ†‘42.0) - 37.9 / 87.6 (โ†‘27.1) 41.9 / 89.9 (โ†‘28.2)

Key Findings

  • Sequential performance degradation across clinical hierarchy: Model competence systematically deteriorates as tasks progress from low-level image quality and anatomy localization to high-level lesion analysis and clinical reporting. While models handle macro-anatomical parsing relatively well, fine-grained lesion identification and morphological deduction serve as the primary performance bottleneck across all architectures.
  • Asymmetric performance across adversarial triads: A persistent gap exists where \(Positive > Neutral > Negative\). Models readily conform to misleading cues under Negative QA, dropping accuracy substantially (e.g., Janus-Pro collapsing from 90.7% to 12.4% in abnormality detection), while achieving inflated accuracy under Positive QA. This demonstrates that models rely on surface text heuristics rather than grounded perceptual verification.
  • Widespread spatial grounding and localization deficiency: Across all 14 evaluated models, performance on Color Box Lesion Selection remains poor (e.g., GPT-5.2-thinking achieving only 34.1/30.2% and Qwen3-VL-235B reaching only 34.2/29.5%), revealing that contemporary vision encoders struggle with fine-grained spatial-semantic coordinates in complex clinical scans.
  • CoT reasoning models excel at multi-hop logic but remain vulnerable to perceptual distractors: GPT-5.2-thinking dominates the Joint QA task with an exceptional score of 90.2%, yet its lesion-level negative adversarial accuracy remains constrained (27.1% on abnormality detection), indicating that advanced chain-of-thought capabilities cannot compensate for underlying multimodal perceptual hallucinations.

Highlights & Insights

  • Clinical workflow alignment provides clear diagnostic transparency: By structuring evaluation around the radiologist's natural decision cascade, Med-R2 pinpoints exact failure modes instead of reporting monolithic end-to-end classification accuracy.
  • Triad adversarial construction unmasks shortcut learning: Symmetrically balancing positive, neutral, and negative prompts offers a rigorous methodology to neutralize prompt exploitation, establishing a gold-standard stress test for medical AI robustness.
  • Stepwise fine-tuning yields substantial robustness gains: Full-parameter fine-tuning on grounded, multi-level tasks elevates negative adversarial robustness (e.g., noise negative QA surging from 5.1% to 90.3%), providing actionable blueprints for developing resilient clinical foundation models.

Limitations & Future Work

  • Static evaluation disconnected from longitudinal clinical context: Med-R2 operates over discrete image slices and static QA, omitting vital longitudinal patient histories, prior imaging comparisons, and lab reports available in clinical practice.
  • 2D slice approximation of 3D volumetric scans: Due to context window and computational bounds in current VLMs, the benchmark selects the most informative 2D slice from 3D/4D volumes, sacrificing contiguous volumetric and spatial depth context.
  • Future trajectories: Developing 3D native multimodal architectures, expanding into multi-temporal follow-up comparisons, and exploring test-time adaptation strategies to actively detect and reject adversarial misleading prompts.
  • vs VQA-RAD / SLAKE / VQA-Med: Early datasets provided small-scale, ungrounded single-step QA without adversarial testing; Med-R2 scales to 110K+ QA pairs across a four-stage clinical hierarchy with rigorous adversarial perturbations.
  • vs OmniMedVQA / MultiMedEval: Broad evaluation toolkits focus primarily on task diversity without chaining multi-step clinical reasoning; Med-R2 explicitly assesses step-by-step reasoning coherence via Joint QA.
  • vs DrVD-Bench: While DrVD-Bench investigates reasoning trajectories, Med-R2 incorporates a larger multimodal dataset (42K+ images), finer task granularity (31 tasks), and comprehensive pixel-grounded bounding box adversarial perturbations.

Rating

  • Novelty: โญโญโญโญโญ First benchmark to systematically synthesize clinical diagnostic workflows with triad adversarial perturbations to audit evidence-grounded VLM reasoning.
  • Experimental Thoroughness: โญโญโญโญโญ Comprehensive evaluation across 14 diverse proprietary and open-source models, paired with fine-tuning ablation experiments.
  • Writing Quality: โญโญโญโญโญ Well-structured, transparently organized tasks, and insightful empirical visualizations.
  • Value: โญโญโญโญโญ Highly impactful diagnostic benchmark for advancing reliable, interpretably grounded medical vision-language systems.