Skip to content

MedVision: Benchmarking Quantitative Medical Image Analysis

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://medvision-vlm.github.io
Area: LLM Reasoning
Keywords: Vision-Language Models, Quantitative Medical Imaging, Chain-of-Thought Reasoning, Reinforcement Fine-Tuning, Process Rewards

TL;DR

The paper uncovers severe metric and spatial measurement failures across general and specialized medical vision-language models, curates MedVision as a large-scale quantitative benchmark across 22 multi-modal datasets with 24.3M single-instance annotations, and introduces MedVision-V0, an end-to-end VLM trained with chain-of-thought SFT and multiplicative process-reward GRPO reinforcement fine-tuning that dramatically reduces numeric measurement errors and aligns with real-world clinical staging criteria.

Background & Motivation

Recent advances in multimodal foundation models and vision-language models (VLMs) have enabled the seamless fusion of high-capacity visual encoders with flexible natural language reasoning engines, opening promising avenues across medical domains. Nevertheless, almost all prevailing medical VLM deployments and benchmarking efforts remain overwhelmingly confined to categorical or descriptive qualitative tasks. Standard pipelines predominantly focus on binary anomaly detection (e.g., classifying a scan as normal versus abnormal), visual question answering over semantic categories, or generating unconstrained radiology narrative reports. While these qualitative diagnostic capabilities are fundamentally important, they fail to cover an indispensable pillar of everyday clinical practice: the extraction of precise, reproducible quantitative measurements from imaging data. Routine patient management, oncological staging, surgical intervention planning, and therapeutic response evaluation fundamentally depend on physical numerical metrics, such as measuring bidirectional lesion diameters under RECIST guidelines, assessing Cobb angles for scoliosis, or calculating articular joint distances.

This clinical blind spot highlights a critical deficiency in contemporary multimodal architectures and evaluation protocols. Existing benchmarks (such as GMAI-MMBench, OmniMedVQA, and RadBench) prioritize text-generation fluency or categorical classification accuracy rather than measurement precision and geometric calibration. Even cutting-edge open-weight general VLMs spanning tens or hundreds of billions of parameters, as well as specialized medical models trained on vast biomedical corpora, exhibit dramatic failure modes when queried for real-world physical metrics (e.g., millimeters or degrees). The fundamental tension stems from the fact that quantitative vision-language tasks demand precise spatial localization across highly heterogeneous imaging modalities (CT, MRI, ultrasound, PET, X-ray) coupled with the algebraic capability to ground image-space coordinates into real-world physical dimensions using DICOM header spacing. Off-the-shelf VLMs fail to internalize spatial scale conventions, producing severe localization drift, dimensional collapse, and wildly erratic numerical estimates.

To bridge this qualitative-quantitative chasm and evaluate whether end-to-end VLMs can internalize multi-step geometric perception without external specialist tools, a rigorous quantitative benchmark and post-training recipe are sorely needed. Core idea: curate MedVision, the first large-scale, multi-modal, multi-anatomy medical quantitative benchmark spanning 22 public datasets and 24.3M annotations, and develop MedVision-V0 via a two-stage post-training paradigm—combining explicit geometric chain-of-thought (CoT) supervised fine-tuning with a multiplicative process-reward GRPO reinforcement fine-tuning strategy to achieve unified, tool-free quantitative perception and measurement.

Method

Overall Architecture

The MedVision framework establishes an automated dataset construction, benchmark evaluation, and end-to-end model training ecosystem. On the data side, the system aggregates 22 public medical imaging repositories possessing segmentation masks or landmark annotations, strictly extracts physical spacing parameters from image file headers, and re-orients 3D volumes into standard RAS+ coordinate space. The benchmark defines three representative quantitative task families: anatomical and pathological bounding-box detection, tumor/lesion (T/L) size estimation (extracting major and minor axis physical lengths via ellipse fitting), and angle/distance (A/D) measurement from anatomical landmarks. To train an end-to-end quantitative reasoner without auxiliary tool dependencies, MedVision-V0 adopts a two-stage post-training progression initialized from Qwen2.5-VL (7B): an initial SFT stage on 121K axial slices instills explicit step-by-step coordinate localization and arithmetic formula reasoning within <think> tags, followed by an RFT stage utilizing GRPO where policy rollouts are optimized against a combination of formatting compliance, point-displacement process rewards, and relative measurement error rewards.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-Modal Medical Datasets & Headers<br/>Physical spacing extraction & RAS+ standardization"] --> B["Automated Quantitative Annotation Pipeline<br/>Bounding box / Fitted ellipse axes / Landmark geometry"]
    B --> C["Chain-of-Thought Supervised Fine-Tuning SFT<br/>Predict intermediate coordinates & apply physical formulas"]
    C --> D["Multiplicative Process & Answer Reward Modeling<br/>Point displacement process reward & relative error outcome"]
    D --> E["GRPO Reinforcement Fine-Tuning RFT<br/>Group relative policy optimization with dynamic curriculum"]
    E --> F["End-to-End Quantitative Inference Output<br/>Structured machine-parseable metrics & clinical decision categories"]

Key Designs

1. Automated Quantitative Annotation Pipeline: bridging physical space and image coordinate grids

Conventional medical VLM benchmarks fail at quantitative reasoning primarily because image inputs are decoupled from their physical voxel spacing information. MedVision resolves this by converting raw mask and landmark annotations from 22 clinical datasets into calibrated geometric targets. For bounding boxes, clusters with fewer than 10 pixels are discarded to avoid spurious noise. For tumors and lesions, ellipses are fitted directly to 2D binary masks in the real-world metric space (computed by multiplying pixel array coordinates by physical pixel dimensions), recording the major and minor axis lengths in millimeters. Landmark coordinates are derived from certified clinical keypoints. Because off-the-shelf VLMs utilize idiosyncratic image-processing strategies (such as aspect-ratio-preserving padding, resizing, or dynamic slicing), the benchmark implements an adaptive prompt-construction pipeline ensuring that the image size and physical pixel spacing provided in the prompt dynamically mirror the exact tensor dimensions ingested by the visual encoder, enforcing a self-consistent physical coordinate reference frame.

2. Chain-of-Thought Supervised Fine-Tuning: decoupling visual landmark perception from metric conversion

Directly prompting models to output scalar floating-point measurements in an unconstrained end-to-end fashion induces severe spatial hallucinations and dimensional drift. To guide structured geometric reasoning, the authors design a modular Chain-of-Thought post-training paradigm that decomposes every measurement task into two sequential operations: first, estimating relative coordinates of landmark keypoints or bounding box corners in image space (bounded in \([0, 1]\) with origin at the lower-left corner); second, applying explicit analytic geometry formulas that combine predicted coordinates with the provided pixel spacing and image dimensions. A training split of 121K axial slices (110K detection, 5.5K T/L size, 5.5K A/D measurement) is curated, wrapping intermediate ground-truth coordinates inside <step-k-reasoning> and <step-k-answer> tags within a <think> block, concluding with machine-parseable <answer> tags. In-depth error analysis (Table 2) demonstrates that tumor/lesion boundary localization is inherently more challenging than anatomical landmarks (normL2 error of 6.0% vs. 1.6%), and arithmetic calculation is considerably more error-prone for angles than distances. SFT provides the foundational inductive bias required to organize this multi-step reasoning pathway.

3. Multiplicative Process & Answer Reward Modeling: penalizing unfaithful shortcuts in reinforcement learning

During reinforcement fine-tuning via GRPO, all ground-truth CoT demonstration answers are stripped, requiring the model to generate its own reasoning rollouts guided strictly by scalar rewards. To prevent the policy from gaming the system by emitting lucky numerical answers from corrupted intermediate localizations, the framework constructs a multiplicative reward function: \(r = r_{\text{format}} + r_{\text{process}} \cdot r_{\text{answer}}\). The format reward \(r_{\text{format}} \in [0, 1]\) checks tag syntax and step progression. The process reward \(r_{\text{process}}\) parses the intermediate coordinates from each <step-k-answer> tag, computes the worst-case Euclidean point displacement error normalized by the unit-square diagonal \(e_k = \|\hat{p}_k - p_k\|_2 / \sqrt{2} \in [0, 1]\), and scores it via exponential decay \(\rho(e_k) = \exp(-e_k)\). The outcome reward \(r_{\text{answer}}\) scores the final relative error or Complete IoU (CIoU) for detection. Under this multiplicative composition, an accurate final measurement receives zero credit if the intermediate geometric localization is degraded, establishing an uncompromising enforcement of verifiable geometric reasoning.

4. Temperature-Scaled Task Balancing and Dynamic Curriculum Learning: counteracting data skew and advantage collapse

The post-training mixture suffers from an extreme intrinsic sample imbalance, containing 110K detection prompts versus only 5.5K T/L size and 5.5K A/D prompts (a 20:1 ratio). Naive joint training would overwhelm the gradients with bounding-box objectives. The framework introduces temperature-scaled task mixing, assigning task sampling probability \(P(t) \propto N_t^{1/T}\), with \(T=5\) in SFT (yielding 47.7% detection and 26.2% for each measurement task) and \(T=8\) in multi-task RFT (42.1% detection, 29.0% each measurement). Furthermore, because GRPO normalizes advantages within a prompt's rollout group, prompts that are consistently solved yield near-zero advantage variance, wasting rollout compute. The multi-task RFT implementation incorporates an epoch-level curriculum that dynamically tracks exponential moving averages of rewards and errors (\(\bar{r}_i, \bar{e}_i\)). Prompts consistently achieving MRE \(< 0.1\) or overlap error \(< 0.25\) over consecutive epochs are promoted to an easy pool (up to 20% per epoch), while a 30% retention mix-in and a 5% rotating audit pool with hysteresis factor \(\lambda=1.5\) protect against catastrophic forgetting and demote degraded samples back into active training.

Loss & Training

MedVision-V0 undergoes full-parameter tuning on 4 NVIDIA H200 GPUs using the GRPO implementation in verl with vLLM rollouts. Training employs a group size of 8 rollouts per prompt, a batch size of 256 prompts, mini-batch policy updates of 128, a constant learning rate of \(3 \times 10^{-6}\), and a reference policy KL regularization coefficient of 0.01. For single-stage multi-task RFT, reward computation maps normalized errors \(e\) through \(\rho(e) = \exp(-e)\): For detection: $\(r = r_{\text{format}} + \exp\left(-\frac{1 - \text{CIoU}(\hat{y}, y)}{2}\right)\)$ For T/L size and A/D measurement tasks: $\(r = r_{\text{format}} + \left(\frac{1}{n}\sum_{k=1}^n \exp(-e_k)\right) \cdot \exp\left(-\frac{1}{d}\sum_{j=1}^d \frac{|\hat{y}_j - y_j|}{y_j}\right)\)$

Key Experimental Results

Main Results

MedVision evaluates 18 state-of-the-art vision-language models across anatomy detection, lesion detection, tumor/lesion size estimation, and landmark angle/distance measurements. The table below reports key quantitative metrics across representative models (detection F1 and IoU for anatomy; MRE for measurement tasks):

Model Parameters Anatomy Det. F1 (%) ↑ Anatomy Det. IoU (%) ↑ T/L Size MRE (%) ↓ Distance MRE (%) ↓ Angle MRE (%) ↓
MedVision-V0 (Ours) 7B 79.1 72.0 26.0 6.4 52.1
Gemma4 31B 22.6 16.7 72.6 37.9 428.0
MiniMax-M3 428B (int4) 17.5 11.8 108.1 30.4 477.1
GLM-4.6V 106B 20.2 13.5 107.3 41.5 322.7
Qwen3-VL-Thinking 32B 19.5 13.4 140.6 197.8 378.2
Lingshu (Medical Specialist) 32B 20.2 13.7 118.6 247.7 512.5
MedGemma (Medical Specialist) 27B 19.1 12.9 1905.0 8616.4 1024.8
Qwen2.5-VL (Base) 7B 16.7 11.3 7680.9 96542.0 724.9
BiomedParse (Specialist Seg.) Specialist 53.8 50.2 52.4 - -

In the Clinical Decision Agreement (CDA) evaluation (Table 19), predicted measurements were mapped onto standard clinical cutoff tables. MedVision-V0 achieved significant agreement beyond chance across all diagnostic scenarios: fair agreement on maxillary position SNA angle (\(\kappa_w = 0.332, p < 0.001\)), moderate agreement on mandibular position SNB angle (\(\kappa_w = 0.439, p < 0.001\)), and substantial agreement on AJCC 8th Edition kidney tumor staging (\(\kappa_w = 0.726, p < 0.001\)). In contrast, off-the-shelf models collapsed towards single arbitrary staging categories due to catastrophic scale drift (e.g., InternVL3 placed 99.1% of predictions into stage T2b because of massive overestimation).

Ablation Study

Ablation experiments evaluate the stage-wise impact of SFT and RFT, compare sequential versus multi-task curriculum RFT, and benchmark reward composition variants (data directly quoted from Tables 14, 16, and 17):

Stage / Configuration Anatomy Det. F1 (%) ↑ T/L Det. F1 (%) ↑ T/L Size MRE (%) ↓ Distance MRE (%) ↓ Angle MRE (%) ↓ Note
Qwen2.5-VL (7B Base) 16.7 6.5 7680.9 96542.0 724.9 Uncalibrated zero-shot baseline; spatial collapse
+ SFT (CoT) 78.3 (+61.6) 44.1 (+37.6) 28.6 (-7652.3) 7.5 (-96534.5) 53.3 (-671.6) Instills structured coordinate reasoning & parsing
+ SFT-RFT (MedVision-V0) 79.1 (+0.8) 46.9 (+2.8) 26.0 (-2.6) 6.4 (-1.1) 52.1 (-1.2) Full sequential RFT pipeline with process supervision
Multi-Task Curriculum RFT 78.2 47.4 26.7 6.6 41.9 Single-stage multi-task RFT; superior angle accuracy
Reward: Additive (\(r_{\text{add}}\)) - - 26.9 6.7 48.1 Additive reward decouples intermediate steps from result
Reward: Multiplicative (\(r_{\text{mul}}\)) - - 26.7 6.6 41.9 Multiplicative reward enforces CoT logical consistency

Key Findings

  • Systemic scale hallucination in off-the-shelf VLMs: Pretrained open-weight models—regardless of scale or domain-specific pretraining—exhibit mean relative measurement errors exceeding 100% (and often reaching tens of thousands of percent). These models completely fail to convert image-space tokens into grounded physical dimensions.
  • Visual perception, not arithmetic, is the primary bottleneck: Diagnostic experiments using a tool-augmented variant (MedVision-V0-tool, Table 2), which executes exact Python arithmetic for all intermediate calculations, eradicated arithmetic calculation errors (reducing MRE from 3.4% and 12.8% to 0.0%) but yielded negligible improvements in final measurement MRE (improving by only 0.3% to 2.1%). This proves that spatial landmark localization and visual boundary perception are the true limiting bottlenecks in medical quantitative reasoning.
  • Robust OOD plane and target generalization: Despite being trained solely on axial slices, MedVision-V0 successfully generalizes to held-out coronal and sagittal imaging planes (Plane-OOD T/L size MRE of 36.4%) and completely unseen anatomical targets (Target-OOD T/L size MRE of 34.7%), consistently outperforming SFT-only baselines.

Highlights & Insights

  • Pioneering physical scale integration in medical multimodal benchmarking: Moves beyond traditional qualitative captioning and VQA by compiling a standardized ground truth of 24.3M physical measurements across 22 datasets, providing the first systematic quantitative testbed for medical VLMs.
  • Multiplicative process rewards enforce verifiable reasoning traces: By constructing \(r = r_{\text{format}} + r_{\text{process}} \cdot r_{\text{answer}}\), the optimization landscape actively prevents models from receiving credit when final answers match ground truth purely by chance while intermediate spatial coordinates are severely erroneous.
  • Validation against certified clinical staging standards: Demonstrates through quadratic-weighted Cohen's \(\kappa_w\) that continuous numerical precision gains directly translate into clinically meaningful agreement on Steiner cephalometric assessments and AJCC renal cell carcinoma staging.

Limitations & Future Work

  • Specialization trade-off vs. broad qualitative generation: MedVision-V0 is optimized as a quantitative specialist; exploring unified architectures that simultaneously retain conversational question-answering and narrative reporting while mastering physical measurement remains an open challenge.
  • Degradation on micro-structures and ill-defined margins: Performance drops sharply on targets occupying less than 5% of image area (box-image-ratio \(< 0.05\)) and diffuse infiltrative lesions (e.g., non-enhancing brain edema and metastases), necessitating high-resolution localized foveation mechanisms.
  • 2D slice approximation of 3D volumetric anatomy: The current benchmark focuses on 2D planar projections; developing native 3D volumetric token representations to capture full 3D tumor volumes and volumetric angulation represents a vital next step.
  • vs RadGPT / RadBench / GMAI-MMBench: Prior benchmarks focus heavily on qualitative diagnostic reports or categorical classification. RadGPT extracts tumor sizes exclusively from text reports in abdominal CT, whereas MedVision provides direct pixel-level ground-truth physical annotations across 22 diverse datasets and five imaging modalities.
  • vs BiomedParse: BiomedParse is a task-specific segmentation foundation model requiring dense mask prediction followed by morphological post-processing. MedVision-V0 demonstrates that an end-to-end 7B VLM with CoT reasoning can directly surpass BiomedParse on both detection (F1 79.1% vs. 53.8%) and lesion size estimation (MRE 26.0% vs. 52.4%).
  • vs MedVLM-R1: MedVLM-R1 uses reinforcement learning to incentivize diagnostic reasoning on multiple-choice questions; MedVision-V0 extends reinforcement learning with process rewards to regression-based continuous geometric and physical coordinate estimation.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ First systematic benchmark targeting quantitative physical measurement in medical VLMs, featuring multiplicative process rewards and curriculum GRPO.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluates 18 foundation models across 22 multi-modal datasets, complete with error decomposition, OOD validation, and clinical decision agreement.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exemplary mathematical clarity, rigorous definition of geometric prompts, and exceptionally detailed appendix documentation.
  • Value: ⭐⭐⭐⭐⭐ Fills a profound gap between qualitative multimodal AI and clinical decision-making, providing open data, code, and model baselines.