Solve the Missing First Step: Can VLMs Standardize Uncalibrated Clinical Images?¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: MDS-Bench
Area: Medical NLP
Keywords: Medical Vision-Language Models, Raw Medical Data Standardization, Agent Benchmark, Multimodal Structure Extraction, Clinical Data Curation
TL;DR¶
Presents MDS-Bench, the first comprehensive benchmark encompassing 100 diverse medical datasets and 1,939 verified target samples to assess agentic VLMs in discovering, converting, and standardizing raw heterogeneous clinical files into VLM-ready image-text atomic units, showing that even top frontier models like Gemini 3 Flash achieve an end-to-end strict success rate of only 48.6%.
Background & Motivation¶
With the rapid maturation of vision-language models (VLMs), building multimodal foundation systems capable of reasoning over complex clinical workflows has emerged as a major goal of medical AI. However, prevailing medical AI evaluation benchmarks—including MultiMedQA, BioASQ, PMC-VQA, and GMAI-MMBench—predominantly focus on evaluating diagnostic reasoning, visual question answering, or report generation on well-curated, isolated multimodal inputs. These benchmarks implicitly assume that diagnostic images have already been correctly windowed and cropped, paired reports have been segmented, and question-answer tuples have been carefully assembled. While this formulation cleanly isolates post-processing clinical reasoning, it entirely bypasses the messy, error-prone, and labor-intensive upstream bottleneck: discovering, parsing, and standardizing raw clinical data as it actually exists in real healthcare systems.
In authentic clinical archives, PACS repositories, and open medical registries, data rarely adhere to a singular, clean organization schema. A single patient case or clinical sample typically involves a web of interrelated artifacts scattered across heterogeneous file types—such as raw DICOM series, volumetric NIfTI scans, gigapixel pathology TIFFs, endoscopy video clips, standalone CSV label spreadsheets, and fragmented JSON metadata. The vital associative links among visual volumes, region-of-interest masks, and clinical diagnoses are often buried implicitly in operating system directory hierarchies, cryptic file-naming conventions, or loose documentation. While agentic frameworks such as MedPerf and MedAgentBench emphasize practical evaluation workflows, they treat clean data preparation as a given prerequisite rather than an active evaluative capability; if this upstream step fails, all downstream clinical decision-making is compromised by invalid or mismatched visual inputs.
This paper bridges this fundamental gap by studying raw medical data standardization within an agentic VLM framework, where a model is situated in an interactive coding-agent environment equipped with file-system inspection tools and script execution capabilities. The core idea is to establish MDS-Bench, the first standardized benchmark encompassing 100 heterogeneous medical datasets and 1,939 verified samples across diverse modalities and tasks, paired with a rigorous eleven-metric deterministic evaluation protocol that holistically quantifies whether VLMs can transform messy clinical directory trees into unified, source-grounded image-JSON atomic representations.
Method¶
Overall Architecture¶
The raw medical data standardization task requires an agentic VLM, given an uncurated dataset root directory and a target JSON schema, to autonomously discover relevant source evidence, convert raw clinical images into unified visual representations, align scattered annotations and metadata, and emit structured outputs. The resulting standardized atomic unit comprises three indispensable components: ① a standardized visual image (converted from multi-slice volumes or specialized formats into standard formats readable by downstream vision encoders); ② a per-image JSON record (capturing sample provenance, primary modality, anatomical site, dimensions, and task-specific payloads such as bounding boxes, class labels, or segmentation masks); and ③ a dataset-level metadata JSON (summarizing global label spaces, modality attributes, and cohort statistics).
To handle the complexity of heterogeneous files, the agent operates through a staged reasoning workflow that coordinates file exploration, code execution, and multimodal alignment:
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Raw Dataset Directory<br/>DICOM/NIfTI/TIFF/Tables/Folders"] --> B["Source Identification & Grounding<br/>Distinguish visual sources from tables"]
B --> C["Visual Input Standardization<br/>Extract slices, render formats, preserve fidelity"]
C --> D["Schema-Constrained Attribute Extraction<br/>Infer modality, align labels/masks/boxes"]
D --> E["Cross-Level Consistency Checking<br/>Verify image-JSON pairs and metadata"]
Key Designs¶
1. Source Identification & Grounding: Gated Evidence Tracing and Provenance Verification
Raw medical datasets frequently contain auxiliary or decoy files, such as binary segmentation masks, pre-computed heatmaps, or clinical text tables, which models easily confuse with primary visual inputs. The agent must first traverse the directory structure, consult the sample manifest, and identify the candidate source files \(A_{\text{pred}}\). To ensure that downstream scores reflect genuine comprehension rather than hallucinations, the evaluation protocol establishes a strict Source Matching Gate \(s_i\):
$\(s_i = \mathbb{I}(A_{\text{pred}, i} \cap A_{\text{gt}, i} \neq \emptyset \land t_i = 1)\)$
where \(t_i\) denotes whether unambiguous source-trace provenance evidence is explicitly provided. If \(s_i = 0\), all subsequent field-level semantic and content scores are set to zero, preventing models from earning partial credit on ungrounded predictions.
2. Visual Input Standardization: Transforming Complex Clinical Volumetrics to Common Visual Inputs
Medical imaging relies heavily on non-standard, multidimensional formats (e.g., 3D .nii.gz volumes, multi-frame .dcm files, .mat arrays, or pyramidal .tiff whole-slide images) that cannot be ingested directly by conventional visual transformers. The agent is empowered to execute Python code (using libraries such as SimpleITK, nibabel, or OpenCV) to adjust Hounsfield unit windowing, extract target cross-sectional slices, or render microscopic fields into standard visual formats. Crucially, this step is optimized not for superficial visual aesthetics, but to preserve the diagnostic morphological patterns and contrast characteristics necessary for downstream clinical inference.
3. Schema-Constrained Attribute Extraction: Multimodal Semantic Disambiguation and Payload Alignment
Once the visual evidence is prepared, the model reasons over directory paths, textual documentation, and image features to extract granular semantic attributes. It infers primary imaging modalities, dimensionalities, anatomical context, and task-specific branches (classification, detection, or segmentation). The agent aligns external labels and coordinates into the unified target schema while adhering to strict branch exclusivity (e.g., preventing spurious population of 3D or video blocks for 2D dermoscopy images). Missing or unverified properties must be left as null rather than guessed.
4. Cross-Level Consistency Checking: Deterministic Multi-Tier Evaluation and Strict End-to-End Metric
To prevent syntactically valid yet clinically inaccurate records from passing evaluation, the benchmark formulates eleven deterministic metrics grouped into five capability tiers: Structure (SV, SSC), Semantics (SC), Content (IC, IV, INR, CF), Metadata (MSC, MSJ), and Joint (SCJ, E2E). The strictest benchmark indicator, the End-to-End Strict Pass rate (E2E), is defined as:
$\(e_i = \mathbb{I}(p_i \land s_i=1 \land g_i \ge 0.85 \land q_i \ge 0.50 \land f_i \ge 0.50)\)$
$\(\text{E2E} = \frac{1}{N} \sum_{i=1}^N e_i\)$
where \(p_i\) enforces physical image readability and JSON path consistency, \(g_i\) measures the fraction of 14 deterministic schema checks passed, \(q_i\) assesses semantic correctness, and \(f_i\) computes exact-match and Jaccard content fidelity against human-verified references. This joint constraint ensures that any failure in the pipeline causes the entire sample to fail.
Benchmark & Evaluation Protocol¶
MDS-Bench curates 100 representative medical datasets from public repositories including Hugging Face, Kaggle, Zenodo, and TCIA, containing 1,939 verified target samples across classification (43 datasets), segmentation (51 datasets), and detection (6 datasets). The catalog covers 8 distinct clinical modalities: CT, MRI, Ultrasound, Microscopy/Pathology, X-ray, Ophthalmic Imaging, Endoscopy, and Mammography. Ground truth annotations were created via a model-assisted draft stage (using Gemini 3 Flash) followed by 100% rigorous verification by six computer science and medical researchers against the raw source evidence, with 20% of samples independently double-checked.
Nine frontier agentic models were evaluated under identical tool-access prompts: Claude Haiku 4.5, Grok 4.20, Kimi K2.5, GPT-5.2, GPT-5.2-Codex, Composer 1.5, Claude Sonnet 4.6, Claude Opus 4.6, and Gemini 3 Flash. All scores are macro-averaged across the 100 datasets to prevent large cohorts from skewing the conclusions.
Key Experimental Results¶
Main Results¶
The performance of the nine evaluated models across the eleven MDS-Bench metrics is summarized below (all figures are dataset macro-averaged percentages %):
| Model | SV (Struct. Valid) | SSC (Struct. Sem.) | SC (Sem. Correct) | IC (Info. Complete) | IV (Info. Valid) | INR (Non-Redund.) | CF (Content Fidel.) | MSC (Meta Sem.) | MSJ (Meta Joint) | SCJ (Source-Content) | E2E (Strict Pass) |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Haiku 4.5 | 82.2 | 49.8 | 49.1 | 47.5 | 41.5 | 42.5 | 40.3 | 56.8 | 34.1 | 38.0 | 24.9 |
| Grok 4.20 | 80.0 | 48.8 | 55.7 | 49.0 | 45.5 | 46.7 | 41.7 | 57.8 | 35.6 | 40.0 | 21.5 |
| Kimi K2.5 | 83.6 | 51.7 | 55.1 | 54.5 | 45.5 | 47.2 | 46.7 | 63.4 | 40.4 | 44.8 | 29.4 |
| GPT-5.2 | 81.6 | 52.8 | 60.2 | 50.0 | 37.5 | 47.3 | 47.7 | 74.6 | 43.4 | 45.4 | 23.6 |
| GPT-5.2-Codex | 83.7 | 53.3 | 57.0 | 55.5 | 49.0 | 48.1 | 47.1 | 65.6 | 40.6 | 44.8 | 28.1 |
| Composer 1.5 | 81.9 | 54.0 | 59.9 | 50.5 | 41.3 | 46.8 | 48.5 | 72.6 | 43.4 | 46.7 | 30.8 |
| Claude Sonnet 4.6 | 85.2 | 56.0 | 62.5 | 60.0 | 56.3 | 50.5 | 49.9 | 69.0 | 42.7 | 48.6 | 33.8 |
| Claude Opus 4.6 | 83.9 | 56.2 | 65.2 | 59.5 | 51.8 | 51.7 | 53.7 | 74.4 | 48.0 | 52.3 | 30.5 |
| Gemini 3 Flash | 88.2 | 62.0 | 63.9 | 67.5 | 62.5 | 58.7 | 58.7 | 78.0 | 53.7 | 56.3 | 48.6 |
Ablation Study: Inference Strategies and Validation Guidance¶
Using Gemini 3 Flash as the base engine, the authors evaluated five distinct test-time inference strategies: - S0 (Direct Generation): Base prompt execution without iterative refinement; - S1 (Self-Refinement): Post-generation self-critique by the same model against raw files; - S2 (Verification-Guided Refinement): Check-list driven refinement targeting high-risk attributes; - S3 (Best-of-n Complete Candidate Selection): Sampling multiple complete candidates and selecting the top-scoring one via a deterministic validator; - S4 (Field-Wise Selection): Selecting individual best fields across multiple candidates on top of a base candidate.
| Strategy Configuration | SV (%) | SC (%) | CF (%) | MSC (%) | SCJ (%) | E2E (%) | Description |
|---|---|---|---|---|---|---|---|
| S0 (Direct Baseline) | 88.2 | 63.9 | 58.7 | 78.0 | 56.3 | 48.6 | Direct zero-shot execution |
| S1 (Self-Refine) | 90.3 | 63.9 | 59.2 | 76.6 | 57.5 | 53.0 | Unconstrained self-critique (+4.4% E2E) |
| S2 (Guided-Refine) | 91.4 | 64.6 | 62.9 | 77.8 | 60.4 | 56.0 | Checklist targeting high-risk fields |
| S3 (Best Complete Candidate) | 94.8 | 69.1 | 67.2 | 78.6 | 66.3 | 63.5 | Best-of-n candidate selection (+14.9% E2E) |
| S4 (Field-Wise Selection) | 92.3 | 69.3 | 67.8 | 78.8 | 65.0 | 59.9 | Locally optimal field splicing hurts coherence |
Key Findings¶
- High Schema Validity Masks Severe Semantic Failures: All models readily satisfy JSON syntactic constraints, achieving SV scores between 80.0% and 88.2%. However, the Schema-Semantic Composite (SSC) plummets to 48.8%–62.0%, proving that generating syntactically valid JSON is a trivial prerequisite that poorly correlates with actual standardization success.
- Content Recovery and Joint Pipeline Consistency Form the Real Bottleneck: In dataset-level failure analysis (datasets scoring below 50%), only 1 to 13 datasets fail on Structure. In contrast, 39 to 67 datasets fail on Content Fidelity, and 51 to 78 datasets fail on Joint metrics. The primary failure modes stem from incorrect source grounding (e.g., picking
.matrather than.parquet), label permutation, or missing payload fields. - Complete Candidate Selection Beats Field-Wise Stitching: While S4 obtains slightly higher local field scores (CF 67.8%, SC 69.3%), its end-to-end success (E2E 59.9%) is distinctly lower than S3 (63.5%). Assembling individually optimal fields across candidates introduces subtle cross-field inconsistencies, whereas selecting an internally coherent complete candidate yields the most robust standardization.
- Modality-Specific Strengths Diverge Across Frontiers: Gemini 3 Flash achieves the highest overall modality stability (ranking 1st in CT, MRI, Ultrasound, Ophthalmic, and X-ray), but Claude Opus 4.6 dominates Microscopy/Pathology (63.4%), and Composer 1.5 leads on Mammography (62.9%), demonstrating that raw data standardization challenges vary across clinical imaging physics.
Highlights & Insights¶
- Exposing the "Missing First Step" in Clinical AI: While the medical AI community has intensely benchmarked diagnosis and reasoning, this work formalizes the unglamorous yet foundational challenge of data wrangling, demonstrating that data standardization cannot be taken for granted in real-world deployments.
- Deterministic, Embedding-Free Verification Matrix: By eliminating subjective LLM judges or embedding cosine similarities in favor of strict set operations, schema checks, and source trace verification, MDS-Bench offers a reproducible and rigorous evaluation standard for agentic systems.
- Inference-Time Validator as an Immediate Performance Multiplier: Without fine-tuning model parameters, incorporating a deterministic validator for Best-of-n selection (S3) elevates Gemini 3 Flash's E2E by 14.9 percentage points (and boosts Claude Haiku 4.5 by 20.7 percentage points), offering an actionable blueprint for practical clinical ETL pipelines.
Limitations & Future Work¶
- Reliance on Curated Public Challenge Repositories: Although the benchmark preserves raw folder layouts and messy formats, public challenge datasets remain cleaner and more structured than real-world on-premise hospital PACS archives and legacy EHR databases.
- Scope Constrained to Core Vision Tasks: The current benchmark focuses exclusively on classification, detection, and segmentation, leaving longitudinal progression tracking, clinical report generation, and multi-visit patient trajectories for future extensions.
- Absence of Stored Disagreement Logs in Human Annotation: Human annotators reached final ground truth through consensus meetings without archiving pre-resolution conflict logs, precluding retrospective statistical calculation of formal inter-annotator agreement metrics (e.g., Cohen's Kappa).
Related Work & Insights¶
- vs GMAI-MMBench / MultiMedQA / BioASQ: Existing medical benchmarks evaluate clinical reasoning on curated image-text inputs; MDS-Bench evaluates whether models can bridge the raw data gap to construct those curated inputs in the first place.
- vs StructEval / JSON-SchemaBench / SO-Bench: General structured output benchmarks test schema-following ability from clean prompts; MDS-Bench demonstrates that in domain-specific medical applications, schema compliance is trivial compared to the difficulty of source grounding and medical content fidelity.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates an overlooked upstream prerequisite for clinical AI and establishes a novel agentic benchmark.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ 100 datasets, 1,939 manually verified samples, 9 frontier models, and 11 fine-grained evaluation metrics.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally structured, with clear formal mathematical definitions, detailed error attribution, and rigorous ablation.
- Value: ⭐⭐⭐⭐⭐ Crucial foundation for deploying autonomous multimodal agents in messy healthcare data ecosystems.