Skip to content

Can MLLMs Perceive and Describe Medical Image Quality Beyond Scores?

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/liujiyaoFDU/MedQ-Bench
Area: Medical NLP
Keywords: Medical Image Quality Assessment, Multimodal Large Language Models, MedQ-Bench, Perception-Description Paradigm, Low-Level Vision

TL;DR

MedQ-Bench establishes the first comprehensive multimodal benchmark for language-based medical image quality assessment across 5 modalities, 40+ degradation types, and 3,308 images. Evaluating 23 frontier MLLMs reveals substantial gaps against human experts, identifying severe performance drops in mild degradations and highlighting a pervasive diagnostic bias where medical-specialized models overlook catastrophic imaging artifacts.

Background & Motivation

Medical Image Quality Assessment (IQA) serves as an essential gatekeeper for clinical safety, determining whether acquired diagnostic scans satisfy rigorous technical and perceptual criteria for downstream physician interpretation and automated algorithmic workflows. Despite its critical clinical role, existing medical IQA methodologies remain overwhelmingly shackled to the scalar scoring paradigm. Conventional frameworks comprise either no-reference (NR) statistical feature extraction models (such as BRISQUE, NIQE, CNNIQA, and MUSIQ) or full-reference (FR) fidelity metrics (such as PSNR, SSIM, and LPIPS). By compressing complex physical degradation states into solitary numerical scalar values, these conventional metrics fail to replicate authentic clinical diagnostic workflows: when assessing an examination, radiologists do not generate abstract numbers; instead, they execute a multi-stage cognitive analysis—identifying specific physical artifacts (e.g., metal streak artifacts, phase-encoding motion, or quantum mottle), evaluating the spatial occlusion of critical anatomical landmarks, and weighing the diagnostic impact of these degradations against clinical utility. Numerical metrics fundamentally lack explainability and display fragile generalization across heterogeneous modalities, frequently exhibiting pathological failures such as rewarding unphysiologically oversmoothed reconstructions that erase critical micro-textures.

The recent emergence of Multimodal Large Language Models (MLLMs) offers a transformative opportunity to overcome the limitations of scalar scoring in medical IQA. In principle, MLLMs possess the capacity to decompose image quality evaluation into structured chains of thought (CoT): localizing low-level defects, determining underlying acquisition or algorithmic failure mechanisms, estimating severity levels, and synthesizing descriptive, expert-aligned quality reports. However, fundamental scientific questions remain unanswered regarding the actual capabilities of MLLMs in specialized medical imaging domains: Can general-purpose or medical-specialized models genuinely generalize across fine-grained physical degradations spanning diverse imaging physics? Are their descriptive evaluations grounded in robust low-level perceptual discrimination, or do they merely produce hallucinated explanations through linguistic pattern matching? While existing MLLM quality benchmarks evaluate natural scene degradations (such as Q-Bench and Q-Instruct) or high-level clinical reasoning (such as GMAI-MMBench and Med-XpertQA), there is no systematic benchmark dedicated to assessing low-level perceptual attributes and explanatory quality descriptions across clinical imaging modalities.

To bridge the gap between scalar metrics and clinical quality workflows, this work introduces a novel perception–description paradigm mirroring expert radiological evaluation, formalized into the first comprehensive medical IQA benchmark, MedQ-Bench. Core idea: reconstruct medical image quality assessment into decoupled low-level visual perception and clinical explanatory description tracks across 5 imaging modalities and 40+ degradation types, evaluated via an expert-aligned four-dimensional LLM-as-a-judge protocol to benchmark 23 frontier MLLMs and diagnose their failure modes.

Method

Overall Architecture

The MedQ-Bench evaluation framework does not propose a single network architecture; instead, it establishes an end-to-end evaluation methodology that closely mirrors radiological quality control practices. The pipeline comprises three tightly coordinated components: multi-source data collection covering diverse physical acquisition degradations, decoupled perception and description evaluation tracks, and an expert-calibrated four-dimensional LLM judging protocol. The flow begins by gathering authentic clinical scans, physics-based simulations, and AI-generated reconstructions across 5 medical modalities; these images are routed through the MedQ-Perception track (probing fundamental low-level visual discrimination via multiple-choice queries) and the MedQ-Description track (evaluating open-ended single-image reporting and paired comparison reasoning); finally, an automated judge model scores the generated natural language descriptions along four calibrated axes.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-Modal Medical Image Inputs<br/>CT / MRI / Histopathology / Endoscopy / Fundus"] --> B["Multi-Modal & Tri-Source Dataset Construction<br/>Authentic Clinical + Physical Simulation + AI-Generated"]
    B --> C["Decoupled Dual-Track Evaluation Tasks<br/>MedQ-Perception + MedQ-Description"]
    C --> D["Expert-Aligned Four-Dimensional Judging Protocol<br/>Completeness + Preciseness + Consistency + Quality Accuracy"]
    D --> E["Comprehensive Multi-Model Diagnostic Analysis<br/>23 Frontier MLLMs Evaluated Zero-Shot"]

Key Designs

1. Multi-Modal and Tri-Source Dataset Construction: Comprehensive physical degradation coverage across 5 imaging modalities

Existing medical quality studies typically concentrate on a single modality (e.g., fundus or CT), failing to account for the heterogeneous physical acquisition processes that govern clinical imaging. MedQ-Bench spans 5 core clinical modalities: Computed Tomography (CT), Magnetic Resonance Imaging (MRI), Endoscopy, Histopathology, and Fundus photography, aggregating 3,308 standardized 2D RGB slices annotated across more than 40 distinct degradation attributes. To capture both clinical authenticity and controlled artifact parameters, data acquisition employs a tri-source strategy: - Authentic clinical cases (41.3% of dataset): Collected from public repositories (Radiopaedia, MR-ART, EndoCV2020, EyeQ) and institutional ultra-high-field 5T MRI scanners under strict de-identification protocols, capturing organic patient motion, structural bone artifacts, and diverse patient demographics; - Physics-based synthetic simulations (33.9% of dataset): Accurately generated according to imaging physics principles. For CT, AAPM CT-MAR reconstructions simulate limited-angle, sparse-view, and metal-induced beam hardening artifacts via SIRT, FBP, and FISTA algorithms; for MRI, SigPy and TorchIO implement zero-space diffusion (DDNM) and wavelet reconstruction to generate accelerated sub-Nyquist undersampling and phase-encoding motion artifacts; - AI-generated algorithmic distortions (24.8% of dataset): Leverages medical domain-adapted diffusion models (BAGEL) and artifact restoration frameworks (HARP) to introduce subtle hallucinated anatomical boundaries, texture loss, and contrast inconsistencies induced by machine learning pipelines.

2. Decoupled Dual-Track Evaluation Tasks: Separating low-level visual perception from clinical explanatory description

To disentangle whether an MLLM fails due to blindness to low-level visual artifacts or deficiencies in clinical linguistic reasoning, MedQ-Bench establishes two complementary evaluation tracks: - MedQ-Perception Track (2,600 multiple-choice questions): Directly evaluates primitive visual perception without requiring complex diagnostic explanations. Queries are categorized into three formats: Yes-or-No (binary artifact presence), What (multi-class artifact classification), and How (severity level grading). Crucially, the benchmark spans two orthogonal axes: Axis 1 divides samples into No Degradation (23.8%), Mild Degradation (44.6%), and Severe Degradation (31.6%); Axis 2 decouples General Medical Questions (42.8%, testing generic clarity/noise) from Modality-Specific Questions (57.2%, requiring precise knowledge of imaging physics such as susceptibility artifacts or aliasing). - MedQ-Description Track (708 expert-annotated samples): Evaluates autonomous natural language generation across two sub-tasks: (1) No-reference description, where the model generates a structured 3-5 sentence report detailing anatomical localization, degradation identification, underlying physical attribution, diagnostic impact, and an explicit clinical usability verdict (Good / Usable / Reject); (2) Comparison description, where image pairs \((I_A, I_B)\) are compared to determine preference \(P(I_A \succ I_B)\) across coarse-grained (obvious contrast/blur differences) and fine-grained (subtle noise, preservation of delicate organ boundaries) regimes.

3. Expert-Aligned Four-Dimensional Judging Protocol: Validated automated evaluation for open-ended clinical descriptions

Evaluating free-form diagnostic text via traditional n-gram matching metrics (BLEU, ROUGE) fails to capture semantic correctness, visual grounding, and logical reasoning. MedQ-Bench establishes an automated LLM-as-a-judge protocol powered by GPT-4o, evaluating descriptions across four distinct dimensions on a three-tier ordinal scale (0, 1, 2) yielding a total score out of 8: - Completeness (0-2): Evaluates whether the generated text identifies all salient low-level visual attributes present in the reference description; - Preciseness (0-2): Penalizes factual contradictions against visual reality (e.g., claiming an image is motion-free when ghosting artifacts are present) while allowing minor stylistic variations; - Consistency (0-2): Measures logical alignment between the detailed description of degradation symptoms and the final quality verdict (e.g., penalizing descriptions that detail severe metal streaking but conclude the image is "good"); - Quality Accuracy (0-2): Measures the ordinal alignment of the final quality judgment against ground truth along the hierarchy Reject < Usable < Good, awarding 2 points for an exact match, 1 point for an adjacent error, and 0 points for complete mismatch.

To guarantee fidelity and eliminate judge bias, 200 randomly sampled cases underwent double-blind evaluation by three board-certified medical imaging specialists. Evaluating ordinal ratings using quadratic weighted Cohen's \(\kappa_w\): $\(\kappa_w = 1 - \frac{\sum_{i,j} w_{ij} O_{ij}}{\sum_{i,j} w_{ij} E_{ij}}, \quad \text{where} \quad w_{ij} = \frac{(i - j)^2}{(k - 1)^2}\)$ Empirical validation demonstrated exceptional human-judge agreement across all axes: Completeness reached 83.3% accuracy (\(\kappa_w = 0.774\)), Preciseness reached 87.0% accuracy (\(\kappa_w = 0.876\)), Consistency reached 90.5% accuracy (\(\kappa_w = 0.840\)), and Quality Accuracy reached 98.0% accuracy (\(\kappa_w = 0.985\)). Multi-judge validation against Claude-4-Sonnet and Gemini-2.5-Pro confirmed inter-judge Intraclass Correlation Coefficients (ICC) exceeding 0.95 and mean absolute deviations below 0.05, validating the absence of self-preference bias.

Benchmark & Evaluation Protocol

The benchmark evaluates 23 state-of-the-art MLLMs across four categories: general open-source models (Qwen2.5-VL variants, InternVL3 models, Qwen3-VL, Qwen3.5), quality-oriented models (Q-Instruct, Q-Insight, VisualQuality-R1), medical-specialized models (BiMediX2, Lingshu, MedGemma), and commercial frontier models (GPT-5, GPT-5.4, GPT-4o, Claude-4-Sonnet, Claude-Sonnet-4.6, Gemini-2.5-Pro, Gemini-3.1-Pro, Grok-4). All evaluations are executed under a strict zero-shot setting using the unified VLMEvalKit harness on NVIDIA A100 (80GB) GPUs. API-based models use greedy decoding (temperature = 0, \(n=1\)), while open-source models operate with top-\(k=1\), top-\(p=0.001\), and temperature = 0.01 to ensure deterministic, reproducible benchmarking.

Key Experimental Results

Main Results: MCQA Perception and No-Reference Description

The table below presents the quantitative evaluation across 23 representative models on the MedQ-Perception and MedQ-Description (No-Reference) tracks:

Model Category Model Variant Perception: Y/N↑ (%) Perception: What↑ (%) Perception: How↑ (%) Perception Overall↑ (%) Desc: Comp.↑ (0-2) Desc: Prec.↑ (0-2) Desc: Cons.↑ (0-2) Desc: Qual.↑ (0-2) Desc Overall↑ (0-8)
Human Baselines Random Guess 50.00 28.48 33.30 37.94 - - - - -
Non-experts 67.50 57.50 57.50 62.50 - - - - -
Human Experts 88.50 77.50 77.50 82.50 1.524 1.483 1.887 1.623 6.502
Open-Source General Qwen2.5-VL-7B 57.89 48.45 54.40 54.71 0.715 0.670 1.855 1.127 4.367
Qwen2.5-VL-32B 67.38 43.02 58.69 59.31 1.077 0.928 1.977 1.290 5.272
Qwen2.5-VL-72B 78.67 42.25 56.44 63.14 0.905 0.860 1.896 1.321 4.982
InternVL3-8B 72.04 47.67 52.97 60.08 0.928 0.878 1.858 1.317 4.983
InternVL3-38B 69.71 57.36 52.97 61.00 0.964 0.824 1.860 1.317 4.965
Qwen3-VL-235B 55.44 67.82 66.27 62.45 0.946 0.720 1.629 1.471 4.765
Qwen3.5-397B 65.78 67.17 65.46 66.67 0.923 0.733 1.396 1.466 4.518
Quality-Oriented Q-Instruct 78.32 36.43 46.22 58.01 0.398 0.330 1.498 1.167 3.394
Q-Insight 65.05 45.74 61.35 59.85 0.597 0.502 0.340 0.977 2.416
VisualQuality-R1 61.65 46.12 55.01 56.09 0.747 0.665 1.783 1.172 4.367
Medical-Specialized BiMediX2-8B 44.98 27.52 27.81 35.10 0.376 0.394 0.281 0.670 1.721
MedGemma-27B 67.03 48.06 50.72 57.16 0.742 0.471 1.579 1.262 4.054
Lingshu-32B 50.36 50.39 51.74 50.88 0.624 0.697 1.932 1.059 4.312
Commercial Frontier Claude-4-Sonnet 71.51 46.51 54.60 60.23 0.742 0.633 1.778 1.376 4.529
Claude-Sonnet-4.6 66.67 59.83 58.63 62.99 1.109 0.873 1.498 1.489 4.968
Gemini-2.5-Pro 75.13 55.02 50.54 61.88 0.878 0.891 1.688 1.561 5.018
Gemini-3.1-Pro 66.13 58.32 64.26 63.07 0.765 0.760 1.647 1.557 4.729
Grok-4 73.30 48.84 59.10 63.14 0.982 0.846 1.801 1.389 5.017
GPT-4o 78.48 49.64 57.32 64.79 1.009 1.027 1.878 1.407 5.321
GPT-5.4 64.35 68.03 67.87 66.59 0.977 0.783 1.620 1.453 4.833
GPT-5 82.26 60.47 58.28 68.97 1.195 1.118 1.837 1.529 5.679

Paired Comparison Description and Perceptual Modality Breakdown

The following table details paired comparison description performance across fine-grained vs. coarse-grained difficulty (Table 9), alongside perception accuracy across imaging modalities (Table 8):

Model Variant Paired Overall↑ Paired: Fine-grained↑ Paired: Coarse-grained↑ CT Acc.↑ (%) MRI Acc.↑ (%) Endoscopy Acc.↑ (%) Histology Acc.↑ (%) Fundus Acc.↑ (%)
Human Experts 7.003 - - - - - - -
GPT-5 6.338 6.175 6.900 71.47 75.90 60.89 65.43 70.09
Kimi-K2.5 6.293 - - - - - - -
Qwen3.5-397B 6.015 - - - - - - -
GPT-4o 5.714 5.660 5.900 72.85 64.75 60.44 58.33 66.67
Grok-4 5.662 5.883 4.900 70.14 65.93 64.49 59.37 61.40
Gemini-2.5-Pro 5.594 5.485 5.967 67.04 60.79 60.89 53.40 59.83
InternVL3-8B 5.534 5.233 6.567 60.66 61.87 67.56 51.54 63.25
Claude-4-Sonnet 5.331 5.320 5.367 64.27 55.04 65.33 54.63 65.81
Lingshu-32B 4.654 4.437 5.400 57.89 61.15 50.67 35.19 48.72
Qwen2.5-VL-72B 4.060 3.602 5.633 65.65 74.82 66.22 47.53 64.96
MedGemma-27B 3.504 3.087 4.933 66.57 57.55 56.00 46.60 59.83
BiMediX2-8B 2.173 1.689 3.833 41.99 44.36 38.74 23.38 47.62

Key Findings

  • Substantial Human-AI Gap and Category Hierarchy: Certified medical imaging specialists attain 82.50% overall perception accuracy and a 6.502 description score. While GPT-5 establishes state-of-the-art AI performance (68.97% perception, 5.679 description), it trails human specialists by 13.53% in perception and remains far from saturated; nevertheless, it outperforms non-expert humans (62.50%). Closed-source commercial frontier models systematically outperform open-source models, with Qwen3.5 (397B) representing the strongest open-source contender (66.67% perception, 6.015 paired description).
  • The Catastrophic Failure of Medical-Specialized Models ("Diagnostic Bias"): Domain-adapted medical MLLMs suffer a pervasive failure mode. BiMediX2-8B achieves an abysmal 35.10% on MCQA perception—underperforming random guessing (37.94%)—and scores only 1.721 in description. Qualitative case analysis (Figure 10) uncovers a severe safety hazard: when examining an abdominal CT scan destroyed by intense metal streak artifacts that mandate clinical rejection, BiMediX2-8B confidently declares the liver and anatomy to be "well-defined, with good contrast and minimal noise, suitable for diagnosis (good)." Medical fine-tuning datasets predominantly train models on diagnostic report generation, inducing a severe bias that prioritizes anatomical entity recognition while entirely blinding the models to catastrophic physical degradations.
  • Quality-Oriented Models Fail to Transfer to Medicine: Natural IQA models (Q-Instruct, Q-Insight, VisualQuality-R1) transfer poorly to medical quality tasks (description scores: 2.416 - 4.367). General vision-language quality pretraining on photographic distortions (Gaussian blur, color jitter) does not equip models to comprehend medical acquisition physics, such as k-space undersampling, beam hardening, or tissue cut artifacts.
  • Mild Degradations Represent a Perceptual Blind Spot: Across degradation severity tiers, models average 72% accuracy on pristine images and 67% on severe degradations, but accuracy collapses to 56% under mild degradations. Subtle, clinically consequential degradations (e.g., slight motion blurring or low-dose noise) routinely escape detection by current vision encoders.
  • Optical and Microscopic Modalities Present the Highest Error Rates: Modality error breakdown for GPT-5 (Table 10) reveals that Endoscopy (39.11% error rate) and Histopathology (34.57% error rate) are substantially harder than CT (25.76%) and MRI (24.10%). At the artifact level, Air Bubbles (95.23% error rate), Specular Reflections (90.48%), and Blood Stains (84.62%) cause near-total failure due to intense high-frequency textural interference with cellular or mucosal backgrounds.
  • "Logically Fluent but Perceptually Empty": Across nearly all MLLMs, internal Consistency scores remain high (1.7 - 1.9 out of 2.0), demonstrating strong instruction-following capabilities. In contrast, Completeness (0.7 - 1.2) and Preciseness (0.6 - 1.1) scores are severely depressed, confirming that the primary bottleneck in medical IQA is low-level perceptual discrimination rather than linguistic reasoning.

Highlights & Insights

  • Establishment of the Perception-Description Paradigm: The benchmark moves beyond uninterpretable scalar metrics, introducing a clinical reasoning framework that decouples low-level perceptual attribute identification from open-ended causal description and paired comparison.
  • Physics-Grounded Tri-Source Dataset Construction: Rather than assembling web-scraped images, the authors incorporate rigorous physics-based artifact simulations (AAPM CT-MAR, SigPy/TorchIO k-space simulations) alongside authentic 5T clinical scans and generative reconstructions across 5 modalities, establishing 3,308 diverse cases.
  • Uncovering the Systematic Diagnostic Bias in Medical AI: The study exposes a critical vulnerability in current clinical AI: medical fine-tuning inadvertently trains models to overlook severe physical corruption in favor of diagnostic assertiveness, underscoring the necessity of dedicated visual quality gatekeeping.
  • Four-Dimensional LLM-as-a-Judge with Rigorous Human Calibration: Utilizing quadratic weighted Cohen's \(\kappa_w\), the evaluation protocol achieves exceptional alignment with three board-certified imaging specialists (\(\kappa_w\) between 0.774 and 0.985), establishing a reliable, automated, and scalable evaluation protocol.

Limitations & Future Work

  • Authors' Acknowledged Limitations: Although the benchmark encompasses 5 major imaging modalities, several critical clinical imaging techniques—notably B-mode ultrasound and fluoroscopic radiography—are omitted. Furthermore, the evaluation is conducted exclusively under a zero-shot setting, leaving the performance upper bound achievable through domain-specific supervised fine-tuning (SFT) or test-time adaptation unexplored.
  • Independent Critical Limitations: While validated against expert consensus, the open-ended text evaluation relies on GPT-4o, which may carry latent stylistic biases on lengthy descriptions. Additionally, medical imaging datasets are formatted as 2D slices, which does not address volumetric spatial continuity in 3D multi-slice CT/MRI series.
  • Concrete Improvement Directions: Extending the benchmark to full 3D volumetric tensor inputs, and developing multi-task reinforcement learning frameworks (e.g., RLHF/RLAIF) that penalize models for outputting clinical diagnoses when severe acquisition artifacts are present.
  • vs Traditional Scalar IQA (BRISQUE, NIQE, CNNIQA, MUSIQ, PSNR, SSIM, LPIPS): Traditional methods produce isolated scalar scores lacking explainability and failing to capture physical degradation causes; MedQ-Bench generates full descriptive chains of thought aligned with radiological practice.
  • vs Natural Image MLLM Quality Benchmarks (Q-Bench, Q-Instruct, DepictQA): Natural IQA benchmarks focus on photographic distortions (lens defocus, JPEG artifacts); MedQ-Bench targets physics-governed medical artifacts (beam hardening, undersampling, histopathological folding).
  • vs High-Level Medical Multimodal Benchmarks (GMAI-MMBench, PMC-VQA, OmniMedVQA): Existing medical benchmarks focus exclusively on downstream clinical diagnoses and disease VQA; MedQ-Bench addresses the neglected upstream visual quality control gatekeeper necessary for safe clinical deployment.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Pioneering perception-description paradigm that reframes medical IQA from scalar scoring to structured visual-linguistic reasoning.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive zero-shot evaluation of 23 diverse MLLMs across 5 modalities and 40+ degradations, supported by extensive human expert validation.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally articulate prose, rigorous mathematical definitions, clear conceptual taxonomy, and insightful error analysis.
  • Value: ⭐⭐⭐⭐⭐ Uncovers critical diagnostic biases in specialized medical models and establishes an indispensable quality control benchmark for trustworthy clinical AI.