Skip to content

MMBU: A Massive Multi-modal Biomedical Understanding Benchmark to Probe the Perception Capabilities of Vision-Language Models

Conference: ECCV 2026
Paper: ECCV 2026 / Hugging Face dataset
Area: Multimodal VLM (biomedical benchmark)
Keywords: Biomedical VLM, Multimodal Benchmark, Visual Perception, Visual Question Answering, Object Detection

TL;DR

MMBU uses 410 public datasets, 11 modalities (35 submodalities), 78K samples and 13 structured metadata attributes per sample to operationalize biomedical visual perception as a unified evaluation matrix of four task types (ungrounded classification, two forms of grounded classification, object detection) under open and closed answer formats; benchmarking 15 open-weight and 2 frontier VLMs shows that even the best result reaches only 0.693 F1 (closed, segmentation-grounded classification), the closed-to-open gap averages 0.26, no model beats the random baseline at object detection, and medical adaptation yields limited gains that fail to transfer to MMBU's out-of-distribution data.

Background & Motivation

Biomedical vision-language models (VLMs) are expected to take on front-line work ranging from lesion detection in chest X-rays to morphological description of cells in microscopy, and all of it rests on a shared prerequisite: fine-grained visual perception. A model must see subtle morphological differences, and it must see them consistently across patient populations, imaging modalities, biological scales, and acquisition conditions. The evaluation ecosystem is nowhere near this requirement. Specialized benchmarks such as PathVQA, VQA-RAD, and SLAKE each cover only 1-3 submodalities and a few thousand question-answer pairs, so they cannot measure within-domain generalization; multi-domain benchmarks such as OmniMedVQA and GMAI-MMBench raise the scale but remain centered on diagnostic imaging (radiology, pathology, endoscopy) with almost no coverage of the fundamental biological modalities—fluorescence microscopy, electron microscopy—that reveal cellular and molecular mechanisms. They are also largely human-centric and classification-only, offering few or no auxiliary tasks.

The harder problem lies in the data itself. Only about 20 datasets are used repeatedly in real evaluation practice, and the training sets behind them have frequently been absorbed into the pretraining or instruction-tuning pipelines of modern biomedical VLMs, creating severe distributional overlap between training and evaluation. The consequence is counterintuitive: a high score on an established benchmark may indicate that the model memorized the data rather than that it actually saw the image. Prior work (µ-Bench and follow-ups on the OmniMedVQA line) has repeatedly observed that medically adapted models are highly sensitive to prompt wording and batch effects—systematic shifts arising from different scanners, acquisition protocols, or institutions—and often fail to consistently outperform their general-domain base models. Existing benchmarks therefore lack not only scale but also attribution ability: without structured metadata there is no way to answer on which modality, specimen, or institution a model actually broke down.

This paper's angle is a thorough redo of both data collection and task definition. Now that large-scale retrieval and standardization are feasible—public repositories hold many biomedical datasets with clear licensing and human annotations, expert committees can define a unified taxonomy, and a discovery agent can fill taxonomy gaps in a targeted way—there is no reason to keep ranking models on a handful of contaminated datasets. The authors assembled a multidisciplinary team of clinicians, bioinformaticians, statisticians, and computer scientists to define a taxonomy of biomedical visual perception tasks, then curated, quality-controlled, and iteratively refined a unified benchmark on top of it. Core idea: instead of asking whether a model knows a piece of medical knowledge, explicitly operationalize perception as four visual tasks (ungrounded classification, mask-grounded classification, box-grounded classification, object detection) under a unified task taxonomy and two answer formats, then weave 410 heterogeneous datasets into one evaluation net through 13 structured metadata attributes so that every failure can be stratified by modality, submodality, specimen, domain, and institution.

Method

Overall Architecture

MMBU takes 410 public biomedical datasets as input (raw images, human labels, sparse metadata) and produces a perception profile for each VLM stratified by task type, imaging modality, submodality, medical/scientific domain, specimen, and acquiring institution. The pipeline has four stages: taxonomy-driven three-round data discovery and quality filtering, which shrinks the candidate pool from 122 → ~1.2K → 440 → 410; standardization of heterogeneous data into a common format and a common metadata schema, followed by four-step question construction ("metadata → question prototype → draft → human finalization"); expansion of the same annotations into 4 task types × 2 answer formats × 3 templates = 24 configurations; and finally a scoring protocol free of subjective judgment applied to 17 VLMs. The deliverable is not a leaderboard but a sliceable diagnostic report.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Public biomedical datasets"] --> B["Three-round discovery<br/>and quality filtering"]
    B --> C["Metadata standardization<br/>and question construction"]
    C -->|three templates: no context / modality / full context| D["Four perception tasks<br/>and open/closed setups"]
    D --> E["Deterministic extraction<br/>and semantic-equivalence judging"]
    E --> F["Stratified perception profile per VLM"]

Key Designs

1. Three-round discovery and quality filtering: turning "which datasets deserve to be evaluated" into a reproducible selection pipeline

Biomedical datasets are scattered across Zenodo, Kaggle, Papers with Code, and Hugging Face with wildly varying formats, licenses, and annotation quality; mixing them together at random makes both high and low scores uninterpretable. MMBU treats data selection itself as part of the method. A committee of two clinicians, two wet-lab specialists, and two bioinformaticians first defines a multi-granularity taxonomy over four iterations, covering medical and scientific specialties, imaging modalities and submodalities, anatomical entities, and biological context. Co-authors then nominate datasets under three inclusion criteria: publicly accessible with clear licensing, high-quality images (not blurry, subject clearly visible, no preprocessing artifacts), and human-generated labels only—datasets annotated by LLMs are excluded. Round one yields 122 datasets. In round two, a purpose-built discovery agent performs targeted retrieval across major public repositories to fill the categories the taxonomy flags as underrepresented; four annotators then cross-check every candidate against the original dataset description and drop any that fail the criteria, adding about 1.2K candidates. In round three, five experts remove entries with missing annotations, corrupted files, unclear licensing, or formatting errors, converging on 440 candidates and finally selecting 410 datasets (250 classification, 84 segmentation, 76 object detection).

One criterion in this pipeline is easy to overlook but decisive: contamination control. The authors deliberately exclude literature-derived datasets that are widely used to train medical VLMs, on the grounds that they very likely already sit inside the evaluated models' pretraining or instruction-tuning pipelines, where they would turn "memorized" into a false "seen." It also explains why absolute scores on MMBU are uniformly low—the benchmark measures out-of-distribution perception, not covered knowledge.

2. Metadata standardization and question construction: making heterogeneous datasets both comparable and leak-proof

The original datasets carry almost no usable structured metadata, and the same anatomical site may be named entirely differently across sources. MMBU first builds a standardization pipeline that chains downloading, metadata extraction with human correction, and image-format unification (TIFF, JPEG, DICOM, NIfTI, SVS, NDPI, MRXS, and OME-TIFF are all converted to PNG), with annotators verifying the extracted fields item by item. Question construction runs in four steps: collect fine-grained metadata from the image and source context (specimen, body part and subpart, modality, submodality, stain); define a question prototype conditioned on those metadata fields; instantiate a draft question targeting clinically meaningful attributes; and finalize it through human design and verification. Hard constraints are placed on wording—every question must be grounded in pixel-level visual content, and disease names, clinical outcomes, any term suggesting malignancy, and acronyms are all forbidden. That constraint is the crux of the paper: it converts the question from "do you know what disease this is" into "can you see the morphology of this cell," leaving vision as the only way to answer.

The same data is then rendered through three templates that form a controlled axis of "how much context is given." No Context is a standard template shared across all tasks with no context at all; Modality keeps the same structure but writes the imaging modality into the question; Full Context has annotators populate a predefined template with the metadata collected for that dataset, specifying specimen type (human tissue, fluid, smear), imaging modality (light microscopy, computed tomography, magnetic resonance imaging), stain or contrast agent where applicable (H&E, T1-weighted), preparation method (smear, section, biopsy), and anatomical site or compartment. These three levels answer a very practical question: did the model fail because it could not see, or because it did not know what modality it was looking at?

3. Four perception tasks under open and closed setups: turning "perception ability" into measurable capability dimensions

MMBU splits perception into four dimensions of increasing difficulty. Ungrounded classification shows only the whole image and asks for a label, probing global visual interpretation. Grounded classification supplies a region of interest, delimited either by a segmentation mask or a bounding box, and asks for a label for that region only, probing local discriminative power. Object detection gives the image and a target class and requires a bounding box, probing spatial localization. Each task has two answer formats: closed, where the model selects from a candidate pool, and open, where it generates the answer directly. The difference between them is recorded as a separate metric, \(\Delta = \text{F1}_{\text{closed}} - \text{F1}_{\text{open}}\), which quantifies how much a model relies on answer-choice cues instead of genuine visual understanding. Four tasks × two formats gives 8 configurations, and with three templates, 24 setups in total.

What makes this an operationalization rather than a mere list of tasks is that each dimension encodes a specific failure-mode hypothesis: weak ungrounded classification means global interpretation fails; a large drop from ungrounded to grounded classification means fine local detail is not resolved; collapsed detection means spatial localization is broken; a large \(\Delta\) means the model is guessing from the options. These dimensions are instantiated over 78K samples and 458 topics, each sample carrying 13 structured metadata attributes (image provenance, dataset name, domain, modality, submodality, stain, specimen, specimen subregion, topic, original task description, context, institution of acquisition, URL), so any failure can be traced back to a specific modality or institution.

4. Deterministic extraction and semantic-equivalence judging: scoring 78K answers without relying on a judge's mood

Closed-ended VQA is where a model most often answers correctly but in the wrong format. MMBU simply abandons LLM-as-a-judge here and uses a fully deterministic extraction and scoring protocol: a hierarchical parser scans model outputs for explicit answer fields, formatting tags, ordered lists, and valid option labels, then compares the extracted label against the ground-truth option set; responses with no parseable answer, or with labels outside the valid option set, count as incorrect. Classification correctness is determined by exact string match after normalization, and object detection requires an Intersection-over-Union (IoU) of at least 0.5 against the ground-truth box.

Open-ended VQA cannot use exact matching because synonymous biomedical terminology (say, the Latin and common names of the same structure) must both count, so Qwen3-32B serves as a semantic-equivalence judge. It receives only the model response and the ground-truth answer and is instructed to apply strict biomedical equivalence, explicitly checking negation, laterality, severity, uncertainty, and diagnostic specificity—"pneumonia" and "no pneumonia" are treated as non-equivalent. The headline metric is micro-averaged F1:

\[\text{F1}_{\text{micro}}=\frac{2\sum_{c}TP_c}{2\sum_c TP_c+\sum_c FP_c+\sum_c FN_c}\]

reported with 95% confidence intervals obtained via 1,000-iteration bootstrap resampling. Note that micro-averaged F1 shifts with candidate-pool size and class imbalance, which is why the random baselines differ across tasks (0.199 / 0.162 / 0.183 / 0.172 in the closed setting) and why comparing absolute F1 across tasks requires care.

A Worked Example

A bone-marrow smear question shows concretely how metadata-driven question construction runs. The original benchmark question is vague: "identify the content of the microscopic image." Four steps transform it. First, metadata is filled in from the image and source context—specimen is human bone marrow, body part is bone marrow smear, modality is light microscopy, submodality is brightfield microscopy, stain is May-Grünwald-Giemsa (Pappenheim). Second, a prototype is defined from those fields: "which {options_name} is the most consistent given the visual characteristics of {sub_modality}?" Third, a draft is instantiated with options naming myeloid lineage cell types—Monocyte, Promyelocyte, Proerythroblast, Band neutrophil, None of the Above. Fourth, the final human-verified question becomes "which myeloid lineage cell type or structure is most consistent with the morphological features observed in this bone marrow smear?" The finalized question contains no disease name and can only be answered from morphological features, and because the smear and stain are stated, an error can be attributed to the specific submodality "brightfield bone-marrow smear" rather than to the vague bucket of "medical QA got it wrong."

Key Experimental Results

Main Results

MMBU evaluates 15 open-weight and 2 frontier autoregressive VLMs, pairing every medically adapted model with its corresponding base model. The table below summarizes micro-averaged F1 on the four tasks (the number in parentheses is the closed − open gap \(\Delta\); Random is the random baseline):

Model Ungrounded Cls. closed / open (∆) Grounded Cls. (det.) closed / open (∆) Grounded Cls. (seg.) closed / open (∆) Object Detection closed / open (∆)
GPT-5.4-mini 0.533 / 0.104 (0.430) 0.410 / 0.132 (0.278) 0.472 / 0.126 (0.346) 0.100 / 0.053 (0.046)
GPT-4.1-mini 0.539 / 0.070 (0.468) 0.430 / 0.087 (0.344) 0.501 / 0.065 (0.436) 0.094 / 0.046 (0.048)
Gemma-3-4B 0.303 / 0.022 (0.281) 0.283 / 0.026 (0.257) 0.389 / 0.009 (0.381) 0.083 / 0.024 (0.058)
MedGemma-4B 0.439 / 0.043 (0.396) 0.371 / 0.066 (0.305) 0.310 / 0.042 (0.268) 0.076 / 0.018 (0.058)
MedGemma-1.5-4B 0.403 / 0.041 (0.362) 0.335 / 0.045 (0.290) 0.428 / 0.042 (0.385) 0.071 / 0.000 (0.071)
InternVL3.5-8B 0.517 / 0.111 (0.406) 0.439 / 0.122 (0.318) 0.514 / 0.088 (0.426) 0.061 / 0.019 (0.042)
Lingshu-7B 0.418 / 0.054 (0.364) 0.360 / 0.047 (0.313) 0.258 / 0.049 (0.209) 0.012 / 0.001 (0.011)
Lingshu-32B 0.469 / 0.073 (0.396) 0.368 / 0.150 (0.218) 0.310 / 0.053 (0.257) 0.067 / 0.004 (0.063)
Qwen2.5-VL-3B 0.381 / 0.029 (0.352) 0.336 / 0.027 (0.308) 0.240 / 0.026 (0.214) 0.014 / 0.016 (-0.002)
Qwen2.5-VL-7B 0.361 / 0.031 (0.330) 0.320 / 0.030 (0.289) 0.357 / 0.030 (0.326) 0.029 / 0.012 (0.017)
Qwen2.5-VL-32B 0.526 / 0.088 (0.437) 0.401 / 0.079 (0.322) 0.693 / 0.115 (0.577) 0.092 / 0.047 (0.045)
Qwen3-VL-4B 0.293 / 0.079 (0.214) 0.363 / 0.093 (0.270) 0.540 / 0.065 (0.475) 0.040 / 0.001 (0.039)
Qwen3-VL-8B 0.472 / 0.086 (0.386) 0.391 / 0.098 (0.292) 0.526 / 0.074 (0.452) 0.006 / 0.001 (0.005)
Qwen3-VL-32B 0.530 / 0.137 (0.393) 0.433 / 0.145 (0.288) 0.630 / 0.182 (0.447) 0.028 / 0.003 (0.024)
LLaVA-v1.5-7B 0.399 / 0.026 (0.374) 0.310 / 0.020 (0.289) 0.299 / 0.024 (0.275) 0.031 / 0.000 (0.031)
LLaVA-Med-7B 0.395 / 0.033 (0.362) 0.340 / 0.038 (0.302) 0.080 / 0.035 (0.045) 0.033 / 0.002 (0.030)
OctoMed-7B 0.494 / 0.070 (0.424) 0.346 / 0.059 (0.287) 0.573 / 0.056 (0.517) 0.033 / 0.009 (0.024)
Random 0.199 0.162 0.183 0.172

Three high-level trends stand out. Absolute scores are uniformly low: apart from Qwen2.5-VL-32B's 0.693 on segmentation-grounded classification, most results fall below the 0.5 threshold for adequate performance. The closed-to-open gap is consistently large, averaging about 0.26 across the three classification tasks (0.38 ungrounded, 0.29 detection-grounded, 0.36 segmentation-grounded), which indicates that models lean heavily on answer-choice cues. And object detection is a systemic collapse: no model surpasses the random baseline of 0.172 in the closed setting, and most models approach zero in the open setting. ⚠️ The main text names Qwen2.5-VL-32B as the best closed-format detector at about 0.10, while the table's highest value is GPT-5.4-mini's 0.100 with Qwen2.5-VL-32B at 0.092; the two disagree slightly, so refer to the original paper.

Ablation Study

MMBU trains no models, so what follows are two analyses: medically adapted models versus their base counterparts, and cross-benchmark transfer from legacy benchmarks to MMBU.

Base model Medically adapted model Closed ungrounded classification F1 (base → adapted) Change
Gemma-3-4B MedGemma-4B 0.303 → 0.439 +0.136
Gemma-3-4B MedGemma-1.5-4B 0.303 → 0.403 +0.100
LLaVA-1.5-7B LLaVA-Med-7B 0.399 → 0.395 −0.004
Qwen2.5-VL-7B Lingshu-7B 0.361 → 0.418 +0.057
Qwen2.5-VL-7B OctoMed-7B 0.361 → 0.494 +0.133
Qwen2.5-VL-32B Lingshu-32B 0.526 → 0.469 −0.057
Model Legacy benchmarks (PathVQA / VQA-RAD / SLAKE) Corresponding MMBU subsets Conclusion
OctoMed-7B improves improves Bidirectional generalization; one of the few adaptations that truly transfers
MedGemma-4B improves roughly flat Legacy-benchmark gains do not transfer to MMBU
Lingshu-7B / 32B improves degrades Closer to overfitting the legacy benchmark distribution

Stratifying by metadata, medically adapted models hold a modest overall edge: by submodality they beat non-medical models 62.28% of the time, but with an average F1 lead of only 0.08, and they do best on near-infrared imaging, phase-contrast microscopy, and specular microscopy. Paired per-question comparison (Fig. 6a) shows that models tie with their base counterpart on at least 70% of questions for every adaptation method, and only about half of the adapted models genuinely outperform their base; the largest gains come from MedGemma-4B (18.7% wins / 13.4% losses), OctoMed-7B (15.5% / 9.9%), and Lingshu-32B (13.8% / 12.4%). Regressing training-data size against win rate gives roughly +0.20% win rate per additional 1M training examples (\(y = 0.20x + 9.74\), \(R^2 = 0.41\))—what determines adaptation success looks more like how much data was fed in than how large the model is.

Key Findings

  • No model wins across the board. GPT-4.1-mini takes ungrounded classification (0.539), Qwen2.5-VL-32B takes segmentation-grounded classification (0.693), InternVL3.5-8B takes detection-grounded classification (0.439) and leads open-format tasks overall, while Qwen3-VL-32B leads open-ended answering (0.137 / 0.145 / 0.182). Together these numbers show there is no single "generalist" biomedical VLM.
  • The perception-versus-localization gap is more serious than the knowledge gap. Models can correctly classify what is inside a given box but cannot draw the box themselves—they can read the content of a region without being able to locate it. This is the central failure mode MMBU exposes and it points directly at spatial modeling as future work.
  • Medical specialization buys classification, not fine-grained grounded reasoning. MedGemma-4B beats the random baseline by +0.24 on closed ungrounded classification (F1 0.439), yet the general-purpose InternVL3.5-8B overtakes it on both grounded tasks (0.439 vs. 0.371 for detection-grounded, 0.514 vs. 0.310 for segmentation-grounded). Injecting medical knowledge and resolving local visual detail are not the same capability.
  • Gains on legacy benchmarks do not guarantee transfer. Every medically adapted model improves on PathVQA / VQA-RAD / SLAKE, but only OctoMed also improves on the corresponding MMBU subsets; MedGemma stays flat and Lingshu degrades. This strongly suggests those legacy training sets are already widely used for fine-tuning or pretraining, so the models optimize for those datasets while MMBU measures much broader biomedical perception.

Highlights & Insights

  • Separating "cannot see" from "does not know" is the paper's best design, and it takes two mechanisms working together: question wording is constrained to pixel-level visual content (no disease names, no clinical outcomes, no acronyms), and every sample carries 13 structured metadata attributes. The former guarantees a question can only be answered by looking; the latter guarantees a wrong answer can be traced to a specific modality and institution. Most benchmarks do one or the other, not both.
  • Deliberately excluding heavily contaminated datasets is a counterintuitive but correct trade-off. Giving up the most popular, easiest-to-game data is what lets the scores be read as out-of-distribution perception rather than memory retrieval. The price is ugly absolute numbers (best 0.693, detection below random), and the payoff is attribution ability.
  • The deterministic extraction protocol is directly reusable. For closed-ended VQA it drops LLM-as-a-judge entirely in favor of a hierarchical parser that pulls option labels from answer fields, formatting tags, and ordered lists, and scores unparseable responses as incorrect. This can be lifted as-is into benchmark-of-benchmark work or any large-scale automatic evaluation, and it removes the variance introduced by a judge model.
  • The naive \(\Delta\) metric carries a surprising amount of information. Closed and open differ only by whether a candidate pool exists, yet their difference is stable around 0.26—effectively a direct readout of how much a model leans on answer-choice cues, and sturdier than any anecdotal case study.

Limitations & Future Work

  • Although MMBU uses segmentation masks and bounding boxes for grounding, it never scores segmentation itself and reports no segmentation metric (e.g. Dice or mIoU). For a field where segmentation is daily work, that leaves an obvious hole.
  • Open-ended VQA relies on Qwen3-32B as a semantic-equivalence judge. The authors designed a fully deterministic extraction protocol for the closed setting precisely to avoid judge bias, yet the open setting still hands scoring to another model, and whether a judge from the same family as some evaluated models (Qwen) introduces preference is not discussed.
  • Micro-averaged F1 varies with candidate-pool size and class imbalance, so the four tasks have different random baselines (0.199 / 0.162 / 0.183 / 0.172) and comparing absolute F1 across tasks is risky; the main text also disagrees slightly with the table on one point (the best closed-format detector).
  • Metadata is not complete for every sample; the availability of the 13 attributes differs across datasets, so metadata-stratified subsets vary substantially in size and statistical power, and win-rate figures such as 62.28% should be read cautiously.
  • Straightforward improvements follow: bring segmentation into the scored task set; use multiple heterogeneous judges for open-ended evaluation and report their agreement; add a human-expert baseline on the same questions to establish a human ceiling for the scores.
  • vs GMAI-MMBench: Both are multi-domain medical benchmarks. GMAI-MMBench covers 32 submodalities, 26K samples, and 18 clinical tasks, organizing data through a lexical tree. MMBU pushes coverage to 35 submodalities and 78K samples, attaches up to 13 structured attributes per sample (including acquisition parameters, stain, and institution), and explicitly includes spatial tasks such as object detection. The former's tree supports coarse slicing; the latter supports batch-effect-level attribution such as "same modality, different institution."
  • vs OmniMedVQA: OmniMedVQA stitches 73 classification datasets into 128K question-answer pairs—large, but classification-only, without fine-grained human-annotated metadata or an open/closed comparison. MMBU is somewhat smaller in scale but adds grounded classification, object detection, three context templates, and the \(\Delta\) analysis, giving it a wider evaluation surface.
  • vs PathVQA / VQA-RAD / SLAKE: These are the legacy benchmarks MMBU explicitly compares against (1-3 submodalities, 2K-6K samples). The most valuable result in the paper comes from this comparison: models improve on all three yet fail to improve—or even degrade—on MMBU, indicating that those benchmarks largely test a distribution the models were trained on rather than generalizable perception.
  • vs MicroVQA / µ-Bench: These focus on microscopy and use relatively small-scale evaluation to show that biomedical VLMs are sensitive to prompt and batch effects. MMBU can be read as scaling that observation to 410 datasets and 11 modalities, and using metadata to turn the sensitivity from a phenomenon into a locatable failure.

Rating

  • Novelty: ⭐⭐⭐⭐ The first large-scale benchmark to operationalize biomedical perception through a unified task taxonomy, open/closed formats, and 13 metadata attributes; the contribution is to evaluation methodology more than to a single dataset.
  • Experimental Thoroughness: ⭐⭐⭐⭐ 17 models (with base/adapted pairings), four tasks, two answer formats, metadata stratification, and cross-benchmark transfer analysis, all with bootstrap confidence intervals; the missing human-expert comparison and segmentation metric are the gaps.
  • Writing Quality: ⭐⭐⭐⭐ The question-construction flow and filtering criteria are explained clearly and the figures are well organized; a few numbers disagree slightly between text and table, and much of the metadata-stratified detail sits in the supplementary materials.
  • Value: ⭐⭐⭐⭐⭐ Provides an attributable evaluation substrate for biomedical VLMs and nails down, with large-scale evidence, the phenomenon that high legacy-benchmark scores mask perception deficiencies—directly steering future model design toward spatial modeling and generalizable adaptation.