Benchmarking Vision-Language Models for Microscopic Plant Image Understanding

Conference: ECCV2026
Paper: Official page · PDF
Code: https://github.com/tqwei05/PlantMicro
Area: Multimodal VLM / Microscopic Plant Image Understanding
Keywords: plant microscopy, visual question answering, fine-grained recognition, spatial grounding, retrieval augmentation

TL;DR

PlantMicro organizes 5,410 plant microscopy images into 9,718 question-answer pairs across 10 tasks, showing that general-purpose VLMs can approach perfect modality recognition while struggling with host and pathogen identity; task-restricted visual retrieval substantially improves recognition, whereas reasoning prompts and a single demonstration do not reliably help.

Background & Motivation

Agricultural visual question answering usually concerns leaves, field crops, or disease symptoms, with benchmarks such as CDDM and AgroBench emphasizing macroscopic appearance. Microscopy shifts the problem to cells, tissues, and subcellular structures: the model cannot inspect the whole plant and must infer biological identity from cell walls, spore morphology, or fluorescence signals. Recognizing a leaf photograph therefore does not establish an understanding of plant tissue under a microscope.

Existing plant microscopy datasets often serve a single task, while their file formats, annotation fields, and imaging modalities differ. Meanwhile, related efforts such as BiomedCLIP and PLIP primarily address biomedical or pathology settings, which do not directly establish competence in plant microscopy. The missing resource is not simply a larger image collection. It is a consistent evaluation that converts expert annotations across sources into comparable questions and distinguishes recognizing an imaging style from identifying a biological entity.

Core idea: construct multilevel VQA from expert-verified microscopy metadata, use both multiple-choice and direct prediction to separate candidate-assisted performance from actual counting and localization, and diagnose limitations through prompting and retrieval experiments.

Method

Overall Architecture

The main contribution is a benchmark and diagnostic protocol, not a newly trained vision network. Inputs are plant microscopy images and metadata from public datasets, publications, and institutional databases. After source standardization and expert review, valid annotation fields are converted into questions; the outputs are model scores for recognition, counting, localization, and inference-time interventions.

The collection covers 25 host species, four biological domains (mycology, nematology, botany, and cell biology), and three microscopy modalities (light, fluorescence, and electron). The 5,410 images do not each receive all 10 question types. An image contributes to a task only when reliable annotations for that task exist, so the 9,718 VQA pairs reflect heterogeneous available metadata rather than automatically completed question sets.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Sources["Microscopy images and source metadata"] --> Curation["Source Standardization<br/>and Dual Review"]
    Curation --> Tasks["Metadata-Driven<br/>Task Construction"]
    Tasks --> Protocol["Choice and Direct<br/>Prediction Protocols"]
    Protocol --> Diagnosis["Prompting and<br/>Retrieval Diagnostics"]
    Diagnosis --> Results["Task scores and error analysis"]

Key Designs

1. Source Standardization and Dual Review: establish trustworthy answers before generating questions

The authors collect datasets from peer-reviewed publications and open repositories, prioritizing resources available for research. For resources that are not directly accessible, they contact the authors for research permission. Each image receives a unique identifier while retaining its provenance, avoiding filename collisions across sources. TIFF, JPEG, and PNG inputs are converted to JPEG at their original resolution. A JSON record stores standardized attributes such as domain, modality, staining method, and host.

These attributes are not guessed by a VLM. They are verified against original publications, dataset providers, or domain experts, then independently reviewed by two biologists, with disagreements excluded. Generated questions undergo a further two-expert review of attribute-question alignment, support from visual evidence, and the biological plausibility of distractors. The first stage checks source labels; the second checks whether those labels yield valid evaluation questions. This distinction prevents an available metadata field from being treated as automatically answerable from an image, although expert agreement alone cannot eliminate source-related shortcuts.

2. Metadata-Driven Task Construction: separate broad appearance from biological identity

The eight recognition tasks form two levels. Image-level tasks cover modality, biological domain, embedding resin, and stain or fluorescent marker. Biological-level tasks cover host, pathogen, organelle, and organ. The former often admit global color, texture, or imaging-style cues, whereas the latter require finer biological interpretation of local structures. Resin categories include preparation materials such as Epon, Spurr, and Quetol-651; staining categories include DAPI, GFP, and RFP. Although the paper calls the organelle task "Organelle Detection," it is four-option identification here, not organelle bounding-box prediction.

Each recognition question has four candidates, with distractors sampled from other valid values of the same metadata field. Host distractors, for example, remain host species instead of introducing an organ name that can be trivially rejected. Counting distractors are nearby integers around the true count. Localization choices combine the true box with similarly sized, spatially shifted alternatives, designed to be non-overlapping. This reduces ambiguity from free-form answer wording, but it also supplies elimination cues. Multiple-choice success alone consequently does not demonstrate independent task execution.

The remaining tasks are target counting and localization. Counting addresses densely arranged objects such as spores or pollen, requiring the exclusion of debris and separation of touching instances; counting images contain 8.2 objects on average, with a median of 6. Localization asks for the specified target's absolute pixel coordinates. These tasks test numerical parsing and spatial grounding, addressing the possibility that a model can name an object without reliably indicating where it is.

3. Choice and Direct Prediction Protocols: candidate assistance is not equivalent to generating an answer

The eight recognition tasks use four-option questions. Counting and localization use both multiple-choice and direct prediction. Direct counting requires an integer, while direct localization requires absolute pixel coordinates \((x_1,y_1,x_2,y_2)\) in a fixed format; invalid-format outputs are failures. Models share a standardized input-processing and task-prompting pipeline. Experiments use a workstation with two NVIDIA A6000 GPUs, which does not imply that proprietary API models run locally on those GPUs.

The paper defines recognition metrics as follows, where \(C\) is the number of classes, \(TP_i\) is the number of correct predictions in class \(i\), and \(N_i\) is the number of samples in that class:

\[ \mathrm{Micro}=\frac{\sum_{i=1}^{C}TP_i}{\sum_{i=1}^{C}N_i},\qquad \mathrm{Macro}=\frac{1}{C}\sum_{i=1}^{C}\frac{TP_i}{N_i}. \]

Micro weights frequent classes more heavily, while Macro gives each class equal weight. There is nevertheless an aggregation ambiguity: the Overall Macro values in Table 2 match the arithmetic mean of the eight task columns, whereas the textual definition describes class averaging. This note quotes the reported table values without inventing an unspecified cross-task aggregation procedure. Class balancing and task balancing are different, and reproductions should verify how the two levels interact.

Direct counting reports MAE and RMSE: the former averages absolute count errors, and the latter penalizes extreme mistakes more strongly. Direct localization reports IoU and mAP at an IoU threshold of 0.5. Selecting a candidate box and generating coordinates are different tasks, so localization multiple-choice accuracy and mAP are not directly comparable measurements of the same event. Keeping these protocols separate helps explain why a model may choose correctly but generate an imprecise box.

4. Prompting and Retrieval Diagnostics: distinguish more reasoning, an example, and a relevant example

The authors do not introduce a new loss function. They instead apply three inference-time interventions. First, an observation-reasoning-selection CoT prompt tests whether explicitly organizing visual evidence improves recognition. Second, a 1-shot demonstration for counting and localization tests whether a single example calibrates outputs. Third, recognition-task RAG retrieves the most visually similar example from the benchmark pool using visual embedding similarity, restricts retrieval to the same task category, and prepends the retrieved example as an in-context demonstration.

This RAG setting is visual example retrieval, not retrieval of explanatory passages from a plant-science literature corpus. Task restriction preserves the type of label information in the demonstration, while visual similarity seeks to reduce morphological mismatch between demonstration and query. It may therefore be more useful than simply supplying an example. However, the 1-shot and RAG experiments address different tasks and are not a strictly controlled comparison. Gains may combine label cues, example matching, and additional domain information.

For failure diagnosis, the authors randomly select 25 erroneous samples per benchmark task and inspect Gemini 2.5-Pro's generated CoT, grouping failures into perception errors, knowledge deficiencies, and irrelevant responses. This provides diagnostic evidence, but a generated explanation is not a direct observation of internal computation. The cached main text also does not specify the retrieval embedding model, query-self exclusion, near-duplicate filtering, or cross-source separation. RAG results should therefore be read as evidence under the reported setup, not as a demonstrated leakage-free generalization improvement.

Key Experimental Results

Main Results

The following values come from Tables 2 and 3. Accuracy, mAP, and IoU retain the paper's percentage scale; MAE and RMSE are measured in object counts. The name QwenVLM-7B is retained as reported, without inferring a more specific model version.

Model	Modality	Host	Pathogen	Overall Macro	Overall Micro
Random Choice	24.97	24.95	24.89	24.93	25.23
GPT-5 mini	99.73	31.40	43.91	57.26	61.17
GPT-5	97.72	34.93	42.71	56.79	60.56
Gemini 2.5-Flash	97.86	45.37	32.67	63.08	64.12
Gemini 2.5-Pro	99.56	46.82	33.19	62.93	65.07
LLaVA-13B	55.50	32.43	28.32	37.24	41.88
QwenVLM-7B	68.05	30.75	25.21	51.98	48.58

The abstract describes GPT-5's 34.93% and the random baseline of 24.95% as pathogen classification results, but Table 2 places both in the Host column. The corresponding Path. values are 42.71% and 24.89%. This note follows the table headers rather than repeating the abstract's task mismatch. Gemini 2.5-Flash has the highest Macro, while Gemini 2.5-Pro has the highest Micro; neither should be described as winning every metric.

Model	Counting MC Accuracy	Counting MAE	Counting RMSE	Localization MC Accuracy	Localization [email protected]	Localization IoU
GPT-5-mini	66.29	2.84	3.70	79.33	23.64	30.91
GPT-5	71.34	2.07	3.02	83.21	56.66	50.53
Gemini 2.5-Pro	41.37	3.30	6.32	65.91	27.60	37.75
LLaVA-13B	28.10	8.71	22.18	32.41	30.78	40.33
QwenVLM-7B	46.45	3.38	9.24	89.44	65.05	68.64

GPT-5 leads counting in Table 3, whereas QwenVLM-7B exceeds GPT-5 in localization by 8.39 percentage points in mAP and 18.11 points in IoU. The result is not uniform proprietary-model dominance: recognition, numerical estimation, and coordinate generation induce different rankings.

Ablation Study

There is no network-component removal experiment. The table below summarizes inference-time interventions from Tables 4 and 6, constituting prompting and retrieval analyses. Scores are percentages, and changes are percentage points.

Model	Setting	Overall Macro	Overall Micro	Macro Change from Base
Gemini 2.5-Pro	Base	62.93	65.07	0.00
Gemini 2.5-Pro	Three-step CoT	61.28	61.44	-1.65
QwenVLM-7B	Base	51.98	48.58	0.00
QwenVLM-7B	Three-step CoT	57.99	57.72	+6.01
QwenVLM-7B	Visual RAG	68.63	69.04	+16.65
LLaVA-13B	Base	37.24	41.88	0.00
LLaVA-13B	Visual RAG	57.75	60.21	+20.51
LLaVA-Next-13B	Base	41.20	39.63	0.00
LLaVA-Next-13B	Visual RAG	58.18	60.05	+16.98

Table 5 further shows inconsistent 1-shot effects. GPT-5's counting MAE worsens from 2.07 to 2.28, and localization mAP falls from 56.66 to 53.73. QwenVLM-7B's counting MAE improves from 3.38 to 3.11, but its localization mAP falls from 65.05 to 62.79. The authors attribute degradation to demonstration-query mismatch; this is an interpretation, not a causal conclusion established by a separate controlled ablation.

Key Findings

• Coarse recognition does not imply fine-grained understanding: GPT-5 reaches 97.72% on modality identification but only 34.93% on host identification, a substantial within-model gap.
• CoT is not a universal improvement: QwenVLM-7B gains 6.01 Macro points, while Gemini 2.5-Pro loses 1.65 points. QwenVLM-7B's pathogen score moves only from 25.21 to 25.26, showing that aggregate gains can conceal almost no progress on the hardest task.
• Figure 6 attributes 81.0% of analyzed errors to knowledge deficiency, 13.5% to perception errors, and 5.5% to irrelevant responses. These are proportions within a particular model's sampled failures, not failure probabilities for every VLM or input.
• RAG brings QwenVLM-7B to 68.63% Macro, but it uses additional retrieved demonstrations. Comparisons against proprietary models without retrieval must acknowledge the different information budgets.

Highlights & Insights

• Separate imaging appearance from biological identity. Modality recognition often depends on global texture, while host and pathogen recognition require finer knowledge. Task-level scores reveal limitations hidden by a single aggregate.
• Combine candidate selection with direct output. Two protocols for counting and localization help expose candidate assistance. Choosing the correct box should not be mistaken for precise spatial generation.
• Study retrieval relevance independently. The results suggest that example matching may matter more than simply supplying an example. A useful next experiment would fix the task, model, and example count while comparing random, visually similar, and expert-matched retrieval.

Limitations & Future Work

• The retrieval protocol needs clarification. The main text does not establish retrieval/evaluation separation, query-self exclusion, or near-duplicate removal. This does not prove leakage occurred, but it limits conclusions about RAG generalization.
• Attribute identifiability and source shortcuts remain entangled. Resin and host labels may correlate with instruments, laboratories, or source datasets. Source- or acquisition-batch-held-out evaluation and independent blinded expert testing would help distinguish these effects.
• Statistical and reproduction details are limited. The local cache ends after the references and does not contain the per-dataset appendix mentioned in the main text. Per-source license details, precise sampling proportions, and decoding parameters cannot be filled in from this source. Cross-task Macro aggregation also needs clarification.
• Error explanations do not prove mechanisms. Labeling generated CoT as showing a knowledge deficiency is informative, but image-free, image-shuffled, and label-prior controls are needed to establish what information the model actually uses.
• This is not yet deployment-reliability evaluation. Four-option questions assess discrimination within supplied candidates. Unknown species, confidence calibration, cross-device transfer, and appropriate abstention in real scientific workflows remain to be tested.

Related Work & Insights

• vs CDDM, AgroBench, and Agri-CM3: These primarily cover macroscopic agricultural settings. PlantMicro moves evaluation inside tissues and cells; its contribution is task coverage and protocol design, not a replacement for broader agricultural knowledge evaluation.
• vs microscopy sources such as PODB and Tomato Spore: Existing resources supply specialized images and metadata. PlantMicro converts heterogeneous sources into unified VQA, adding curation, task mapping, and model comparisons rather than claiming every image was newly acquired.
• vs BiomedCLIP, PLIP, and MicroBench: Biomedical pretraining and microscopy evaluation provide relevant context, but plant cellular organization and pathogen knowledge are distinctive. Medical-domain scores cannot be directly transferred to this benchmark.
• Research direction: Establish task-restricted retrieval with source isolation and near-duplicate controls before comparing visual-feature improvements with plant-domain knowledge injection. This would distinguish gains from better perception and gains from more appropriate examples.

Rating

• Novelty: 4/5. The plant-microscopy VLM benchmark addresses a clear gap, with innovation in data integration and capability decomposition rather than architecture.
• Experimental Thoroughness: 4/5. It covers 13 models, 10 tasks, and multiple prompting interventions, but retrieval isolation, statistical uncertainty, and cross-source generalization need stronger evidence.
• Writing Quality: 3/5. Tasks and main tables are readable, but the abstract's Host/Pathogen mismatch and unclear Macro aggregation require correction.
• Value: 4/5. Useful for fine-grained domain understanding and retrieval augmentation, with stricter evaluation needed before deployment claims.