MedInsightBench: Evaluating Medical Analytics Agents on Multi-Modal Clinical Records¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: Medical LLM
Keywords: Medical agent benchmark, Digital pathology, Clinical insight discovery, Whole slide imaging, Multi-agent orchestration
TL;DR¶
This paper introduces MedInsightBench, the first benchmark evaluating medical analytics agents on goal-driven, multi-step insight discovery from paired whole-slide pathology images and clinical reports, and proposes MedInsightAgent, an agentic framework that synergizes visual root finding, specialized pathology tool inspection, and iterative question composition to substantially surpass baseline multi-modal models and generic agents.
Background & Motivation¶
Recent advancements in large multi-modal models (LMMs) have significantly expanded the frontier of AI in digital pathology and biomedical reasoning, progressing from basic visual question answering to detailed image captioning. However, in authentic oncological practice, clinicians rarely need single-turn descriptions of obvious histological elements; instead, they require actionable clinical insights derived from multi-gigapixel whole-slide images (WSIs) coupled with lengthy, unstructured pathology reports. Discovering such insights—such as correlating subtle invasive architectural patterns with lymphovascular invasion, estimating surgical margin clearance, evaluating nodal metastatic burden, or inferring biomarker and therapeutic responsiveness—demands deep, goal-directed clinical cognition that current models struggle to sustain.
Existing benchmarks fail to assess these cognitive demands because they primarily probe surface-level facts in closed-form QA, visual recognition, or text-to-SQL tasks on structured databases. Benchmarks designed for enterprise analytics agents (e.g., Spider 2.0, InfiAgent-DABench, InsightBench) operate predominantly over tabular relational data where insights can be retrieved through deterministic code execution. In sharp contrast, multi-modal pathological data presents vast morphological heterogeneity, unstandardized diagnostic narratives, and complex cross-modal dependencies. When evaluated on complex medical reasoning, current general-purpose LMMs exhibit severe cognitive deficiencies: they either retreat into generic, overly conservative statements to avoid errors, or succumb to catastrophic chain-of-thought fragility and visual hallucination, failing to bridge local microscopic cues with macro-level clinical prognosis.
To resolve this critical evaluation gap and empower automated clinical analytics, this paper establishes a systematic protocol for both benchmark construction and multi-agent clinical discovery. Sourcing authentic cancer cases from The Cancer Genome Atlas (TCGA), the authors develop an expert-verified pipeline that standardizes WSIs, extracts structured question-insight-evidence triplets across six clinically grounded categories, and synthesizes clear overarching diagnostic goals. Furthermore, recognizing that monolithic LMMs lack domain-specific visual parsing and multi-step reasoning stability, the authors propose a collaborative agent architecture. Core idea: formulate multi-modal clinical insight discovery as an iterative tripartite collaboration (MedInsightAgent) that systematically decomposes diagnostic goals into visual root queries, scrutinizes microscopic patterns via a specialized pathology tool, and expands findings through goal-directed follow-up questioning.
Method¶
Overall Architecture¶
The overall architecture of MedInsightAgent decomposes the medical insight discovery workflow into three synergistic, specialized agents: the Visual Root Finder (VRF), the Analytical Insight Agent (AIA), and the Follow-Up Question Composer (FQC). Given a downsampled whole-slide pathology image \(I\) and an explicit overarching clinical goal \(G\) (e.g., assessing prognostic risk and staging parameters in lingual squamous cell carcinoma), VRF first extracts salient visual features and retrieves authoritative literature to generate an initial set of root questions \(Q\). Next, AIA leverages an external domain-specialized vision-language model as an Image-Analysis Tool to inspect targeted morphological regions, yielding grounded factual evidence and intermediate clinical insights. Finally, FQC scores and selects the most promising follow-up question to iteratively feed back into AIA, driving progressive, multi-round hypothesis expansion across exploration depth \(p\) before outputting the final accumulated insights \(S\).
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
IN["Inputs: Pathology Image I + Overarching Goal G"] --> A["Visual Root Anchoring & Knowledge Retrieval<br/>ISM feature summary + WRM literature retrieval yields root questions Q"]
A --> B["Pathology Tool Scrutiny & Insight Generation<br/>PathGen-LLaVA ROI inspection + AIA evidence-grounded inference"]
B --> C["Goal-Directed Question Composition & Iterative Refinement<br/>FQC candidate sampling + best question scoring and feedback loop"]
C -->|Multi-round reasoning cycle| B
C -->|Iteration budget reached| OUT["Final Outputs: Grounded Insight Set S & Traceable Evidence"]
Key Designs¶
1. Visual Root Anchoring & Knowledge Retrieval: Semantic gap between abstract diagnostic goals and raw gigapixel morphology -> Coupling low-magnification image summarization with web-scale medical knowledge retrieval -> Establishing a concrete, clinically grounded root question set
Directly prompting an LMM with an overarching clinical goal on high-dimensional pathology images frequently triggers wandering reasoning trajectories and superficial queries. VRF addresses this by first routing the slide \(I\) through an Image-Summarization Module \(\text{ISM}_{img}\), which surveys tissue architecture, stromal reaction, nuclear atypia, and necrotic foci to extract prominent visual observations \(F\) alongside key pathology descriptors \(K\). A Web-Retrieval Module \(\text{WRM}\) then queries medical domain resources using \(K\), returning top-10 authoritative literature and clinical guideline snippets \(D = \{d_1, \dots, d_{10}\}\). Finally, the Root Question Generator synthesizes \(I\), \(G\), \(F\), and \(D\) into \(m\) distinct, non-overlapping root questions \(Q = \{q_i\}_{i=1}^m = \mathcal{L}(I, G, F, D)\), anchoring abstract diagnostic intent in concrete histological targets.
2. Pathology Tool Scrutiny & Insight Generation: General-purpose visual encoders lack fine-grained micro-anatomical sensitivity -> Employing PathGen-LLaVA as a dedicated image-analysis tool for explicit regional evidence extraction -> Grounding multi-step clinical deduction in verifiable visual findings
Off-the-shelf multi-modal models often fail to discern critical subtle hallmarks such as perineural sheath invasion, angiolymphatic emboli, or micro-calcifications, leading to ungrounded claims. AIA mitigates this vulnerability by isolating targeted visual inspection from high-level reasoning. For each inquiry \(q_i\), AIA queries PathGen-LLaVA—a pathology-specialized model pre-trained on 1.6 million histopathology image-text pairs—serving as an Image-Analysis Tool \(\text{IAT}\). \(\text{IAT}\) outputs structured visual findings and bounding-box regions of interest \(E_i = \text{IAT}(I, q_i)\). The Answers and Insights Generator then consumes \(q_i\), \(I\), and \(E_i\) to generate a direct clinical answer \(a_i\) alongside an actionable medical insight \(s_i\), ensuring every generated insight remains traceable to visible microscopic evidence.
3. Goal-Directed Question Composition & Iterative Refinement: Single-turn inference suffers from premature reasoning convergence and stochastic inquiry coverage -> Multi-candidate follow-up sampling with multi-criteria quality scoring -> Progressively steering cognitive exploration toward deeper prognostic and molecular deductions
Static root questions typically capture only initial diagnostic layers, missing complex downstream relationships between histological grades, surgical margin status, and therapeutic resistance. FQC maintains an episodic memory of historical questions, answers, and visual cues. A candidate generator \(\mathcal{F}\) proposes \(n\) potential follow-up questions \(C = \{c_i\}_{i=1}^n = \mathcal{F}(I, G, q_i, a_i, F, D)\) strictly constrained by relevance to goal \(G\) and distinctness from prior queries. The Question Selector \(\mathcal{S}\) ranks candidate inquiries based on semantic diversity, diagnostic utility, and grounding feasibility, forwarding the optimal query \(c_{best} = \mathcal{S}(C)\) back to AIA. Controlled by iteration budget \(p\), this recursive loop expands the insight volume to \(r \times (p + 1)\), transforming passive descriptive reporting into active, exploratory hypothesis testing.
Benchmark & Evaluation Protocol¶
The MedInsightBench dataset comprises 332 curated cancer pathology cases spanning diverse malignancies from TCGA, annotated with a total of 3,933 expert-verified clinical insights. Its creation follows a stringent three-stage pipeline: 1. Whole-Slide Image Preprocessing: High-resolution SVS whole-slide images are downsampled using optimal pyramid ratios, standardized via color normalization, and filtered through dual LMM automated screening and expert pathologist verification to remove blurred or artefactual tissue. 2. Report Processing & Triplet Extraction: Corresponding diagnostic PDF reports undergo OCR, followed by mandatory clinical checklist auditing (verifying unambiguous diagnosis, tumor size, histological grade, margin status, nodal counts, laterality, and IHC markers). Standardized insights are extracted across six operational categories: Descriptive (1,580), Diagnostic (1,122), Evaluative (518), Exploratory (263), Predictive (256), and Prescriptive (194). Samples span four difficulty tiers, with Level 2 (48.2%) and Level 3 (47.0%) forming the cognitively demanding core. 3. Overarching Goal Synthesis & Metric Formulation: An overarching goal is synthesized from the analytical question trajectories to guide end-to-end discovery. Evaluation utilizes an orthogonal four-dimensional metric suite: Insight Recall (coverage of evidence-backed reference insights via \(\mathbb{E}_{gt \sim Unif(GT)} \max_{i \in I} S(gt, i)\)), Insight Precision (factual fidelity and hallucination prevention via \(\mathbb{E}_{i \sim Unif(I)} \max_{gt \in GT} S(i, gt)\)), Insight F1 (harmonic balance between coverage and precision), and Insight Novelty (evaluating clinically plausible discoveries beyond ground truth using a 3-LMM panel with CoT voting).
Key Experimental Results¶
Main Results¶
The authors evaluate leading commercial multi-modal foundation models (GPT-4o, GPT-5), open-source vision-language baselines (DeepSeek-VL2, InternVL3-38B, Qwen2.5-VL-32B, Qwen3-VL-32B), the domain-specific PathGen-LLaVA, and representative agent frameworks (CoT, ReAct, Plan-and-Solve, Self-Refine, Reflexion) implemented on the unified Qwen3-VL-32B-Instruct backbone against MedInsightAgent.
| Method Paradigm | Model / Framework | Recall (ROUGE-1) | Recall (G-Eval) | Precision (ROUGE-1) | Precision (G-Eval) | F1 (ROUGE-1) | F1 (G-Eval) | Novelty (Original) | Novelty (Innovation) |
|---|---|---|---|---|---|---|---|---|---|
| Direct Decoding (LMM-Only) | GPT-4o | 0.1799 | 0.2978 | 0.2091 | 0.3577 | 0.1934 | 0.3250 | 0.1293 | 0.2094 |
| Direct Decoding (LMM-Only) | GPT-5 | 0.1866 | 0.3052 | 0.1851 | 0.3649 | 0.1858 | 0.3324 | 0.1322 | 0.2131 |
| Direct Decoding (LMM-Only) | Deepseek-VL2 | 0.1816 | 0.3226 | 0.2274 | 0.4070 | 0.2019 | 0.3599 | 0.1959 | 0.2712 |
| Direct Decoding (LMM-Only) | InternVL3-38B | 0.1774 | 0.3390 | 0.2002 | 0.3993 | 0.1881 | 0.3667 | 0.1607 | 0.2553 |
| Direct Decoding (LMM-Only) | Qwen2.5-VL-32B | 0.1917 | 0.3975 | 0.2139 | 0.4848 | 0.2022 | 0.4368 | 0.3493 | 0.4171 |
| Direct Decoding (LMM-Only) | Qwen3-VL-32B | 0.2064 | 0.4295 | 0.2267 | 0.5372 | 0.2170 | 0.4774 | 0.3531 | 0.4488 |
| Direct Decoding (LMM-Only) | PathGen-LLaVA | 0.1849 | 0.3420 | 0.1989 | 0.3845 | 0.1916 | 0.3620 | 0.1893 | 0.2936 |
| Agentic Methods (Qwen3-VL) | Chain-of-Thought (CoT) | 0.2053 | 0.4379 | 0.2281 | 0.5374 | 0.2161 | 0.4826 | 0.3549 | 0.4583 |
| Agentic Methods (Qwen3-VL) | ReAct (Generic Tools) | 0.2092 | 0.4377 | 0.2344 | 0.5390 | 0.2211 | 0.4831 | 0.3492 | 0.4456 |
| Agentic Methods (Qwen3-VL) | ReAct + Same Tools | 0.2105 | 0.4562 | 0.2397 | 0.5698 | 0.2242 | 0.5067 | 0.3853 | 0.4679 |
| Agentic Methods (Qwen3-VL) | Plan-and-Solve | 0.2117 | 0.4534 | 0.2378 | 0.5633 | 0.2240 | 0.5024 | 0.3749 | 0.4613 |
| Agentic Methods (Qwen3-VL) | Self-Refine | 0.2080 | 0.4467 | 0.2342 | 0.5554 | 0.2203 | 0.4952 | 0.3584 | 0.4493 |
| Agentic Methods (Qwen3-VL) | Reflexion | 0.2125 | 0.4573 | 0.2395 | 0.5670 | 0.2252 | 0.5063 | 0.3826 | 0.4548 |
| Agentic Methods (Qwen3-VL) | MedInsightAgent (Ours) | 0.2233 | 0.4703 | 0.2483 | 0.5921 | 0.2351 | 0.5242 | 0.4274 | 0.4995 |
Ablation Study & Cost-Performance Analysis¶
Component Ablation within MedInsightAgent (Qwen3-VL-32B Backbone, G-Eval Metrics)
| Architecture Configuration | Insights Recall | Insights Precision | Insights F1 | Insights Novelty | Observations & Impact |
|---|---|---|---|---|---|
| Direct Decoding Baseline | 0.4295 | 0.5372 | 0.4774 | 0.4488 | Standard zero-shot multimodal generation |
| MedInsightAgent (Full Model) | 0.4703 | 0.5921 | 0.5242 | 0.4995 | Full tripartite collaborative multi-agent pipeline |
| w/o Image-Summarization Module (ISM) | 0.4615 (-0.0088) | 0.5764 (-0.0157) | 0.5126 (-0.0116) | 0.4780 (-0.0215) | Loss of global visual overview and guided keywords |
| w/o Web-Retrieval Module (WRM) | 0.4623 (-0.0080) | 0.5739 (-0.0182) | 0.5121 (-0.0121) | 0.4705 (-0.0290) | Sharpest drop in Novelty (-0.0290), impairs external knowledge integration |
| w/o Image-Analysis Tool (IAT) | 0.4537 (-0.0166) | 0.5604 (-0.0317) | 0.5014 (-0.0228) | 0.4842 (-0.0153) | Sharp drop in Precision (-0.0317) due to lack of pathology micro-evidence |
| w/o Follow-Up Question Composer (FQC) | 0.4391 (-0.0312) | 0.5511 (-0.0410) | 0.4888 (-0.0354) | 0.4653 (-0.0342) | Greatest overall decline in F1 (-0.0354), collapsing multi-step depth |
Inference Computational Overhead & Latency Profile (Per-Sample Averages)
| Method | API Calls / Sample | Avg. Input Tokens | Avg. Output Tokens | Wall-Clock Latency (s) | G-Eval F1 |
|---|---|---|---|---|---|
| Direct Decoding | 1.0 | 2,352 | 521 | 15.8 | 0.4774 |
| Chain-of-Thought (CoT) | 1.0 | 2,685 | 769 | 27.6 | 0.4826 |
| ReAct | 12.4 | 10,907 | 2,854 | 133.4 | 0.5067 |
| MedInsightAgent | 32.0 | 65,372 | 7,692 | 958.9 | 0.5242 |
Key Findings¶
- Orchestration Trumps Naive Tool Access: Providing ReAct with the identical pathology toolset improves its G-Eval F1 from 0.4831 to 0.5067, yet it remains significantly inferior to MedInsightAgent (0.5242, \(p = 0.010\) across 10,000 bootstrap resamples). This underscores that raw tool availability is insufficient; structured role specialization and directed information routing are vital to solving high-stakes pathology workflows.
- Pervasive Model Conservatism: Across every baseline and metric (ROUGE-1 and G-Eval), Precision substantially outstrips Recall (e.g., Qwen3-VL achieves 0.5372 Precision vs. 0.4295 Recall). Foundation models consistently favor safe, trivial observations to minimize penalties, severely curtailing exploratory depth in open-ended diagnostic tasks.
- Divergent Roles of Visual Tools and Retrieval: As highlighted by ablation data, the specialized pathology tool (IAT) drives factual grounding, accounting for the largest share of Precision loss when omitted (-0.0317). In contrast, the Web-Retrieval Module (WRM) primarily drives exploratory innovation, causing the largest decline in Novelty (-0.0290). Joint ablation confirms that even with zero external tools, MedInsightAgent's structured pipeline achieves 0.4889 F1, surpassing CoT (0.4826).
- Clinical Alignment Verified by Expert Double-Blind Audit: Blind auditing of Novelty-passing claims by expert pathologists reveals that 43.3% of MedInsightAgent's discoveries represent genuine clinical novelties (Category A, vs. 26.7% for Qwen3-VL and 30.0% for Reflexion). Furthermore, 86.6% of its generated statements are clinically valid and non-trivial (Categories A+B), verifying that automated novelty scoring reflects true clinical utility rather than superficial paraphrasing.
Highlights & Insights¶
- Decoupled Architecture for Multimodal Complexity: Segregating global morphological parsing (VRF), targeted microstructural analysis (AIA), and iterative cognitive expansion (FQC) prevents contextual saturation and eliminates hallucinations inherent to monolithic end-to-end decoders.
- Closed-Loop Hypothesis Evolution via Dynamic Query Selection: Implementing an explicit selection mechanism over candidate follow-up inquiries ensures that the agent advances along high-yield diagnostic trajectories aligned with the overarching clinical objective.
- Multidimensional Benchmarking Paradigm: Shifting evaluation from simple lexical overlap to a four-part metric matrix (Recall, Precision, F1, and Novelty) paired with G-Eval semantic modeling establishes a robust standard for benchmarking exploratory medical agents.
Limitations & Future Work¶
- Cohort and Modality Constraints: Sourced exclusively from TCGA surgical resections, the benchmark lacks non-oncological conditions, radiological cross-referencing (CT/MRI), and diverse demographic cohorts needed to reflect broader epidemiological variation.
- Information Loss from Whole-Slide Downsampling: Condensing multi-gigapixel WSIs into fixed-size pyramid levels risks obscuring microscopic hallmarks such as mitotic figures or isolated tumor cells. Future iterations will incorporate dynamic patch zooming and attention-guided multi-resolution navigation.
- Inference Latency and Resource Demands: Requiring 32 API calls and 958.9 seconds per case, MedInsightAgent is tailored for comprehensive diagnostic review and academic research rather than real-time intraoperative frozen section analysis.
Related Work & Insights¶
- vs. Structured Data Insight Agents (InsightBench, InfiAgent-DABench): While existing agents generate multi-step insights by querying clean tabular databases via code or SQL, MedInsightBench pioneers open-ended reasoning over high-dimensional, unstructured whole-slide imaging and paired narrative reports.
- vs. Generic Multimodal Agent Baselines (ReAct, Reflexion): Generalist agents rely on unguided self-critique and generic web search, leading to frequent off-target exploration in pathology. MedInsightAgent demonstrates that embedding dedicated domain-specialized vision tools (PathGen-LLaVA) within structured tripartite workflows is imperative for clinical viability.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Pioneers goal-driven multi-step medical insight discovery on multimodal WSIs and clinical text with a clinically grounded benchmark]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive benchmarking across open/closed LMMs, generic agent baselines, module ablations, bootstrap statistics, and double-blind human audits]
- Writing Quality: ⭐⭐⭐⭐⭐ [Clear problem formulation, elegant mathematical notation, comprehensive qualitative case studies, and fully documented appendix prompts]
- Value: ⭐⭐⭐⭐⭐ [Sets an authoritative standard for evaluating medical analytics agents and provides a battle-tested blueprint for multi-modal agentic workflows]