Skip to content

Adaptive Knowledge Routing for Complex Reasoning in LLMs

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Multimodal Medical Insights, Agentic Collaborative Reasoning, Visual Root Finding, Adaptive Follow-Up Questioning, Knowledge Routing

TL;DR

To tackle the widespread failure of Large Multimodal Models in complex pathological reasoning—where models stagnate at surface-level descriptions and ungrounded hallucinations—this paper establishes MedInsightBench (332 cases, 3,933 insights) and proposes MedInsightAgent, a multi-agent framework combining visual root finding, grounded insight analysis, and adaptive follow-up questioning, boosting Qwen3-VL-32B to 0.5242 G-Eval F1 and 0.4995 Novelty.

Background & Motivation

In modern computational pathology and precision clinical diagnostics, transforming whole-slide images (WSIs) alongside unstructured clinical reports into actionable medical insights—such as evaluating tumor margins, determining accurate pathologic staging, and detecting nodal metastasis—serves as the foundational cornerstone for personalized oncology. While recent Large Multimodal Models (LMMs) have demonstrated impressive performance on standard visual question answering and coarse visual recognition tasks, their transition into real-world pathological workflows reveals substantial cognitive deficiencies. Contemporary medical benchmarks primarily assess surface-level facts, single-turn entity extraction, or binary report classification. They fail to evaluate whether models can navigate occult tissue morphology (e.g., subtle lymphovascular invasion, perineural infiltration, or desmoplastic stroma), map visual hallmarks to underlying molecular alterations, and synthesize complementary textual records to establish comprehensive prognoses.

The underlying tension lies in the severe discrepancy between open-ended clinical exploration and the ungrounded fragility of general-purpose LMMs. When deployed in zero-shot or direct-decoding paradigms, existing models frequently generate shallow, generic descriptions without diagnostic depth, often succumbing to unconstrained hallucinations when pressured to deliver advanced clinical inferences. General-purpose agentic prompting strategies like ReAct, Chain-of-Thought, and Reflexion attempt to introduce intermediate reasoning steps, yet they lack pathology-specific visual tools and structured evidence-grounding constraints. As a consequence, these models either drift away from the overarching diagnostic objective or speculate wildly without grounding in microscopic slide features.

To bridge this fundamental gap, this work approaches medical insight discovery through structured multi-agent collaboration and goal-directed adaptive knowledge routing. Core idea: Establish MedInsightBench, a goal-oriented multi-step medical insight benchmark grounded in expert-verified pathology data, and introduce MedInsightAgent, a multi-agent framework that orchestrates visual root finding, tool-grounded analytical inference, and adaptive follow-up question composition to convert complex multimodal pathology exploration into a robust, evidence-backed closed loop.

Method

Overall Architecture

MedInsightAgent is engineered to transform high-resolution pathological images and high-level clinical objectives into faithful, verifiable, and novel clinical insights. The architecture decouples the complex reasoning trajectory into three specialized collaborative agents supported by external analytical tools in an iterative refinement loop: First, the Visual Root Finder ingests the pathology slide alongside the analytical goal, deploying a slide-level summarizer and web retrieval module to formulate an initial set of high-priority root questions. Second, the Analytical Insight Agent answers each question by invoking a specialized pathology foundation model (PathGen-LLaVA) as an image analysis tool, extracting explicit visual cues and grounded evidence to synthesize intermediate insights. Third, the Follow-Up Question Composer evaluates the intermediate discoveries against the overarching goal, generating and selecting the most clinically informative follow-up questions to route back into the Analytical Insight Agent across multiple iterative rounds. The accumulated findings across all cycles are finally synthesized into a structured insight catalog.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Pathology Image I + Clinical Goal G"] --> B["Visual Root Finder<br/>Visual Summarization + Web Retrieval + Root Question Generation"]
    B --> C["Analytical Insight Agent<br/>PathGen-LLaVA Evidence Extraction + Grounded Q&A and Insight Synthesis"]
    C --> D["Follow-Up Question Composer<br/>Candidate Question Generation + Goal-Aligned Question Selection"]
    D -->|Adaptive Closed-Loop Routing: Optimal c_best Fed Back| C
    D -->|Terminating Exploration Cycles, Aggregating Evidence| E["Output: High-Fidelity Insights S and Evidence Chains A"]

Key Designs

1. Visual Root Finder: Visual Summarization and Knowledge-Retrieval Grounded Root Question Formulation

General-purpose LMMs tasked with analyzing whole-slide images frequently suffer from unguided attention, formulating questions that are disconnected from the primary clinical diagnostic goal. To resolve this issue, the Visual Root Finder (VRF) decomposes inquiry initialization into visual grounding and domain knowledge retrieval rather than unconstrained querying. An Image-Summarization Module \(\text{ISM}_{\text{img}}\) first examines the downsampled pathology slide to identify prominent histological patterns, tissue architecture, cellularity, and necrosis, outputting key visual observations \(F\) and representative clinical keywords \(K\). Simultaneously, a Web-Retrieval Module \(\text{WRM}\) queries external clinical guidelines and biomedical literature using \(K\), retrieving the top-10 most relevant domain knowledge entries \(D = \{d_1, \dots, d_{10}\}\). Finally, a Root Question Generator synthesizes the pathology image \(I\), clinical goal \(G\), visual cues \(F\), and external knowledge \(D\) into a focused set of root questions \(Q = \{q_i\}_{i=1}^m\): $\(Q = \mathcal{L}(I, G, F, D)\)$ This mechanism anchors the downstream exploration in observable morphology and verified pathology standards, preventing arbitrary reasoning trajectories.

2. Analytical Insight Agent: Pathology-Specialized Tooling for Evidence-Grounded Insight Derivation

Direct end-to-end question answering on pathology slides regularly induces fatal clinical hallucinations, where models extrapolate severe prognoses without verifiable histological support. To overcome this vulnerability, the Analytical Insight Agent (AIA) enforces a two-tier paradigm of explicit evidence extraction followed by grounded insight synthesis. For each question \(q_i\), AIA queries an Image-Analysis Tool \(\text{IAT}\) instantiated via PathGen-LLaVA—a vision-language model pre-trained and instruction-tuned on 1.6 million pathology image-text pairs. The tool scans slide \(I\) to output structured morphological findings, regions of interest (ROI), estimated physical dimensions, and confidence boundaries \(E_i = \text{IAT}(I, q_i)\). Subsequently, the Answers and Insights Generator \(\mathcal{G}\) ingests \(q_i\), image \(I\), and the explicit evidence packet \(E_i\) to generate a concise, factual answer \(a_i\) accompanied by a grounded clinical insight \(s_i\): $\((A, S) = \bigcup_{i=1}^m \mathcal{G}(q_i, I, E_i)\)$ By forcing all clinical inferences to pass through an intermediate, auditable evidence extraction layer, AIA strictly penalizes unsupported extrapolation.

3. Follow-Up Question Composer: Multi-Perspective Candidate Generation and Goal-Directed Adaptive Routing

Initial root questions inherently possess bounded diagnostic breadth and stochastic variation, often failing to probe occult secondary features such as multifocal vascular invasion or margin clearance. To overcome this limitation, the Follow-Up Question Composer (FQC) establishes an adaptive, iterative inquiry loop. The module comprises a Follow-Up Question Generator \(\mathcal{F}\) and a Question Selector \(\mathcal{S}\). Generator \(\mathcal{F}\) inspects the current analysis state—including original image \(I\), goal \(G\), current question-answer pair \((q_i, a_i)\), global visual observations \(F\), and retrieved context \(D\)—to formulate a diverse candidate pool of \(n\) follow-up questions \(C = \{c_j\}_{j=1}^n\). The Question Selector \(\mathcal{S}\) then scores each candidate based on its semantic complementarity with previous questions and relevance to goal \(G\), selecting the single optimal candidate \(c_{\text{best}} = \mathcal{S}(C)\). This optimal question is dynamically routed back to AIA for subsequent evidence extraction and insight deduction. Controlled by an exploration depth hyperparameter \(p\), the framework expands each root question across \(p\) successive rounds, yielding a cumulative volume of \(\text{Ins} = r \times (p + 1)\) validated insights.

Benchmark & Evaluation Protocol

To benchmark goal-driven medical insight discovery, the authors created MedInsightBench. Originating from The Cancer Genome Atlas (TCGA), the benchmark curates 332 representative oncology cases spanning diverse malignancies (e.g., breast invasive carcinoma, lung adenocarcinoma, prostate adenocarcinoma). WSIs are standardized via pyramidal downsampling and color normalization, while corresponding pathology reports undergo OCR extraction, LLM-assisted decomposition, and multi-tier human pathologist verification. The resulting dataset contains 3,933 expert-validated question-insight-evidence triplets categorized into 6 distinct insight types (Descriptive: 1,580; Diagnostic: 1,122; Predictive: 256; Prescriptive: 194; Evaluative: 263; Exploratory: 518) across 4 difficulty tiers (Level 1: 1.8%; Level 2: 48.2%; Level 3: 47.0%; Level 4: 3.0%).

The evaluation framework introduces four complementary metrics: 1. Insight Recall: Assesses the coverage of ground-truth clinical findings (\(GT\)) within the generated insight set (\(I\)) using an evaluator \(S\) (ROUGE-1 or G-Eval based on GPT-3.5-Turbo and Gemini-2.5-Pro): $\(\text{Score}_{\text{recall}}(S) = \mathbb{E}_{gt \sim \text{Unif}(GT)} \max_{i \in I} S(gt, i)\)$ 2. Insight Precision: Measures the factual groundedness of generated insights, penalizing hallucinations: $\(\text{Score}_{\text{precision}}(S) = \mathbb{E}_{i \sim \text{Unif}(I)} \max_{gt \in GT} S(i, gt)\)$ 3. Insight F1: Harmonic balance between completeness and clinical faithfulness: $\(\text{Score}_{F1} = \frac{2 \cdot \text{Score}_{\text{recall}} \cdot \text{Score}_{\text{precision}}}{\text{Score}_{\text{recall}} + \text{Score}_{\text{precision}}}\)$ 4. Insight Novelty: Quantifies the system's capacity to uncover valid, unannotated clinical insights beyond reference labels. Insights that pass precision filtering are adjudicated by an ensemble of three distinct LMMs; an insight is deemed genuinely novel when validated by \(\ge 2\) evaluators, computing both Original (\(\delta=0\)) and Innovation (\(\delta=1\)) scores.

Key Experimental Results

Main Results

The primary benchmark evaluation compares direct-decoding LMM baselines against representative agent frameworks, with all agentic methods standardized on the Qwen3-VL-32B-Instruct backbone under an identical total inquiry budget (4 iterations with 3 questions per round).

Model / Method Paradigm / Backbone ROUGE-1 Recall G-Eval Recall ROUGE-1 Precision G-Eval Precision ROUGE-1 F1 G-Eval F1 Novelty Original Novelty Innovation
GPT-4o Direct Decoding (LMM) 0.1799 0.2978 0.2091 0.3577 0.1934 0.3250 0.1293 0.2094
GPT-5 Direct Decoding (LMM) 0.1866 0.3052 0.1851 0.3649 0.1858 0.3324 0.1322 0.2131
Deepseek-VL2 Direct Decoding (LMM) 0.1816 0.3226 0.2274 0.4070 0.2019 0.3599 0.1959 0.2712
InternVL3-38B Direct Decoding (LMM) 0.1774 0.3390 0.2002 0.3993 0.1881 0.3667 0.1607 0.2553
PathGen-LLaVA Direct Decoding (LMM) 0.1849 0.3420 0.1989 0.3845 0.1916 0.3620 0.1893 0.2936
Qwen2.5-VL-32B Direct Decoding (LMM) 0.1917 0.3975 0.2139 0.4848 0.2022 0.4368 0.3493 0.4171
Qwen3-VL-32B Direct Decoding (LMM) 0.2064 0.4295 0.2267 0.5372 0.2170 0.4774 0.3531 0.4488
Chain-of-Thought Qwen3-VL-32B 0.2053 0.4379 0.2281 0.5374 0.2161 0.4826 0.3549 0.4583
ReAct (General Tools) Qwen3-VL-32B 0.2092 0.4377 0.2344 0.5390 0.2211 0.4831 0.3492 0.4456
ReAct + Same Tools Qwen3-VL-32B 0.2105 0.4562 0.2397 0.5698 0.2242 0.5067 0.3853 0.4679
Plan-and-Solve Qwen3-VL-32B 0.2117 0.4534 0.2378 0.5633 0.2240 0.5024 0.3749 0.4613
Self-Refine Qwen3-VL-32B 0.2080 0.4467 0.2342 0.5554 0.2203 0.4952 0.3584 0.4493
Reflexion Qwen3-VL-32B 0.2125 0.4573 0.2395 0.5670 0.2252 0.5063 0.3826 0.4548
MedInsightAgent Qwen3-VL-32B 0.2233 0.4703 0.2483 0.5921 0.2351 0.5242 0.4274 0.4995

Ablation Study

The ablation analysis systematically isolates the individual and joint contributions of each module within MedInsightAgent on Qwen3-VL-32B-Instruct.

Configuration Retained Modules (ISM/WRM/IAT/FQC) G-Eval Recall G-Eval Precision G-Eval F1 Insights Novelty Impact / \(\Delta\) F1
MedInsightAgent (Full) ✓ / ✓ / ✓ / ✓ 0.4703 0.5921 0.5242 0.4995 Full pipeline baseline
w/o Image-Summarization (ISM) ✗ / ✓ / ✓ / ✓ 0.4615 0.5764 0.5126 0.4780 Loss of visual anchor; F1 drops 0.0116
w/o Web-Retrieval (WRM) ✓ / ✗ / ✓ / ✓ 0.4623 0.5739 0.5121 0.4705 Loss of domain context; Novelty drops 0.0290
w/o Image-Analysis Tool (IAT) ✓ / ✓ / ✗ / ✓ 0.4537 0.5604 0.5014 0.4842 Loss of pathology grounding; F1 drops 0.0228
w/o Follow-Up Composer (FQC) ✓ / ✓ / ✓ / ✗ 0.4391 0.5511 0.4888 0.4653 Absence of multi-turn depth; F1 plummets 0.0354
Joint: w/o ISM & WRM ✗ / ✗ / ✓ / ✓ - - 0.4996 0.4607 Synergistic loss; F1 drop (-0.0246) exceeds sum
Joint: w/o ISM, WRM, IAT ✗ / ✗ / ✗ / ✓ - - 0.4889 0.4524 Zero-tool pipeline still exceeds CoT (0.4826)

Key Findings

  • Orchestration Matters More Than Tool Availability: When ReAct is equipped with identical external modules (ReAct + same tools), its G-Eval F1 reaches 0.5067, significantly lagging behind MedInsightAgent's 0.5242 (Bootstrap significance \(p=0.010\), \(B=10,000\)). This empirically demonstrates that unstructured tool calling cannot match structured task decomposition and role-specialized interaction.
  • Critical Role of FQC and IAT: Ablation reveals that removing FQC incurs the largest degradation across all metrics (\(-0.0354\) F1), proving that single-step questioning cannot unravel complex multi-hop pathological dependencies. Removing IAT causes the sharpest precision drop (\(-0.0317\)), confirming that specialized pathology visual perception is indispensable for curbing clinical hallucinations.
  • Precision Outstrips Recall Across All Models: Precision universally exceeds recall across all evaluated models, indicating that models naturally adopt conservative generation strategies that favor safe, trivial observations over aggressive exploration of occult pathological patterns.
  • Computational Investment Translates to Expert-Audited Value: While MedInsightAgent incurs higher inference latency (958.9s and 32 API calls per sample vs. 133.4s for ReAct), independent human expert auditing of novelty-filtered insights indicates that 43.3% of its discoveries exhibit genuine clinical novelty (Category A), with only 13.4% categorized as trivial (Category C), compared to 26.7% and 30.0% for raw Qwen3-VL-32B.

Highlights & Insights

  • Formulation of Pathological Knowledge Routing: Successfully formalizes complex medical insight generation as an adaptive graph search problem: anchoring initial exploration via visual root finding, grounding local observations via specialized microscopic tools, and expanding the inquiry frontier through goal-aligned follow-up generation.
  • Rigorous Multimodal Evaluation Harness: Combines lexical overlap (ROUGE-1), LLM-based clinical semantic alignment (G-Eval via GPT-3.5 and Gemini-2.5), and multi-judge cross-validation (Novelty) to provide a mathematically sound protocol for open-ended medical evaluation.
  • High Transferability: The underlying multi-agent design—comprising coarse visual grounding, domain-specific retrieval, high-resolution regional tool execution, and goal-directed inquiry selection—is readily applicable to radiology analysis, complex surgical video reasoning, and multi-omics scientific discovery.

Limitations & Future Work

  • Cohort and Modality Scope: The benchmark is currently derived from TCGA whole-slide oncology data. Incorporating radiology scans (CT/MRI) and multi-center electronic health record (EHR) longitudinal data remains necessary to evaluate broader generalizability.
  • Whole-Slide Downsampling Constraints: Compressing large SVS whole-slide images to fixed-dimension PNGs risks obscuring fine nuclear atypia and individual mitotic counts; future iterations will benefit from dynamic multi-scale patch zooming.
  • Benchmarking vs. Prospective Clinical Utility: The evaluation measures retrospective concordance against diagnostic reports; real-world clinical trials will be essential to establish whether agent-derived insights improve patient survival outcomes.
  • vs InsightBench & Agent Poirot: InsightBench pioneers multi-step analytics over tabular enterprise data using Python/SQL agents; MedInsightBench extends this philosophy into complex, high-resolution visual pathology by coupling visual grounding with clinical narratives.
  • vs ReAct & Reflexion: Generic agent prompting architectures interleave unconstrained thought and tool tokens within a single model context, frequently wandering off-topic in dense biomedical scenarios; MedInsightAgent implements specialized agent roles with strict input-output contracts.
  • vs PMC-VQA & MedRepBench: Existing vision-language benchmarks target single-turn visual QA or report summarization; MedInsightBench uniquely evaluates goal-directed, multi-step hypothesis generation and actionable insight discovery.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ First comprehensive benchmark and specialized multi-agent framework dedicated to goal-driven multi-step insight discovery from pathology images.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive benchmarking of 7 LMMs, 5 agent architectures, extensive ablations, bootstrap statistical validation, and blind human pathologist audits.
  • Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical formalization, cohesive narrative structure, and exhaustive disclosure of prompts, cases, and computational cost profiles.
  • Value: ⭐⭐⭐⭐⭐ Sets a new standard for evaluating and developing clinical analytics agents capable of genuine scientific exploration.