Toward Interpretable Analysis of Whole-slide Pathology Images via Large Language Model-based Agentic Reasoning

Conference: ECCV 2026
Paper: ECCV 2026
Code: https://github.com/G14nTDo4/PathAgent
Area: LLM Reasoning / Multimodal VLM / Medical Imaging
Keywords: Whole-slide Pathology Images, Agentic System, Multi-step Reasoning, Visual Question Answering, Interpretability

TL;DR

PathAgent is a training-free LLM-based agent framework for whole-slide pathology image (WSI) analysis that emulates the iterative, reflective inspection workflow of human pathologists using a Navigator, a Perceptor, and an Executor, establishing fully traceable and clinically grounded evidence trajectories.

Background & Motivation

Analyzing whole-slide pathology images (WSIs), which frequently span up to \(100\text{k} \times 100\text{k}\) pixels, presents formidable challenges in computational pathology (CPath). An ideal computational pipeline must concurrently identify sparse micro-scale diagnostic cues, navigate gigapixel fields of view, and reason over shifting visual contexts. Existing computational methods struggle to balance global context, fine-grained morphology, and decision transparency: CLIP-style cross-modal alignment models rely on rigid predefined templates and lack long-form reasoning; multimodal vision-language models (VLMs) have limited context windows and receptive fields, forcing severe downsampling into thumbnails or uncontextualized random patch crops; and recent patch-aggregated LLMs conduct black-box predictions that lack traceable visual evidence to validate their diagnostic conclusions.

This disconnect fundamentally conflicts with genuine clinical pathology workflows. Pathologists analyze glass slides not via a single-pass monolithic scan, but through a reflective, goal-driven iterative process: they first survey the WSI at low magnification (e.g., 5×) to locate suspicious regions of interest (RoIs), progressively zoom into fine-grained magnifications (e.g., 20× or 40×) to scrutinize diagnostic cytological details such as cellular atypia and mitotic figures, and self-correct or gather additional evidence if the current findings remain inconclusive, terminating only when diagnostic sufficiency is achieved.

This paper addresses this gap by decoupling vision models and reasoning engines without requiring expensive task-specific fine-tuning or end-to-end retraining, replicating the pathologist's multi-scale examination logic via an agentic architecture. Core idea: construct PathAgent, a training-free pathology agent that frames WSI analysis as a closed-loop "Navigator retrieval - Perceptor description - Executor multi-step reasoning" trajectory, dynamically deciding between horizontal evidence retrieval, vertical adaptive zooming, or diagnostic conclusion based on explicit self-reflection.

Method

Overall Architecture

PathAgent formulates WSI analysis as a sequential, evidence-grounded agentic workflow. The formal trajectory across iterations is represented by state-action-finding triplets \(\{(S_t, A_t, X_t) \mid 1 \le t \le T\}\), where \(T\) is the computational iteration budget, \(S_t\) represents accumulated analytic states, \(A_t\) denotes the chosen action, and \(X_t\) denotes question-relevant visual findings.

The framework coordinates three core off-the-shelf modules: the Navigator for rapid slide-level RoI filtering, the Perceptor for fine-grained morphological visual cue extraction, and the Executor as the central reasoning orchestrator. At iteration \(t\), the Executor conducts multi-step reasoning to evaluate evidence sufficiency and select one of three actions: Action 1 (exploring additional evidence in unexamined regions), Action 2 (zooming in for finer details at higher magnifications), or Action 3 (concluding the analysis with an evidence-grounded rationale).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: WSI and clinical diagnostic question $q$"] --> B["Navigator-guided RoI Retrieval<br/>Compute visual-text relevance to prioritize top candidate patches"]
    B --> C["Perceptor-driven Morphological Extraction<br/>Generate pathology descriptions and build analytic state $S_t$"]
    C --> D["Executor Multi-Step Reasoning<br/>Predict answer, evaluate evidence sufficiency, and explore missing info"]
    D -->|Evidence insufficient & needs broader context| E["Explore Additional Evidence<br/>Retrieve unexamined patches using missing info query"]
    D -->|Evidence insufficient & needs cytological details| F["Adaptive Multi-scale Zooming<br/>Partition into higher magnification and merge local descriptors"]
    E --> C
    F --> C
    D -->|Evidence sufficient or budget exhausted| G["Synthesize Traceable Diagnostic Conclusion<br/>Output final answer grounded on multi-scale spatial coordinates"]

Key Designs

1. Navigator-guided RoI Retrieval: Overcoming gigapixel scale exploration bottlenecks

Directly ingesting tens of thousands of patches from a gigapixel WSI into an LLM causes severe context window overflow and brings substantial background noise. PathAgent introduces a pathology CLIP model (such as PLIP) as the Navigator \(F_N\). The WSI is initially decomposed into non-overlapping patches \(X = \{x_i\}_{i=1}^N\) at low magnification (5×). The Navigator computes the cross-modal relevance score \(r_i^1\) between the visual feature embedding of each patch \(x_i\) and the clinical question \(q\): $\(X_1 = \{x_i \mid i \in \operatorname{Top}_{k_1}(\{r_i^1\}_{i=1}^N)\}\)$ Setting the selection ratio to \(k_1 = \lceil 0.1 N \rceil\) filters out over 90% of redundant tissue/background while reliably capturing potential tumor nests within a manageable computational budget.

2. Perceptor-driven Morphological Extraction: Grounding visual pathology semantics

To equip the text-centric LLM with grounded morphological comprehension, the Perceptor \(F_P\) employs an advanced pathology VLM (such as Patho-R1). For all candidate patches in \(X_1\), the Perceptor produces baseline morphological descriptions \(\operatorname{Des}(x_i)\). Furthermore, to avoid missing task-critical cues, the Perceptor feeds a targeted question-guided prompt ("Please describe the pathology features related to the question: [QUESTION] in this image") to the top 5 most relevant patches, synthesizing a structured initial analytic state: $\(S_1 = \{(M_1, \operatorname{Loc}(x_i), \operatorname{Des}(x_i)) \mid x_i \in X_1\}\)$ where \(M_1\) denotes magnification (5×), \(\operatorname{Loc}(x_i)\) stores the spatial coordinates, and \(\operatorname{Des}(x_i)\) provides clinical visual descriptions.

3. Executor Multi-Step Reasoning: Dynamic, reflective decision trajectories

Rather than relying on ungrounded single-step generation, the Executor \(F_E\) decomposes the decision cycle into three chained reasoning phases: - Predict Answer: Generates a tentative diagnosis \(\hat{Y}_t\) alongside intermediate thinking steps \(\hat{R}_t\) grounded on state \(S_t\) and question \(q\); - Self Reflect: Evaluates whether the current visual findings are morphologically sufficient to confidently substantiate \(\hat{Y}_t\), outputting a binary signal \(C \in \{\text{Yes}, \text{No}\}\); - Explore Missing Information: If \(C = \text{No}\), the Executor articulates missing visual evidence \(I_t\) (e.g., infiltrative growth, stromal reactions), recommends an appropriate magnification scale \(M_t\) (e.g., 20× or 40×), and provides an explicit zoom recommendation \(Z \in \{\text{Yes}, \text{No}\}\).

4. Adaptive Multi-scale Zooming: Unifying macroscopic architecture and microscopic cytology

Certain clinical tasks (e.g., Nottingham histological grading) strictly require cellular and nuclear details. When Action 2 is triggered, the target patch is partitioned into finer non-overlapping patches at magnification \(M_t\), forming \(X_{\text{mag}}\). To curb token explosion, the Navigator refilters \(X_{\text{mag}}\) guided by the missing information query \(I_t\), and the Perceptor extracts fine-grained descriptors for the selected high-magnification patch \(x_{\text{mag}}\), updating the cumulative state: $\(S_t \leftarrow S_t \cup \{(M_t, \operatorname{Loc}(x_{\text{mag}}), \operatorname{Des}(x_{\text{mag}}))\}\)$ When Action 3 terminates the loop, the Executor performs extractive summarization over historical states, retaining relevant morphological cues and producing a final answer grounded on an auditable chain of spatial coordinates.

A Worked Example

Consider a breast segmental mastectomy diagnostic case (TCGA-BH-A202-01Z) evaluating the question: "What type of carcinoma was found in the breast segmental mastectomy?" (Options: A. invasive ductal carcinoma, B. papilloma, C. no special type, D. fibroadenomatous changes): 1. Round 1 (5× Survey): The Navigator retrieves 5 initial low-power patches. The Perceptor notes ductal structures and tumor presence at coordinates (16016, 7240). The Executor infers tentative presence of carcinoma but concludes {“sufficient”: “No”} during self-reflection, issuing the exploration directive: "The carcinoma is located in the upper left quadrant, but infiltrative margins require examination at 20×." 2. Round 2 (20× Zoom & Target Retrieval): High-magnification sampling focuses on candidate regions; the Navigator selects patch (64576, 14992). The Perceptor at 20× reports: "Hyperchromatic nuclei varying in size, indicating high cellular atypia. Surrounding stroma exhibits marked inflammation and fibrosis, characteristic of invasive carcinoma." 3. Conclusion: The Executor self-reflects that diagnostic criteria for invasion are satisfied ({“sufficient”: “Yes”}), successfully outputs A. invasive ductal carcinoma, and provides an evidence chain referencing both the 5× ductal architecture and the 20× invasive stromal reaction coordinates.

Loss & Training

PathAgent is completely training-free. It requires no parameter updates, instruction-tuning datasets, or reinforcement learning policies. Instead, it relies on structured prompt schemas, JSON state outputs, and multi-model collaboration. The computational budget is constrained by setting \(k_1 = \lceil 0.1 N \rceil\), subsequent iteration retrieval \(k_t = \lceil 0.05 N \rceil\), patch size \(512 \times 512\) at initial 5× magnification, and a maximum iteration threshold \(T=5\).

Key Experimental Results

Main Results

The framework is evaluated across five benchmark datasets: SlideBench-VQA, WSI-VQA, WSI-Bench, PathMMU, and PathVQA. In the offline configuration (PathAgent-offline), PLIP, Patho-R1-7B, and Qwen3-4B serve as Navigator, Perceptor, and Executor, respectively; in PathAgent-online, the Executor utilizes Qwen3-235B-A22B.

The table below reports zero-shot visual question answering performance on SlideBench-VQA (BCNB), WSI-VQA, and PathVQA (key results from Table 1 of the paper):

Method (Input Format)	Tumor Type	Receptor Status	HER2 Expression	Histological Grading	Molecular Subtype	BCNB Average Acc (%)	WSI-VQA Acc (%)	PathVQA Acc (%)
Qwen3-VL (Thumbnail)	41.48	50.63	19.37	25.95	11.03	30.70	21.84	-
GPT-4o (30 Random Patches)	34.69	59.32	23.95	28.63	23.15	38.94	27.57	46.59
LLaVA-Med (Patch)	23.95	42.52	23.72	18.99	15.05	30.10	21.55	27.78
Quilt-LLaVA (Patch)	77.14	56.46	23.18	18.23	19.82	44.43	30.17	20.76
WSI-VQA (Slide)	3.90	40.82	10.53	30.00	0.00	23.35	46.90	33.59
SlideChat (Slide)	90.17	73.09	25.05	23.11	17.49	54.14	-	55.03
WSI-LLaVA (Slide)	85.32	60.89	24.75	46.28	29.20	52.68	54.63	54.97
GIANT (Slide)	57.37	42.55	14.79	48.52	26.63	44.97	47.05	-
PathAgent-offline	87.52	61.33	25.34	55.95	30.21	55.72	56.32	58.36
PathAgent-online	91.31	75.87	27.26	56.21	35.88	59.31	60.74	62.23

On the patch-level benchmark PathMMU, PathAgent-online achieves an overall accuracy of 72.54%, surpassing human expert performance (71.82%) and existing state-of-the-art models such as SlideChat (50.82%) and GPT-4o (49.31%).

Ablation Study

Ablation analysis on the Multi-Step Reasoning (MSR) mechanism and Navigator RoI filtering on SlideBench-VQA (BCNB) (data from Table 4 of the paper):

Multi-Step Reasoning	Navigator	Tumor Type	Receptor Status	HER2 Expression	Histological Grading	Molecular Subtype	BCNB Average Acc (%)	Average Iterations
✗	✗ (Random Sampling)	38.46	52.38	20.35	39.42	20.19	41.62	1.83
✓	✗ (Random Sampling)	52.43	53.89	23.13	44.16	26.28	44.97	1.59
✗	✓	78.88	58.27	23.40	45.46	27.42	50.68	1.32
✓	✓	87.52	61.33	25.34	55.95	30.21	55.72	1.21

Key Findings

• Synergy of Navigation and Reflection: Introducing the Navigator alone boosts average accuracy by +9.06% (41.62% \(\to\) 50.68%), underscoring that locating diagnostic regions is essential. Adding Multi-Step Reasoning further elevates accuracy to 55.72% while reducing average iterations from 1.83 to 1.21, proving that reflection drives faster and more targeted convergence.
• Differential Domain Gains: The Navigator provides the largest gain in Receptor Status (52.38% \(\to\) 58.27%), highlighting spatial localization benefits. Conversely, Multi-Step Reasoning contributes the most to Molecular Subtype (20.19% \(\to\) 26.28%), confirming that complex inferential classification requires structured multi-step logical deduction.
• Initial Patch Ratio Saturation: Evaluating patch sampling ratios indicates that accuracy peaks at 10% sampling. Increasing patches to 15% or 20% degrades accuracy due to redundant background noise interfering with LLM reasoning.

Highlights & Insights

• Pragmatic Training-free Paradigm: Eliminates the need for expensive, proprietary expert reasoning trajectory datasets. Orchestrating off-the-shelf foundation models achieves state-of-the-art diagnostic accuracy with zero training overhead.
• Clinical Fidelity via Visual Turing Test: In a visual Turing test with 22 human evaluators, judges distinguished random patches from pathologist-selected RoIs with 85.1% accuracy, but achieved only 48.3% accuracy distinguishing PathAgent-selected RoIs from pathologist selections (close to the 50.0% random guess threshold), confirming expert-level selection fidelity.
• Seamless Human-in-the-Loop Collaboration: The transparent trajectory allows pathologists to intervene at any intermediate step (adjusting RoIs, refining descriptions, or overriding magnification). Statistical testing confirms that human intervention significantly increases diagnostic accuracy and reduces iteration counts.

Limitations & Future Work

• Morphological Ceiling for Molecular Profiling: On tasks like HER2 expression and histological sub-grading, accuracy gains remain modest (HER2 ~27%) because clinical standards rely on molecular IHC or FISH assays rather than H&E morphological architecture alone.
• Text-Visual Retrieval Drift: The zero-shot CLIP Navigator can occasionally miss subtle diffuse lesions in specialized sub-benchmarks (such as PathCLS) due to domain vocabulary discrepancies, suggesting the need for topology-aware spatial graphs.

Related Work & Insights

• vs SlideChat / WSI-LLaVA: While patch-aggregated VLMs process compressed WSI features through monolithic black-box projections, PathAgent logs explicit multi-scale patch coordinates and reasoning justifications at every step, offering genuine clinical traceability.
• vs PathFinder / CPathAgent: Prior agentic frameworks depend on complex multi-stage supervised fine-tuning or proprietary clinical trajectories. PathAgent demonstrates that modular zero-shot coordination of open models can achieve superior diagnostic performance without retraining.

Rating

• Novelty: ⭐⭐⭐⭐☆ (Deconstructs clinical pathology inspection workflows into an intuitive, training-free agentic reasoning trajectory)
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Evaluated on five diverse benchmarks, comprehensive ablations, a visual Turing test, and interactive clinician trials)
• Writing Quality: ⭐⭐⭐⭐⭐ (Crisp methodology framing, elegant figures, transparent ablation metrics)
• Value: ⭐⭐⭐⭐⭐ (Presents a highly practical and interpretable blueprint for gigapixel medical imaging analysis)

Related Papers

• [ECCV 2026] TIR-Bench: A Comprehensive Benchmark for Agentic Thinking-with-Images Reasoning
• [ECCV 2026] BrepCoder: A Unified Multimodal Large Language Model for Multi-task B-rep Reasoning
• [ICLR 2026] SportR: A Benchmark for Multimodal Large Language Model Reasoning in Sports
• [ECCV 2026] Human-Level Accuracy, Non-Human Strategies: Revealing Model-Human Divergence in Video Physical Reasoning
• [ICML 2026] Vision-aligned Latent Reasoning for Multi-modal Large Language Model