AGENTEHR: Advancing Autonomous Clinical Decision-Making via Retrospective Summarization¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/BlueZeros/AgentEHR
Area: LLM Reasoning
Keywords: Electronic Health Records, clinical decision-making agent, retrospective summarization, evolving memory bank, long-horizon temporal reasoning
TL;DR¶
Addressing the critical information loss and fractured reasoning continuity caused by unidirectional context compression in noisy real-world EHRs, this paper introduces the AGENTEHR benchmark covering six patient lifecycle tasks and proposes RETROSUM, a framework uniting retrospective summarization with an evolving dual-experience memory bank to retain cross-temporal clinical dependencies while reducing systematic interaction errors by up to 92.3%.
Background & Motivation¶
Large Language Models have demonstrated profound utility across medical dialogue, automated documentation, and diagnostic question answering, functioning as versatile catalysts for clinical AI applications. However, existing research applying LLMs to Electronic Health Records (EHRs) remains predominantly confined to heavily curated inputs, isolated risk prediction tables, or simplified single-turn factual retrieval. Such artificial experimental setups sidestep the fundamental characteristics of authentic clinical workflows: extreme observational noise, cross-table schema heterogeneity, and longitudinal patient trajectories spanning months or years. Consequently, models that excel in idealized laboratory settings frequently falter when deployed in raw healthcare environments.
Concurrently, recent agent-based EHR exploration systems typically restrict the agent's agency to query rewriting or factual lookup—such as identifying a patient's most recently prescribed medication. This formulation effectively downgrades the model into an interactive search engine interface, failing to engage its multi-step diagnostic reasoning, hypothesis testing, and therapeutic synthesis capabilities. More critically, to handle the rapid context bloat inherent in navigating extensive relational database tables, prevailing agent architectures rely on vanilla unidirectional summarization methods (such as ReSum). These paradigms compress context strictly forward in time, summarizing past observations locally and replacing the raw interaction trace with a condensed abstract.
In medical diagnostics, this unidirectional compression introduces a fatal cognitive failure: clinical symptoms and laboratory findings exhibit profound cross-temporal latency. A mild fever, slightly elevated white blood cell count, or minor electrolyte shift observed in early turns may appear non-specific and be discarded by local summarization; only dozens of turns later, upon observing a spiked lactate level or positive blood culture, does that early evidence become definitive for diagnosing septic shock. Unidirectional compression irreversibly severs these latent cross-temporal dependencies, and stripping away raw syntactic SQL outputs leaves the agent vulnerable to formatting loops and dead ends. The core idea is to formalize long-horizon clinical decision-making over raw relational EHR databases as AGENTEHR across six core hospitalization tasks, and propose RETROSUM—a framework that retains full raw history for the actor, periodically re-evaluates complete trajectories via retrospective summarization, and retrieves dual procedural-and-salience heuristics from an evolving experience memory bank to eliminate context-loss deadlocks.
Method¶
Overall Architecture¶
The AGENTEHR framework casts clinical problem-solving as an interactive database trajectory over discrete steps \(i \in \{1, \dots, K\}\). Given a patient query instance \(X = \{p, t, I\}\)—comprising patient identifier \(p\), temporal censoring timestamp \(t\) (strictly precluding future outcome leakage), and clinical instruction \(I\)—the agent interacts with an underlying SQLite database \(E\) spanning over 30 clinical tables. At step \(i\), the agent observes context \(\hat{H}_{i-1}\) and generates an action \(a_i \sim \pi_\theta(a_i \mid \hat{H}_{i-1}, X)\). The execution yields observation \(o_i = E(a_i)\), culminating in a termination signal that produces final prediction set \(Y = \{y_1, \dots, y_m\}\).
To overcome the twin vulnerabilities of lost latent dependencies and fractured deduction logic, RETROSUM establishes a dual-view reasoning loop: the actor executes actions while retaining visibility of the uncompressed interaction history augmented with the latest retrospective summary, while the summarizer periodically conditions on both recent windowed actions and distant historical records to re-calibrate evidence significance. In addition, an offline reflection loop distills procedural action rules and salience guidelines into an external experience memory bank indexed by patient state embeddings, guiding both generation modules during test-time inference.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Clinical Query X and Censored EHR Database E"] --> B["Multi-Functional MCP Toolbox Interaction<br/>Record querying, schema inspection, and candidate alignment"]
B --> C["Dual-View Retrospective Summarization<br/>Jointly conditioning on recent window and distant full history"]
C --> D["Full-History Augmented Retrospective Inference<br/>Retaining raw execution trace with cognitive summary guidance"]
D --> E["Dual-Reflection Evolving Experience Memory Bank<br/>Retrieving procedural actor and salience summarizer heuristics"]
E -->|Termination condition unmet| B
E -->|Finish tool invoked| F["Output Standardized Clinical Prediction Set Y"]
Key Designs¶
1. Multi-Functional MCP Toolbox Interaction: Standardized hierarchical database operations
To enable robust, scalable interaction across relational clinical data without flooding the model context, the framework hosts a suite of 19 specialized tools categorized into five functional modules via a Model Context Protocol (MCP) server. The Record module supports targeted SQL queries, timestamp-bounded filtering, keyword extraction, exact column matching, and event frequency counts, allowing agents to extract longitudinal measurements without loading gigabyte-scale logs. The Table module provides schema inspection, returning table lists, column types, and database dictionary descriptions so agents formulate syntactically sound queries. The Candidate module bridges free-text medical hypotheses and standardized ontology targets (such as CCS-mapped ICD codes and ATC drug classes) using fuzzy text matching and BioLORD-2023 biomedical semantic embeddings. The Inner module separates internal deliberation (Think) from task resolution (Finish), while the Knowledge module connects the agent to PubMed, medical textbooks, and StatPearls clinical guidelines for evidence-based differential diagnosis.
2. Dual-View Retrospective Summarization: Re-evaluating distant evidence in light of fresh findings
Standard incremental summarizers irrevocably discard subtle, early clinical indicators because their diagnostic significance is not yet apparent. RETROSUM introduces a periodic retrospective summarization process triggered every \(w\) steps (where \(j \equiv 0 \pmod w\)). At each trigger point \(j\), the trajectory is conceptually partitioned into distant history \(H_{\text{dist}} = \{(a_1, o_1), \dots, (a_{j-w}, o_{j-w})\}\) and the recent window \(H_{\text{rec}} = \{(a_{j-w+1}, o_{j-w+1}), \dots, (a_j, o_j)\}\). The Summarizer module \(M_\theta\) (parameter-shared with policy \(\pi_\theta\)) generates updated summary \(S_j\) by retrospectively conditioning on both partitions, the preceding summary, and the clinical query: $\(S_j = M_\theta(H_{\text{rec}}, S_{j-w}, H_{\text{dist}}, X)\)$ By re-evaluating the complete trajectory against newly observed laboratory or microbiological results, \(M_\theta\) recovers dormant cross-temporal connections—such as linking early borderline vital signs with recent elevated lactate to confirm a previously obscured sepsis trajectory.
3. Full-History Augmented Retrospective Inference: Preserving syntactic precision and unbroken deduction
Prior summarization agents replace raw conversation traces with highly condensed text abstracts, which inadvertently strips the exact column names, numerical thresholds, and SQL tokens needed to formulate consistent follow-up queries. RETROSUM preserves reasoning continuity by maintaining the complete raw trajectory while overlaying the latest retrospective summary as an attention regularizer. At any step \(i\), the augmented context \(\hat{H}_i\) is updated as: $\(\hat{H}_i = \begin{cases} \hat{H}_{i-1} \setminus \{S_{i-w}\} \cup \{(a_i, o_i), S_i\}, & \text{if } i \equiv 0 \pmod w \\ \hat{H}_{i-1} \cup \{(a_i, o_i)\}, & \text{otherwise} \end{cases}\)$ The agent samples next action \(a_i \sim \pi_\theta(a_i \mid \hat{H}_{i-1}, X)\). This dual-view formulation provides the actor with an unbroken chain of exact execution details, preventing structural syntax violations while utilizing the summary \(S\) as an executive clinical map that anchors reasoning on pertinent diagnostic milestones.
4. Dual-Reflection Evolving Experience Memory Bank: Offline strategy crystallization without model fine-tuning
General-purpose foundation models frequently lack the domain-specific intuition required to navigate idiosyncratic clinical schemas and detect subtle diagnostic signals. Rather than updating model parameters through costly gradient descent, RETROSUM adopts a non-parametric evolving strategy. On training episodes, the agent completes trajectory \(H_K\) yielding final summary \(S_{\text{final}}\) and prediction \(Y\). Reflection module \(R_\theta\) compares predictions with ground truth \(Y^*\) to extract two distinct types of heuristics: procedural actor guidelines \(E_{\text{act}} = R_\theta(H_K, Y, Y^*)\) (such as optimal tool sequencing and avoiding repetitive table polling) and information salience guidelines \(E_{\text{sum}} = R_\theta(H_K, S_{\text{final}}, Y, Y^*)\) (distinguishing crucial comorbidity signs from clinical noise). Memory bank \(\mathcal{B} = \{(e, E_{\text{act}}, E_{\text{sum}})_k\}_{k=1}^M\) stores these pairs keyed by patient state embedding \(e\). During test-time inference, the Top-1 most similar clinical profile is retrieved via bge-m3 and supplied to both modules: $\(S_i = M_\theta(H_{\text{rec}}, S_{i-w}, H_{\text{dist}}, X, E_{\text{sum}}), \quad a_{i+1} \sim \pi_\theta(a_{i+1} \mid \hat{H}_i, X, E_{\text{act}})\)$
Benchmark & Evaluation Protocol¶
The AGENTEHR benchmark is constructed from MIMIC-IV (~300k patients) and MIMIC-III (~1k patients). To rigorously test generalization, instances are stratified into three subsets: MIMIC-IV-Common (in-distribution baseline testing), MIMIC-IV-Rare (long-tail conditions assessing label-shift robustness via inverse-frequency weighting \(w_S\)), and MIMIC-III (systemic schema differences, 20× shorter timeline with 2× higher record density). Evaluation spans six comprehensive tasks: Diagnoses, Labevents, Microbiology, Prescriptions, Procedures, and Transfers. Performance is assessed using F1 score, alongside Best@K F1 to evaluate test-time scaling across trajectory sample budgets \(K \in [1, 256]\).
Key Experimental Results¶
Main Results¶
On the core MIMIC-IV-Common benchmark, RETROSUM and its evolving variant were evaluated across five LLM backbones against static baselines (ReAct, Reflexion, ReSum) and evolving baselines (ReflecTool, ReasoningBank). All values report F1 scores (extracted from Table 2 and Table 7 of the paper):
| LLM Backbone | Agent Method | Evolved | Diagnoses | Labevents | Microbiology | Prescriptions | Procedures | Transfers | Avg. F1 |
|---|---|---|---|---|---|---|---|---|---|
| Qwen3-30B-A3B | ReAct | ✗ | 0.0955 | 0.1295 | 0.1274 | 0.0756 | 0.2472 | 0.2772 | 0.1587 |
| Reflexion | ✗ | 0.1216 | 0.0983 | 0.0989 | 0.0901 | 0.2273 | 0.2292 | 0.1442 | |
| ReSum | ✗ | 0.1753 | 0.0881 | 0.1166 | 0.1000 | 0.2717 | 0.2320 | 0.1639 | |
| ReflecTool (CS) | ✓ | 0.1647 | 0.0821 | 0.1265 | 0.0614 | 0.2572 | 0.2589 | 0.1585 | |
| ReasoningBank | ✓ | 0.0385 | 0.1275 | 0.1250 | 0.0448 | 0.2931 | 0.2103 | 0.1399 | |
| RETROSUM (Ours) | ✗ | 0.2368 | 0.0977 | 0.1584 | 0.1039 | 0.2983 | 0.2791 | 0.1957 | |
| RETROSUM Evolved | ✓ | 0.2514 | 0.1092 | 0.1568 | 0.1289 | 0.3068 | 0.3171 | 0.2117 | |
| Qwen3-Next-80B-A3B | ReAct | ✗ | 0.1836 | 0.0793 | 0.1895 | 0.1075 | 0.2857 | 0.2346 | 0.1800 |
| ReSum | ✗ | 0.2161 | 0.1083 | 0.1597 | 0.1301 | 0.3503 | 0.2317 | 0.1994 | |
| ReasoningBank | ✓ | 0.2519 | 0.1998 | 0.1726 | 0.1267 | 0.3743 | 0.2793 | 0.2341 | |
| RETROSUM Evolved | ✓ | 0.3148 | 0.1990 | 0.1830 | 0.1429 | 0.3578 | 0.2573 | 0.2425 | |
| Qwen3-235B-A22B* | ReAct | ✗ | 0.1671 | 0.1172 | 0.1545 | 0.0670 | 0.2739 | 0.1413 | 0.1535 |
| ReSum | ✗ | 0.2804 | 0.1009 | 0.1570 | 0.0728 | 0.2907 | 0.0803 | 0.1637 | |
| RETROSUM Evolved | ✓ | 0.3700 | 0.1951 | 0.1791 | 0.1346 | 0.3556 | 0.2342 | 0.2448 | |
| GPT-5-mini | ReAct | ✗ | 0.4270 | 0.2151 | 0.1867 | 0.0764 | 0.2693 | 0.2053 | 0.2300 |
| ReSum | ✗ | 0.4024 | 0.2181 | 0.2090 | 0.0752 | 0.2668 | 0.1993 | 0.2285 | |
| RETROSUM Evolved | ✓ | 0.4127 | 0.2989 | 0.2097 | 0.1536 | 0.2603 | 0.2327 | 0.2613 | |
| Grok-4.1-fast | ReAct | ✗ | 0.3841 | 0.1856 | 0.2032 | 0.1688 | 0.3762 | 0.1826 | 0.2501 |
| ReSum | ✗ | 0.3822 | 0.0882 | 0.2024 | 0.1618 | 0.3186 | 0.1888 | 0.2237 | |
| RETROSUM (Ours) | ✗ | 0.4270 | 0.1859 | 0.2296 | 0.1846 | 0.3480 | 0.1901 | 0.2609 | |
| RETROSUM Evolved | ✓ | 0.4734 | 0.2321 | 0.2236 | 0.1861 | 0.3397 | 0.2729 | 0.2880 |
Cross-subset validation on out-of-distribution environments (Table 3) demonstrates consistent domain resilience: - MIMIC-IV-Rare: On Qwen3-30B-A3B, ReAct obtains an average F1 of 0.1255, ReSum reaches 0.1491, and ReasoningBank drops to 0.1068. RETROSUM Evolved achieves 0.1613, with Diagnoses rising from ReAct's 0.0863 to 0.2325. - MIMIC-III: When subjected to severe schema divergence and high record density, ReSum collapses to 0.1201 (below ReAct's 0.1319), while RETROSUM Evolved maintains a leading score of 0.1545 (surpassing ReSum by 28.6% and raising Procedures from 0.0185 to 0.0473).
Ablation Study¶
To isolate the individual and interactive benefits of the retrospective mechanism on the Actor and Summarizer, alongside memory evolution, component ablations were conducted on Qwen3-30B-A3B on MIMIC-IV-Common (Table 4):
| Config | Actor Retrospect | Actor Evolved | Summarizer Retrospect | Summarizer Evolved | Diagnoses | Labevents | Microbiology | Prescriptions | Procedures | Transfers | Avg. F1 | Description |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ① | ✗ | ✗ | ✗ | ✗ | 0.1753 | 0.0881 | 0.1166 | 0.1000 | 0.2717 | 0.2320 | 0.1639 | Vanilla ReSum baseline |
| ② | ✓ | ✗ | ✗ | ✗ | 0.2257 | 0.1041 | 0.1418 | 0.1030 | 0.2932 | 0.2577 | 0.1876 | Actor retains full history trace |
| ③ | ✗ | ✗ | ✓ | ✗ | 0.1759 | 0.1097 | 0.1710 | 0.1119 | 0.2320 | 0.2782 | 0.1798 | Summarizer retrospects full history |
| ④ | ✓ | ✗ | ✓ | ✗ | 0.2368 | 0.0977 | 0.1584 | 0.1039 | 0.2983 | 0.2791 | 0.1957 | RETROSUM Base (synergistic base) |
| ⑤ | ✓ | ✓ | ✓ | ✗ | 0.2505 | 0.1387 | 0.1541 | 0.1354 | 0.2853 | 0.2982 | 0.2104 | Base + Actor experience memory |
| ⑥ | ✓ | ✗ | ✓ | ✓ | 0.2437 | 0.1151 | 0.1565 | 0.1082 | 0.3234 | 0.2919 | 0.2065 | Base + Summarizer experience memory |
| ⑦ | ✓ | ✓ | ✓ | ✓ | 0.2514 | 0.1092 | 0.1568 | 0.1289 | 0.3068 | 0.3171 | 0.2117 | RETROSUM Evolved (Full Model) |
Key Findings¶
- Unidirectional Compression Failure on Frontier Models: On highly capable reasoning models such as Grok-4.1-fast and GPT-5-mini, unidirectional ReSum strictly underperforms standard ReAct (0.2237 vs 0.2501 on Grok-4.1-fast). Because frontier models possess strong multi-step synthesis capacity, the irreversible information loss from lossy local compression outweighs its context-saving advantages. By contrast, RETROSUM's dual-view architecture empowers Grok-4.1-fast to reach 0.2880 (+28.7% over ReSum).
- Complementary Dynamics across Window Sizes: Sweeping the summarization interval \(w\) from 1 to 30 steps reveals that Actor-Only retrospection dominates under frequent summarization (\(w \le 5\)) by shielding the agent from fractured syntax. In contrast, Summarizer-Only retrospection dominates over extended horizons (\(w \ge 15\)) by reviving distant diagnostic cues. The joint RETROSUM architecture delivers consistent superiority across the entire window frequency spectrum.
- Systemic Deadlock Eradication and Efficiency: While baseline methods like ReasoningBank and ReAct frequently exhaust the 100-round ceiling due to repeated information-seeking loops, RETROSUM shifts the interaction distribution to a focused peak between 20 and 40 turns. This reduces total systematic errors by up to 92.3%, compresses average input tokens per sample from 2.06M (ReAct) to 0.42M (4.9× reduction), and decreases per-instance execution latency from 158.66s to 133.08s.
Highlights & Insights¶
- Dual-View Context Augmentation: Decoupling macroscopic cognitive guidance from raw execution traces solves the trade-off between semantic abstraction and execution syntax, preventing LLM agents from losing tabular schema context.
- Retrospective Re-evaluation of Latent Evidence: Conditioned on both distant trajectories and current observations, the summarizer can reinterpret previously trivial symptoms into actionable clinical syndromes, mirroring the diagnostic reasoning of expert clinicians.
- Dual-Experience Memory Partitioning: Segregating procedural navigation rules from diagnostic salience heuristics enables plug-and-play adaptation across distinct EHR table formats without parameter retraining.
Limitations & Future Work¶
- Single-Center Cohort Constraints: Although spanning both MIMIC-IV and MIMIC-III, all data originates from Beth Israel Deaconess Medical Center, leaving hospital-specific administrative workflows, coding practices, and geographical demographic biases unexamined.
- Multimodal Gap: The framework operates on structured relational records and text notes; raw radiological pixel data (e.g., volumetric CT/MRI) and high-frequency physiological waveforms remain dependent on intermediate textual impressions.
- Automation Bias and Safe Deployment: Over-reliance on highly fluent agent-synthesized summaries risks clinician oversight of rare contraindications; integrating human-in-the-loop audit protocols and calibrated confidence estimation is critical for bedside translation.
Related Work & Insights¶
- vs EHRAgent / EHR-MCP: Prior EHR agents primarily benchmark factual information lookup (e.g., retrieving previous drug dosages); AGENTEHR introduces proactive clinical deduction tasks (diagnosis, therapy, and lab planning) over uncurated, multi-table databases.
- vs ReSum / MEM1: Conventional long-horizon memory methods perform irreversible forward compression; RETROSUM demonstrates that clinical causality requires bidirectional retrospection and maintains raw history to preserve operational continuity.
- vs ReasoningBank / ReflecTool: Unconstrained evolving memory frameworks frequently stagnate or terminate prematurely on EHR tasks; RETROSUM's retrospective foundation provides a stable context that amplifies the efficacy of retrieved procedural experiences.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Pioneers retrospective evidence re-evaluation to eliminate unidirectional context loss in multi-table clinical decision-making]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Thorough benchmarking across 5 LLMs, 6 hospital tasks, 3 distribution-shift splits, and deep ablations across intervals, errors, and compute latency]
- Writing Quality: ⭐⭐⭐⭐⭐ [Rigorous formal task specification, clean conceptual abstractions, and comprehensive error taxonomy visualizations]
- Value: ⭐⭐⭐⭐⭐ [Establishes a foundational standard and an effective open-source architecture for long-horizon autonomous clinical agents]