Skip to content

EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Medical Agents, Clinical Database Reasoning, Interactive Evaluation, Electronic Health Records, Trajectory SFT

TL;DR

This paper introduces EHR-Complex, the first large-scale interactive clinical database reasoning benchmark constructed on the complete MIMIC-IV substrate (365K patients, 31 tables, 500M+ records), featuring 52K executable tasks across 6 clinical intents and dual scopes, with queries averaging 31.93 SQL structural components, systematically exposing that frontier LLMs struggle with cohort-level temporal alignment and medical-code grounding (achieving at most 62.3% exact-match accuracy).

Background & Motivation

Integrating large language models (LLMs) and autonomous agents into electronic health record (EHR) systems represents a transformative paradigm for democratizing clinical database access, empowering clinicians and healthcare administrators to retrieve and interrogate patient data through natural language rather than esoteric SQL or database expertise. Modern hospital environments generate massive, heterogeneous, longitudinal, and semantically dense clinical data. Dependable clinical agents could substantially mitigate technical barriers for retrospective outcome analysis, longitudinal cohort building, and multi-visit patient tracking. However, realistic EHR database interrogation is inherently an interactive, closed-loop problem-solving process that demands inspecting intermediate feedback, resolving schema ambiguities, locating medical identifiers, and iteratively refining query formulations.

Existing clinical text-to-SQL and medical agent benchmarks fail to reflect the complexity of actual hospital analysis. Foundational clinical datasets like MIMICSQL, emrKBQA, EHRSQL, and EHR-SeqSQL predominantly adhere to static, single-turn text-to-SQL evaluation paradigms that completely lack execution feedback. Furthermore, to retain tractability, legacy benchmarks rely on heavily sanitized and drastically filtered database substrates—often restricted to fewer than 1,000 patients and 5 to 17 simplified tables—stripping away real-world scale, noise, and compositional joins. While recent agentic frameworks like EHRAgent and MedAgentBench provide sandboxed execution, they largely adapt tasks from legacy static datasets or operate on abstract high-level APIs rather than full-database relational schema. Consequently, current models achieve near-saturated scores on legacy benchmarks yet fail dramatically when confronted with real-world longitudinal multi-table joins.

To bridge this critical divide between synthetic simplicity and clinical reality, the paper models natural language clinical information needs as grounded program synthesis and sandboxed environment interaction. Core idea: leveraging the full MIMIC-IV v3.1 database (365K patients, 31 tables, 500M+ records), the authors synthesize patient event graphs and clinical evidence paths into ~52K verified tasks possessing authentic real-world complexity (averaging 31.93 SQL components per query), providing a rigorous interactive testbed that thoroughly evaluates agent robustness across individual trajectories and cross-patient cohort aggregations.

Method

Overall Architecture

EHR-Complex frames clinical database reasoning as an interactive decision process: given a natural-language clinical query \(x \in \mathcal{X}\), a relational database \(\mathcal{D}\), and schema \(\Sigma\), an agent outputs a multi-turn trajectory \(\tau \in \mathcal{T}\) by executing SQL queries or Python code in an isolated sandbox, receiving dynamic execution feedback, and returning a normalized final answer \(y \in \mathcal{Y}\) matching verified reference \(y^*\). The task creation pipeline enforces two fundamental properties: groundedness (every clinical question is rooted in an empirically verified chain of clinical events in the source database) and compositionality (tasks are generated by composing reusable temporal and logical operators rather than surface-level template enumeration).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Full MIMIC-IV Substrate<br/>365K patients / 31 tables / 500M+ records"] --> B["Patient Event Graph Construction<br/>concept-timestamp-attribute normalization & clinical relations"]
    B --> C["Clinical Evidence Path Sampling<br/>ordered chains satisfying temporal & clinical consistency"]
    C --> D["Evidence-to-SQL Template Compilation<br/>LLM compilation + execution non-empty check & exact answer anchoring"]
    D --> E["Template Parameter Resampling & Linguistic Rewriting<br/>796 unique templates expanded to 52K tasks (Pat. & Pop.)"]
    E --> F["Interactive Sandboxed Evaluation & Trajectory SFT<br/>multi-turn execution feedback + expert clinical audit"]

Key Designs

1. Patient Event Graph Construction: Unified Temporal Graph for Heterogeneous Clinical Records

Raw EHR entries are fragmented across distinct functional tables (e.g., admissions, ICU transfers, laboratory measurements, diagnosis codes, prescriptions, and charted observations) with disparate schemas. To establish a coherent substrate for compositional reasoning over patient timelines, EHR-Complex constructs a heterogeneous temporal graph \(G_p = (V_p, E_p, \phi_V, \phi_E)\) for each patient \(p\). Heterogeneous clinical events are normalized into unified nodes \(v = (c, t, a)\), where \(c\) denotes the standardized clinical concept (e.g., ICD code or lab item ID), \(t\) denotes the clinical event timestamp, and \(a\) captures numerical measurements, abnormal flags, or dosage attributes. Typed relationship edges \(e = (v_i, r, v_j)\) with \(r \in \mathcal{R}\) encode clinical and temporal dependencies such as precedes, indicates, or treated_by, providing an expressive, interconnected representation of longitudinal patient trajectories.

2. Clinical Evidence Path Sampling: Extracting Temporally and Clinically Consistent Chains

Rather than filling arbitrary schema slots that risk generating clinically nonsensical or permanently empty queries, EHR-Complex systematically samples concrete clinical evidence paths from the event graph. An evidence path is formally defined as an ordered sequence \(P = (v_1, v_2, \ldots, v_k)\) satisfying monotonic chronological order \(t(v_1) \le t(v_2) \le \cdots \le t(v_k)\) alongside clinical plausibility constraints. For example, a sampled path might trace an initial qualifying admission for acute respiratory failure, followed by an emergency ICU transfer, and terminating at a specific arterial blood gas measurement. This approach guarantees that every generated reasoning task is grounded in an actual sequence of clinical encounters, ensuring verifiable existence and clinical realism.

3. Evidence-to-SQL Template Compilation: Mapping, Execution Verification, and Deterministic Anchoring

Sampled clinical evidence paths are compiled into executable SQL templates via strong foundation models through a structured mapping \(g: P \mapsto s\), where \(s \in \mathcal{S}\) represents a SQL program operationalizing the intended reasoning steps. For patient-level queries, the program tracks temporal anchors and measurements within an individual trajectory; for population-level queries, the program aligns the identical evidence-path topology across the entire patient cohort and performs aggregation (e.g., mean intervals, proportions, or counts). Every synthesized template is rigorously tested against the full database using \(\text{Exec}(s, \mathcal{D})\). Only templates that execute cleanly without syntax errors, produce non-empty result sets, and permit deterministic answer extraction under standardized tie-breaking rules are retained, yielding 796 structurally distinct, execution-validated SQL templates.

4. Parameter Resampling and Natural Language Diversity Expansion: Scaling to 52K Benchmark Tasks

To achieve both structural depth and broad linguistic variance, the curation pipeline scales the 796 templates along two independent dimensions. First, it resamples database parameters—substituting eligible patient sets, time delta windows, ICD diagnosis codes, laboratory test item IDs, and medication names—while holding the relational reasoning topology invariant. Second, for each executable instance, multiple independent natural language paraphrases are generated (averaging 6 distinct sentence structures and clinician phrasing styles). This yields 48,092 training instances and 3,915 test instances across 6 clinical intents. Crucially, task complexity is validated via AST parsing (sqlglot), showing that EHR-Complex queries average 31.93 SQL structural components (nesting, aggregation, joins, complex conditions, grouping, and window functions), dwarfing legacy benchmarks (3 to 17 components).

Loss & Training

Beyond benchmarking, the authors investigate whether successful multi-turn interactive trajectories serve as effective procedural supervision for training open-weight clinical agents. They collect 6,000 successful trajectories generated by Kimi-K2.5 on the training partition (test-disjoint, averaging 20.9 messages per trajectory including tool calls, execution error feedback, code revisions, and final outputs). Using full-parameter supervised fine-tuning (SFT) for 1 epoch, open models (Qwen3-14B, Qwen3-32B, and Qwen3-8B) are trained with standard autoregressive causal language modeling:

\[\mathcal{L}_{\text{SFT}}(\theta) = -\sum_{i \in \text{Agent Turns}} \log P_{\theta}(u_i \mid u_{<i})\]

Training utilizes bfloat16 precision, a maximum context length of 32,768 tokens, learning rate \(1 \times 10^{-5}\) with a 5% warmup, and an effective batch size of 128 distributed across 64 H800 GPUs via DeepSpeed ZeRO-3.

Key Experimental Results

Main Results

Evaluation across 12 frontier models on the full EHR-Complex test set (3,915 tasks, temperature 0.0, maximum 50 interaction turns) demonstrates that real-world clinical database reasoning remains formidable. The table below presents exact-match accuracy across patient-level (Pat.) and population-level (Pop.) scopes for all 6 clinical intents:

Model Category Model Name Demographics (Pat. / Pop.) Vitals (Pat. / Pop.) Medications (Pat. / Pop.) Cost (Pat. / Pop.) Labs (Pat. / Pop.) Diagnoses (Pat. / Pop.) Macro Average (Pat. / Pop. / Overall)
Proprietary API GPT-4o 0.52 / 0.10 0.45 / 0.25 0.33 / 0.07 0.53 / 0.33 0.45 / 0.19 0.38 / 0.08 0.44 / 0.17 / 0.31
Proprietary API GPT-4.1 mini 0.75 / 0.27 0.60 / 0.22 0.67 / 0.21 0.88 / 0.60 0.65 / 0.30 0.59 / 0.17 0.69 / 0.29 / 0.49
Proprietary API GPT-4.1 0.76 / 0.21 0.55 / 0.16 0.58 / 0.17 0.89 / 0.50 0.72 / 0.26 0.63 / 0.18 0.69 / 0.25 / 0.47
Proprietary API Gemini 2.5 Pro 0.44 / 0.19 0.17 / 0.20 0.43 / 0.15 0.59 / 0.38 0.38 / 0.27 0.34 / 0.15 0.39 / 0.22 / 0.31
Open-Weight ≥100B Qwen3-235B 0.75 / 0.27 0.65 / 0.25 0.72 / 0.25 0.91 / 0.55 0.80 / 0.36 0.70 / 0.20 0.76 / 0.31 / 0.53
Open-Weight ≥100B Qwen3.5-397B 0.88 / 0.39 0.72 / 0.30 0.80 / 0.38 0.91 / 0.63 0.91 / 0.45 0.78 / 0.32 0.83 / 0.41 / 0.62
Open-Weight ≥100B DeepSeek-V3.1 0.81 / 0.33 0.70 / 0.27 0.65 / 0.24 0.89 / 0.67 0.82 / 0.39 0.72 / 0.21 0.77 / 0.35 / 0.56
Open-Weight ≥100B DeepSeek-V3.2-Exp 0.83 / 0.36 0.75 / 0.27 0.70 / 0.30 0.88 / 0.65 0.89 / 0.44 0.77 / 0.26 0.80 / 0.38 / 0.59
Open-Weight ≥100B Kimi-K2.5 0.89 / 0.34 0.80 / 0.27 0.73 / 0.35 0.92 / 0.72 0.89 / 0.42 0.84 / 0.29 0.85 / 0.40 / 0.62
Open-Weight <100B Qwen3-4B 0.24 / 0.07 0.28 / 0.11 0.18 / 0.01 0.48 / 0.12 0.17 / 0.06 0.16 / 0.03 0.25 / 0.07 / 0.16
Open-Weight <100B Qwen3-14B (Base) 0.44 / 0.14 0.20 / 0.18 0.43 / 0.15 0.63 / 0.33 0.40 / 0.19 0.39 / 0.10 0.41 / 0.18 / 0.30
Trajectory SFT Qwen3-14B-SFT 0.69 / 0.22 0.42 / 0.14 0.60 / 0.19 0.84 / 0.51 0.74 / 0.22 0.63 / 0.18 0.65 / 0.24 / 0.45 (+0.15)
Open-Weight <100B Qwen3-32B (Base) 0.51 / 0.20 0.30 / 0.24 0.47 / 0.20 0.73 / 0.35 0.50 / 0.21 0.42 / 0.15 0.49 / 0.23 / 0.36
Trajectory SFT Qwen3-32B-SFT 0.83 / 0.36 0.55 / 0.29 0.72 / 0.30 0.86 / 0.55 0.80 / 0.34 0.69 / 0.26 0.74 / 0.35 / 0.55 (+0.19)

Ablation Study

To evaluate whether trajectory fine-tuning genuinely instills generalizable procedural reasoning rather than superficial template memorization, the authors fine-tune Qwen3-8B under controlled cross-template and cross-intent regimes:

Evaluation Setting Model Configuration Scope Demographics Vitals Medications Cost Labs Diagnoses All Avg. Held-out Avg.
(a) Cross-Template (204 unseen templates) Qwen3-8B (Base) Patient 17.6% 5.9% 11.8% 35.3% 0.0% 17.6% 14.7% —
(a) Cross-Template (204 unseen templates) Qwen3-8B (Base) Population 0.0% 5.9% 0.0% 11.8% 17.6% 0.0% 5.9% —
(a) Cross-Template (204 unseen templates) Qwen3-8B-SFT Patient 64.7% 17.6% 41.2% 82.4% 64.7% 35.3% 51.0% —
(a) Cross-Template (204 unseen templates) Qwen3-8B-SFT Population 23.5% 11.8% 17.6% 35.3% 52.9% 0.0% 23.5% —
(b) Cross-Intent (Held out Vitals/Meds/Labs) Qwen3-8B (Base) Patient 29.4% 29.4% 5.9% 41.2% 17.6% 0.0% 20.6% 17.6%
(b) Cross-Intent (Held out Vitals/Meds/Labs) Qwen3-8B (Base) Population 5.9% 5.9% 0.0% 11.8% 11.8% 0.0% 5.9% 5.9%
(b) Cross-Intent (Held out Vitals/Meds/Labs) Qwen3-8B-SFT Patient 82.4% 11.8% 52.9% 94.1% 52.9% 58.8% 58.8% 39.2% (+21.6%)
(b) Cross-Intent (Held out Vitals/Meds/Labs) Qwen3-8B-SFT Population 29.4% 17.6% 35.3% 41.2% 29.4% 11.8% 27.5% 27.4% (+21.5%)

Key Findings

  • Dramatic Performance Drop on Population-Level Queries: Across all evaluated models, population-level tasks exhibit severe performance degradations compared to patient-level tasks. Kimi-K2.5 plummets from 0.85 to 0.40, Qwen3.5-397B drops from 0.83 to 0.41, and GPT-4.1 falls from 0.69 to 0.25. The degradation is particularly acute in Diagnoses, where cohort-wide temporal sequencing is critical (Kimi-K2.5 drops to 0.29; GPT-4.1 drops to 0.18). This reveals that minor per-patient alignment errors silently distort the aggregated cohort before the final computation, leading to erroneous yet superficially plausible SQL results.
  • Pass\(^k\) Exposes Stochastic Fragility: Evaluating consistency on a 20% stratified subset across \(k=1 \sim 4\) independent rollouts shows that while Pass@\(k\) increases with sampling budget (e.g., Qwen3.5-397B reaches 70.20% at \(k=4\)), Pass\(^k\) (requiring all \(k\) rollouts to be strictly correct) plunges from 62.3% to below 50% for nearly all models at \(k=4\). This instability indicates that current clinical agents are highly sensitive to rollout variability and lack the determinism mandated by clinical decision support.
  • Divergent Error Taxonomies Between Leading Models: A fine-grained analysis of 3,814 failed trajectories shows sharp mechanistic divergence: Kimi-K2.5 failures are dominated by Semantic Misunderstanding (37.9%) and Medical-Code Lookup Error (28.6%), often deviating before query synthesis; conversely, GPT-4.1 failures stem primarily from SQL Logic Error (41.1%) and Medical-Code Lookup Error (32.1%), indicating that it grasps broad clinical intent but commits subtle logical mistakes in JOIN keys, aggregation keys, or deduplication units.
  • Rapid Saturation Over Turn Budgets: An analysis of 712 tasks solved by all 5 representative models shows that single-turn generation (\(T=1\)) yields nearly zero success. Success climbs sharply up to \(T=10\) and largely plateaus by \(T=15\). Unproductive interaction loops (Stuck-in-Loop) average 44.7 turns for Kimi-K2.5, proving that simply expanding turn budgets without hypothesis revision fails to resolve deeper reasoning blockages.

Highlights & Insights

  • Hospital-Scale Grounded Substrate: By establishing the benchmark directly on the complete 365,000-patient MIMIC-IV database with 31 tables, the benchmark removes artificial laboratory simplifications, providing the first realistic testbed for large-scale clinical database exploration.
  • Graph-Induced Compositional Task Synthesis: Constructing heterogeneous patient event graphs to sample monotonic clinical evidence paths elegantly reconciles grounded data realism with high structural complexity (31.93 SQL components per query).
  • Trajectory SFT Closes the Frontier Model Gap: Training open-weight models on just 6,000 multi-turn interactive trajectories enables Qwen3-32B-SFT (0.55) to comfortably outperform GPT-4.1 (0.47) and rival Qwen3-235B (0.53), demonstrating strong out-of-distribution transfer to unseen clinical templates and novel medical intents.

Limitations & Future Work

  • Single-Center Schema and Coding Conventions: EHR-Complex is constructed on MIMIC-IV v3.1 originating from Beth Israel Deaconess Medical Center. Its schema conventions and billing code patterns reflect a single US tertiary medical institution; evaluating multi-center transferability or standardized common data models (such as OMOP-CDM) remains an important next step.
  • Absence of Multimodal and Free-Text Integration: Over 80% of critical clinical nuances reside in unstructured clinical notes, discharge summaries, and medical imaging. Currently, EHR-Complex evaluates structured relational reasoning; extending this interactive agent paradigm to hybrid structured-unstructured clinical reasoning is a vital direction.
  • Active Hypothesis Revision in Agent Loops: Models frequently fail to escape repetitive query loops when encountering unexpected database responses. Future work should develop agent architectures with explicit hierarchical reflection, hypothesis backtracking, and external biomedical ontology grounding.
  • vs. EHRSQL & EHR-SeqSQL: Legacy benchmarks evaluate static one-shot SQL strings over small filtered subsets (<1K patients, 17 tables) with simple query topologies (11 to 17 AST components). EHR-Complex enables dynamic multi-turn execution feedback over 365K patients and 31 tables, featuring 31.93 AST components per query.
  • vs. MedAgentBench & EHRAgent: MedAgentBench evaluates clinical agents via high-level synthetic API calls, abstracting away relational database logic; EHR-Complex focuses on native SQL/Python execution over real-world relational schemas, capturing authentic multi-table aggregation and cohort-level join failures.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Groundbreaking construction of a large-scale interactive clinical database benchmark directly on full MIMIC-IV via patient event graph synthesis.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across 12 frontier models, 3,800+ trajectory failure audits, multi-sample Pass\(^k\) analysis, and cross-template/cross-intent SFT validation.
  • Writing Quality: ⭐⭐⭐⭐⭐ Highly structured formalisms, precise AST-based complexity quantification, and exhaustive empirical documentation.
  • Value: ⭐⭐⭐⭐⭐ Crucial reality check for medical AI agents, demonstrating why static SQL metrics overestimate real-world clinical readiness and charting a clear path for trajectory-based training.