Understanding the Limits of Agentic ICD Coding¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Clinical Coding, ICD-10-CM, Agentic Systems, Multi-Step Reasoning, Tool-Augmented LLMs
TL;DR¶
This paper systematically investigates the performance boundaries of neural classifiers, workflow pipelines, and free-form agentic architectures on ICD-10-CM clinical coding, identifying two orthogonal failure modes—frequency-driven rarity bias and multi-step guideline-following collapse—and shows that tool-layer discovery grounding coupled with dynamic index orchestration recovers up to 0.34 micro-F1 on complex multi-step injury cases.
Background & Motivation¶
Accurate assignment of International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM) codes is critical for healthcare reimbursement, epidemiological tracking, and clinical research. The task requires selecting applicable diagnostic and procedural labels from a taxonomy of over 70,000 alphanumeric codes based on unstructured clinical discharge summaries. In authentic hospital settings, certified professional coders must navigate the Alphabetical Index, verify intermediate candidates against tabular inclusion and exclusion criteria, and enforce mandatory sequencing conventions established in official guidelines. Historically, automated ICD coding has been cast as extreme multi-label classification (XMC), exemplified by label-attention architectures such as CAML and the modern state-of-the-art PLM-ICD. However, because real-world clinical records feature a severely long-tailed label distribution, standard evaluation benchmarks rely heavily on aggregate metrics dominated by high-frequency codes. This practice conceals a devastating performance drop on tail codes—neural models exhibit an acute 0.43 micro-F1 chasm between rare and frequent diagnostic codes.
To bypass the distribution bias inherent in supervised neural encoders, recent investigations have embraced large language models (LLMs) through workflow-driven coding systems like CodeSeeker. These architectures orchestrate specialized LLM calls into a deterministic, linear pipeline (typically spanning Analyze, Locate, Assign, and Verify stages) to emulate human workflow. Yet, this rigid structural paradigm presupposes that medical coding can be resolved through an unvarying feedforward trajectory. In clinical reality, assigning complex interdependent conditions—such as Chapter 19 (Injuries and Poisoning, S/T codes) and Chapter 20 (External Causes of Morbidity, V/W/X/Y codes)—demands dynamic coordination where primary trauma codes must be coupled with consistent external mechanisms, mandatory place-of-occurrence markers (Y92 codes), and synchronized 7th-character episode extensions. Because static workflows cannot dynamically loop back to re-consult indexes or update intermediate hypotheses, they suffer an catastrophic collapse on multi-step coding, plummeting to near-zero micro-F1 (0.010–0.027) and falling far behind even unguided neural classifiers.
This stark dichotomy raises a fundamental inquiry: are automated coding failures on rare and multi-step cases a structural limitation of model capacities, or do they stem from misaligned interaction architectures that prevent flexible deliberation? The core idea of this paper is to decouple label rarity from multi-step navigation complexity into orthogonal diagnostic axes, constructing a controlled, free-form agentic framework with tool-layer discovery grounding that empowers LLMs to dynamically orchestrate official ICD-10-CM reference materials, thereby localizing procedural bottlenecks and recovering substantial performance on multi-step clinical guidelines.
Method¶
Overall Architecture¶
The framework formalizes automated clinical coding as an autonomous, multi-turn tool-use interaction environment centered around official ICD-10-CM reference materials. Taking an unstructured discharge summary as input, an LLM agent navigates a suite of nine specialized tools spanning four operational categories: navigation, tabular verification, guideline reference, and final submission. Unlike static pipelines where each sub-stage operates under a rigid, predetermined schedule, this architecture grants the agent complete autonomy at runtime to decide when and how to drill down hierarchical categories, consult specialized tables, or cross-reference clinical conventions.
A foundational architectural safeguard is tool-layer discovery grounding: the environment tracks all candidate codes encountered across search and browse tool outputs during the active session. If an agent attempts to submit a code that was never explicitly returned by upstream exploration, the submission tool raises an immediate rejection. This hard physical gate eliminates alphanumeric hallucinations while preserving flexible decision-making. Guideline-mandated sequencing rules (such as mandatory external cause pairing and 7th-character consistency) are communicated via system prompts rather than hard-coded into tool wrappers, rigorously testing the agent's contextual reasoning and protocol compliance.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
A["Unstructured Clinical Note<br/>(MIMIC-IV Discharge Summary)"] --> B["Stratified & Multi-Step Diagnostic Benchmark<br/>Rarity bins (Rare/Uncommon/Common) + Ch. 19/20 subset"]
B --> C["Tool-Layer Discovery Grounding Mechanism<br/>Session discovery ledger + Hard submission gating"]
C --> D["Dynamic Index Orchestration & Companion Guidance<br/>Alphabetical/Drug/Neoplasm/Cause tables + Prompt sequencing"]
D --> E["Neural Priming & Coding Funnel Tracing<br/>PLM-ICD tool toggle + Discovery-Verif-Submit stage audit"]
E --> F["Structured Validated ICD-10-CM Predictions<br/>(Primary diagnosis + Cause + Place + 7th character)"]
Key Designs¶
1. Stratified & Multi-Step Diagnostic Benchmark: Decoupling aggregate frequency bias from multi-step graph traversal failure
Standard clinical coding evaluations sample discharge summaries uniformly, inadvertently masking severe deficiencies on low-frequency and complex procedural conditions. This work curates two targeted evaluation cohorts from MIMIC-IV test sets: first, a rarity-stratified suite of 150 summaries sampled equally (50 notes each) across rare (0–100 training occurrences), uncommon (101–1,000), and common (>1,000) frequency tiers; second, an objective multi-step navigation subset comprising 153 summaries containing 480 code instances from Chapter 19 (S/T injury codes) and Chapter 20 (V/W/X/Y external cause codes). Among these, 117 notes require co-assigning both chapters, 123 demand a Y92 place-of-occurrence code, and 142 necessitate a 7th-character extension. This benchmark provides the first experimental proof that degradation on rare codes and failure on multi-step guideline navigation represent orthogonal structural vulnerabilities.
2. Tool-Layer Discovery Grounding Mechanism: Enforcing provenance constraints to eliminate alphanumeric hallucination
LLMs tasked with generating alphanumeric taxonomic codes are notoriously vulnerable to plausible-sounding hallucinations or unverified code extrapolations. Rather than relying on soft conversational instructions, the framework implements a discovery ledger directly within the tool sandbox. An agent interacts with navigation tools (search_alphabetical_index, search_drug_table, search_neoplasm_table, search_external_cause_index, and browse_code_children), which dynamically populate an in-memory session discovery set. When submit_codes is invoked, the environment inspects candidate codes and strictly rejects any entry not previously observed in navigation returns. Comprehensive analysis reveals that 74%–95% of subsequent false positives are codes that were explicitly verified via verify_code_in_tabular before submission, proving that the discovery gate successfully blocks hallucinated code generation and confines residual error strictly to clinical judgment boundaries.
3. Dynamic Index Orchestration & Companion Guidance: Unshackling fixed workflows to enable adaptive multi-table traversal
Static pipelines falter on external cause coding because their linear feedforward design cannot accommodate opportunistic branching when an unexpected secondary diagnosis is discovered. The free-form agentic setup allows arbitrary call sequences across the nine tools, paired with prompt-level companion-code sequencing directives. Whenever a traumatic injury (T or S code) is identified, the model is prompted to initiate companion searches for external causes, assign default place-of-occurrence codes (Y92.9 when unstated), and synchronize 7th-character extension indicators. This design empowers Claude Opus 4.6 to achieve a micro-F1 of 0.276–0.363 on multi-step codes, overcoming the near-total collapse (0.010–0.027) experienced by the fixed-workflow CodeSeeker architecture.
4. Neural Priming & Coding Funnel Tracing: Decomposing selection bottlenecks and uncovering early discovery signals
To assess whether discriminative classifiers can aid autonomous agents, the framework integrates PLM-ICD as an optional suggestion tool (suggest_candidate_codes). Tracing missed codes across the four-stage lifecycle (discovery \(\rightarrow\) verification \(\rightarrow\) submission \(\rightarrow\) specificity) reveals that the primary bottleneck lies in discovery: 66%–86% of missed multi-step codes are never returned by search tools, whereas verification accounts for only 0%–3% of failures (demonstrating that once a base code is retrieved, tabular lookup reliably secures correct 7th characters). Across 63K–178K verified candidates per model, less than 4% are ultimately submitted, with precision hovering between 36% and 56%—proving that tabular verification provides minimal negative filtering signal and that candidate selection is the true hurdle. Crucially, statistical testing confirms that true-positive codes are discovered significantly earlier in the session than false positives across all backbones (Mann–Whitney U \(p < 10^{-13}\), Cliff's \(\delta \in [0.20, 0.30]\)), providing a robust signal for post-hoc precision calibration.
Benchmark & Evaluation Protocol¶
- Data Slices: MIMIC-IV 3.1 discharge summaries. The all-codes cohort comprises 150 notes stratified into 50 rare, 50 uncommon, and 50 common cases. The multi-step cohort comprises 153 notes featuring 480 Chapter 19/20 code instances (193 rare, 142 uncommon, 145 common).
- Reference Backend: Built on official CMS 2020 XML databases, housing 45,907 valid tabular codes, complete Alphabetical and External Cause Indexes, Drug/Neoplasm tables, and official sectional guidelines.
- Compared Systems:
- Discriminative Baseline: PLM-ICD (RoBERTa encoder with label-wise attention tuned on MIMIC-IV, evaluated at optimal \(\tau=0.45\)).
- Workflow Systems: CodeSeeker with Claude Opus 4.6 and GPT-5.2 backbones.
- Free-Form Agents: Six frontier LLMs—Claude Opus 4.6, GPT-5.2, GLM-5 Turbo, Qwen 3.5 Plus, Kimi K2.5, and Mistral Large 3.
- Session Limits & Fault Tolerance: Hard budget caps of 200 interaction turns, 200 tool invocations, and 600 seconds per sample. Periodic status telemetry injected every 10 turns; context compression triggered at 80% window utilization (preserving system instructions, the first user message, and the 5 latest tool outputs while summarizing historical rounds). Stuck loops or timeouts automatically trigger submission of accumulated verified codes.
Key Experimental Results¶
Main Results¶
The table below reports micro-F1 across frequency tiers for the all-codes cohort (150 notes) and the multi-step navigation subset (153 notes), alongside 95% bootstrap confidence intervals based on 10,000 resamples:
| Model / Paradigm | All Codes: Rare (0-100) | All Codes: Unc. (101-1k) | All Codes: Common (>1k) | Multi-Step: Rare (193 inst.) | Multi-Step: Unc. (142 inst.) | Multi-Step: Common (145 inst.) |
|---|---|---|---|---|---|---|
| PLM-ICD (Discriminative SOTA) | .227 [.17, .28] | .471 [.44, .51] | .656 [.63, .68] | .256 [.20, .31] | .375 [.30, .45] | .435 [.36, .51] |
| CodeSeeker (Opus 4.6) | .560 [.50, .61] | .483 [.45, .52] | .607 [.58, .63] | .010 [.00, .03] | .027 [.00, .07] | .013 [.00, .04] |
| CodeSeeker (GPT-5.2) | .341 [.26, .42] | .353 [.31, .39] | .436 [.40, .47] | .021 [.00, .05] | .028 [.00, .07] | .000 [.00, .00] |
| Claude Opus 4.6 (Ours) | .354 [.30, .41] | .425 [.39, .46] | .577 [.55, .60] | .276 [.22, .33] | .363 [.28, .45] | .280 [.21, .35] |
| GPT-5.2 (Ours) | .329 [.27, .39] | .368 [.33, .40] | .538 [.51, .57] | .190 [.13, .25] | .250 [.17, .33] | .265 [.19, .34] |
| GLM-5 Turbo (Ours) | .304 [.26, .35] | .357 [.32, .39] | .554 [.53, .58] | .154 [.11, .20] | .296 [.21, .38] | .183 [.11, .25] |
| Qwen 3.5 Plus (Ours) | .281 [.23, .33] | .364 [.33, .40] | .546 [.52, .58] | .220 [.16, .29] | .291 [.20, .38] | .249 [.18, .32] |
| Kimi K2.5 (Ours) | .217 [.18, .26] | .314 [.28, .35] | .493 [.47, .52] | .160 [.10, .22] | .218 [.14, .30] | .178 [.12, .24] |
| Mistral Large 3 (Ours) | .209 [.17, .25] | .275 [.24, .31] | .485 [.45, .52] | .156 [.11, .20] | .209 [.14, .28] | .186 [.13, .25] |
Diagnostic Analysis & Ablations¶
The quantitative findings below summarize the coding funnel dynamics (Table 2), multi-step failure breakdown across stages (Table 3), and ablation of sequencing prompts and tool access (Tables 4 and 5):
Table 2: Agentic Coding Funnel Metrics
| Evaluated Backbone | Total Discovered Codes | Submission Rate (Sub%) | Precision of Submitted (Prec%) | Tool Calls per True Positive |
|---|---|---|---|---|
| Claude Opus 4.6 | 67K | 2.8% | 56% | 8.7 |
| GPT-5.2 | 63K | 3.1% | 50% | 8.4 |
| GLM-5 Turbo | 124K | 1.7% | 49% | 11.4 |
| Qwen 3.5 Plus | 117K | 1.7% | 49% | 10.9 |
| Mistral Large 3 | 72K | 3.6% | 36% | 12.0 |
| Kimi K2.5 | 178K | 1.6% | 36% | 13.5 |
Table 3: Per-Stage Failure Decomposition for Missed Multi-Step Codes
| ICD-10-CM Chapter / Category | Missed in Discovery (Disc.) | Missed in Verification (Verif.) | Missed in Submission/Spec. (Sub./Spec.) |
|---|---|---|---|
| Ch. 19 S/T Codes (Traumatic injury) | 66% | 9% | 3% |
| Ch. 20 V/W/X Codes (External mechanism) | 86% | 0% | 0% |
| Ch. 20 Y92 Codes (Place of occurrence) | 83% | 6% | 0.3% |
| Ch. 20 Y-Other Codes (Supplemental activity) | 79% | 14% | 0% |
Table 4: Prompt-Level Sequencing Ablation (Multi-Step Micro-F1)
| Configuration | Rare Bin | Uncommon Bin | Common Bin |
|---|---|---|---|
| Opus 4.6 (Full Prompt) | .267 [.10, .46] | .357 [.21, .50] | .318 [.19, .44] |
| Opus 4.6 (Sequencing Removed) | .241 [.08, .44] | .385 [.24, .53] | .316 [.18, .44] |
| GPT-5.2 (Full Prompt) | .151 [.00, .30] | .243 [.11, .37] | .322 [.19, .44] |
| GPT-5.2 (Sequencing Removed) | .143 [.00, .31] | .274 [.13, .41] | .091 [.00, .20] |
Table 5: Tool-Access Ablation (Full Tools vs Parametric Memory Only)
| Model Backbone | All Codes: Full Tools | All Codes: No Tools | Multi-Step: Full Tools | Multi-Step: No Tools |
|---|---|---|---|---|
| Claude Opus 4.6 | .354 / .425 / .577 | .182 / .285 / .498 | .276 / .363 / .280 | .142 / .249 / .050 |
| Qwen 3.5 Plus | .281 / .364 / .546 | .153 / .314 / .566 | .220 / .291 / .249 | .110 / .130 / .038 |
| GLM-5 Turbo | .304 / .357 / .554 | .134 / .288 / .513 | .154 / .296 / .183 | .069 / .144 / .050 |
Key Findings¶
- No Single System Dominates All Conditions: The discriminative classifier PLM-ICD excels on high-frequency codes (.656 on all-codes common, .435 on multi-step common) but degrades drastically on rare categories (.227). CodeSeeker performs strongly on generic rare codes (.560 with Opus 4.6) but collapses completely on multi-step external causes (.010–.027). The free-form agent achieves the most balanced recovery across complex multi-step rules (.276–.363 on rare/uncommon).
- Discovery Rather Than Verification Dominates Failure: Across missed multi-step instances, 66%–86% are never returned by search tools. Once base categories are retrieved, tabular verification reliably resolves 7th-character extensions (only 0%–3% error). Furthermore, candidate verification filters out minimal noise—the bottleneck is post-verification clinical selection.
- Early Discovery Correlates Heavily with Ground Truth: Correct positive codes (TP) are discovered significantly earlier in the normalized session trajectory than false positives (FP) across every evaluated LLM (\(p < 10^{-13}\), Cliff's \(\delta \in [0.20, 0.30]\)), providing an empirical basis for early stopping or confidence-weighted rank truncation.
- Neural Tool Priming Acts as a Double-Edged Sword: Supplying PLM-ICD candidates via
suggest_candidate_codeshelps under-exploring backbones like GPT-5.2 on generic codes, but actively impairs strong autonomous navigators like Claude Opus 4.6 (reducing autonomous rare-code discovery by 31% and diverting tool budgets into candidate verification), while harming multi-step accuracy for both models.
Highlights & Insights¶
- Orthogonal Failure Decomposition: Conclusively establishes that frequency-driven long-tail bias and procedural guideline-following failure are distinct, orthogonal error modes in clinical AI, invalidating aggregate metric claims in prior XMC literature.
- Physical Tool-Layer Discovery Gating: By enforcing code provenance at the API boundary (rejecting unindexed submissions), the architecture guarantees zero alphanumeric hallucination without needing complex self-correction prompts.
- The Pitfalls of Naive Neural Priming in Agents: Demonstrates that injecting discriminative model predictions into autonomous LLM agents can induce search myopathy, starving exploratory index traversal and undermining multi-step performance.
Limitations & Future Work¶
- Single-Center Cohort & Label Noise: Experiments rely solely on MIMIC-IV discharge summaries from Beth Israel Deaconess Medical Center; hospital-specific coding conventions and documentation noise (e.g., billing codes absent from clinical text) introduce uncertainty into absolute F1 figures.
- Focused Diagnostic Scope: The multi-step benchmark concentrates exclusively on Chapters 19 and 20; other complex guideline-heavy areas (such as obstetrics, neoplasm morphology, or congenital malformations) remain to be explored.
- Coarse Tool-Level Ablation: The tool-environment ablation tests an all-or-nothing configuration rather than isolating individual tools (such as the External Cause Index versus child browsing), leaving fine-grained attribution for future work.
Related Work & Insights¶
- vs PLM-ICD (Huang et al., 2022): PLM-ICD provides high-throughput, accurate scoring for high-frequency labels via label-wise attention, but fails to generalize to tail distributions and cannot enforce procedural constraints; the agentic framework trades throughput for dynamic reasoning and rule compliance on complex cases.
- vs CodeSeeker / CLH (Motzfeldt et al., 2025): CodeSeeker introduces multi-agent pipelines for clinical coding but binds LLMs to fixed, sequential roles; this paper demonstrates that unconstrained, runtime-orchestrated tool use is essential for navigating intricate, non-linear coding guidelines.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Decouples rarity from multi-step reasoning failure and establishes discovery-gated agentic coding.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous stratification, multi-model evaluation, bootstrap confidence intervals, and full-funnel tracing.
- Writing Quality: ⭐⭐⭐⭐⭐ Lucid, precise analytical prose backed by rigorous statistical testing.
- Value: ⭐⭐⭐⭐⭐ Sets a new benchmark methodology for medical coding agents and clarifies how to combine tools, LLMs, and neural models.