Skip to content

SARL-ICD: Structure-aware Contrastive Representation Learning with LLM-augmented Label Descriptions for Automatic ICD Coding

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/ZhuLab-Fudan/SARL-ICD
Area: Medical NLP
Keywords: automatic ICD coding, contrastive learning, shortcut learning, LLM-augmented descriptions, mixture of experts

TL;DR

SARL-ICD mitigates structural shortcut learning in semi-structured clinical discharge summaries and enriches medical code semantics through Jaccard-guided narrative-truncated MoCo pretraining, a lightweight LLM-augmented label-text interaction module, and an adaptive MoE fusion network, establishing new state-of-the-art performance across MIMIC-III-full and MIMIC-IV.

Background & Motivation

Automatic International Classification of Diseases (ICD) coding is a cornerstone of modern electronic health records and clinical billing systems, formalizing the task of assigning standardized diagnostic and procedural codes to highly complex, unstructured clinical discharge summaries. While recent advances in pretrained language models (PLMs)—most notably PLM-ICD and PLM-CA—have substantially surpassed classical CNN- and RNN-based architectures in extreme multi-label classification (XML), they encounter a severe, previously unaddressed vulnerability: clinical discharge summaries exhibit highly repetitive semi-structured formatting across different patients. Sections such as the hospital course, physical examinations, and discharge instructions adhere to virtually identical templates regardless of the underlying pathology. Consequently, standard contextual encoders rapidly overfit to these superficial structural regularities during pretraining and fine-tuning, mapping clinically distinct cases into deceptively close proximity in the latent space solely because they share standardized documentation templates.

Compounding this structural shortcut problem is the profound underutilization of semantic knowledge embedded in ICD code definitions. Standard ICD codes are paired with terse, highly condensed phrases that fail to articulate the underlying pathological mechanisms, anatomical locations, and surgical indications necessary to resolve clinical ambiguity. Prior attempts to integrate label descriptions into predictive pipelines have suffered from either rigid architectural coupling that prevents general deployment or superficial lexical matching that fails to overcome semantic sparsity. Furthermore, prevailing neural architectures exhibit pronounced inductive bias trade-offs: cross-attention architectures like PLM-CA excel at gathering procedural and temporal evidence distributed across extended notes but struggle on semantic-heavy chronic diagnoses, whereas label-wise attention architectures like PLM-ICD display the exact inverse specialization. Existing paradigms lack an adaptive, principled mechanism to reconcile and exploit these architectural complementarities.

Rather than pursuing unconstrained parameter scaling, this work addresses the foundational challenge of representation discriminability under structural regularities. Core idea: eliminate structural shortcuts by restricting Jaccard-guided MoCo pretraining strictly to narrative history sections, while simultaneously injecting rich LLM-augmented code semantics via a lightweight interaction module and dynamically harmonizing complementary architectures through adaptive MoE fusion.

Method

Overall Architecture

SARL-ICD operates as a modular, two-stage representation learning and classification framework comprising three core components: (1) JMoCo contrastive pretraining designed to sever structural shortcuts, (2) an architecture-agnostic label-text semantic interaction module driven by LLM-expanded descriptions, and (3) a lightweight Mixture-of-Experts (MoE) fusion network that dynamically reconciles heterogeneous prediction heads.

During the pretraining phase, the system builds an ICD hierarchical ancestry tree to compute soft positive sample affinities via augmented Jaccard similarity. An automated rule-based regex parser extracts solely the clinical Narrative History block, discarding repetitive hospital courses and templated discharge instructions to train a MoCo encoder on patient-specific illness trajectories. During downstream fine-tuning, the JMoCo-pretrained weights are transferred to both PLM-CA and PLM-ICD backbones processing full-length records, while frozen Sentence-BERT embeddings of LLM-generated code explanations interact with document hidden states via token-level cosine similarity. Finally, the raw output logits from both backbones are routed through a sparse MoE gating layer to produce instance- and code-adaptive ensemble predictions.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Raw Clinical Notes & ICD Code Sets"] --> B["JMoCo Structure-Aware Contrastive Pretraining<br/>Hierarchical Jaccard Pairs + Narrative Section Truncation"]
    B --> C["Dual-Branch Downstream Backbones<br/>Load JMoCo Weights & Process Full Clinical Notes"]
    C --> D["LLM-Augmented Label-Text Interaction Module<br/>Qwen Code Expansion + Dense Semantic Interaction Fusion"]
    D --> E["Lightweight MoE Adaptive Fusion Network<br/>Category-Specific Dynamic Gating Routing"]
    E --> F["Final Multi-Label ICD Prediction Distribution"]

Key Designs

1. JMoCo Structure-Aware Contrastive Pretraining: Shortcut Severance via Narrative Filtering and Hierarchical Jaccard Guidance

Conventional unsupervised or multi-label contrastive objectives trained directly on full discharge summaries gravitate towards superficial template alignments rather than patient-specific pathologies. JMoCo overcomes this failure mode through coordinated input purification and semantic pair construction. Literal code overlaps fail to reflect clinical affinity when closely related diagnoses possess distinct string identifiers (e.g., 251.1 and 251.2 both denote hypoglycemia without sharing digits). To address this semantic sparsity, the label set \(Y_i\) of document \(t_i\) is recursively expanded across the entire ICD hierarchy to include all parent chapters and ancestor categories: $\(\tilde{Y}_i = Y_i \cup \bigcup_{y \in Y_i} \text{Ancestors}(y)\)$ Pairwise positive relationships are established by evaluating the augmented Jaccard similarity \(s_{ij} = |\tilde{Y}_i \cap \tilde{Y}_j| / |\tilde{Y}_i \cup \tilde{Y}_j|\), designating pairs satisfying \(s_{ij} \ge \theta\) as positive set \(\mathcal{P}\). Crucially, an automated parser segments the text into Narrative History, Hospital Course, and Discharge Information. JMoCo discards the latter two repetitive blocks during pretraining, feeding only narrative sections (chief complaint, present illness, past medical history, admission medications) into query encoder \(f_q\) and momentum encoder \(f_k\). This deliberate distribution shift severs the structural shortcut channel during representation learning, ensuring downstream encoders align representations strictly with disease phenotypes.

2. LLM-Augmented Label-Text Interaction Module: Architecture-Agnostic Semantic Injection

Official ICD descriptions are typically terse noun phrases that provide insufficient contextual clues for subtle differential diagnosis. This module leverages generative language models to construct dense, informative label embeddings that enrich document representations without altering the base architecture. Using Alibaba Cloud DashScope API, Qwen-7B-Chat (qwen-turbo) is prompted with clinical guidelines to expand each ICD description into a 2-3 sentence explanation detailing surgical techniques, clinical indications, anatomical targets, and potential complications. Both raw and expanded descriptions are encoded using Sentence-BERT into fixed vectors \(e_i \in \mathbb{R}^d\) and frozen to preserve semantic stability. Given chunked document representations \(H \in \mathbb{R}^{L \times d}\) from the backbone encoder, the module computes a token-level cosine similarity matrix \(S \in \mathbb{R}^{N_c \times L}\): $\(S_{ij} = \frac{e_i \cdot h_j}{\|e_i\| \|h_j\|}\)$ Applying adaptive average pooling across the sequence dimension followed by a linear projection yields fixed-dimensional interaction features \(F \in \mathbb{R}^{N_c \times d_{\text{interact}}}\). Concatenating \(F\) with the base model's internal contextual representations prior to the final linear classification head injects rich semantic signals with negligible computational overhead, readily plugging into both PLM-CA and PLM-ICD.

3. Lightweight MoE Adaptive Fusion: Dynamic Exploitation of Heterogeneous Inductive Biases

Comprehensive architectural profiling demonstrates that PLM-CA and PLM-ICD possess highly polarized, complementary strengths across the ICD taxonomy. PLM-CA's cross-attention queries the full document globally, securing decisive advantages on conditions with distinct temporal and procedural markers, such as perinatal conditions (100% win rate), pregnancy complications (86.0% win rate), and congenital anomalies (80.3% win rate). Conversely, PLM-ICD's label-wise attention matches precomputed code descriptions locally, dominating chronic physiological diseases rich in semantic documentation, including circulatory diseases (68.1% win rate) and endocrine disorders (67.3% win rate). To maximize synergistic gain, a lightweight Mixture-of-Experts network receives the paired unnormalized logits \((\ell_i^P, \ell_i^B)\) from PLM-CA and PLM-ICD for each document and code: $\(\tilde{\ell}_i = \sum_{k=1}^K g_k(\ell_i^P, \ell_i^B) \cdot \text{Expert}_k(\ell_i^P, \ell_i^B)\)$ The gating network implements Top-\(k\) sparse routing over \(K\) MLP experts, dynamically weighting the two paradigms according to patient context and specific diagnostic category, consistently outperforming static linear blending.

Loss & Training

Optimization follows a decoupled, staged protocol. The pretraining objective minimizes a similarity-weighted InfoNCE loss across positive pairs \(\mathcal{P}\): $\(\ell_{ij} = \log\left(1 + \sum_{n \in \mathcal{N}} \exp\left(\frac{q_i^\top (k_n - k_j)}{\tau}\right)\right), \quad \mathcal{L}_{\text{pretrain}} = \frac{1}{|\mathcal{P}|}\sum_{(i,j)\in \mathcal{P}} s_{ij} \cdot \ell_{ij}\)$ Pretraining hyperparameters utilize temperature \(\tau=0.07\), momentum coefficient \(m=0.999\), FIFO queue capacity \(K=4096\), projection dimension 256, and RoBERTa-base-PM-M3-Voc4 backbone with learning rate \(2\times 10^{-5}\) for 3 epochs. Similarity threshold \(\theta\) is set to 0.86 for MIMIC-III-full, 0.97 for MIMIC-IV ICD-9, and 0.82 for MIMIC-IV ICD-10. Downstream fine-tuning restores full-length notes (4096 tokens) optimized under standard multi-label binary cross-entropy (BCE) loss at learning rate \(5\times 10^{-5}\) for 20 epochs. Finally, the backbone weights are frozen, and the MoE fusion module (expert hidden dimension 128, Top-\(k\) routing) is trained on the cached logit outputs.

Key Experimental Results

Main Results

Evaluated on MIMIC-IV ICD-10, MIMIC-III-full, and MIMIC-IV ICD-9 benchmarks, SARL-ICD resolves the historical trade-off between Macro-F1 and Micro-F1, establishing dominant performance across all evaluated metrics.

Dataset Model Macro-F1 Micro-F1 P@8 P@15 Macro-AUC Micro-AUC
MIMIC-IV ICD-10 CAML 0.210 0.584 0.668 0.522 0.910 0.982
LAAT 0.222 0.574 0.684 0.539 0.944 0.989
PLM-ICD 0.211 0.585 0.699 0.550 0.966 0.992
PLM-CA 0.261 0.596 0.709 0.561 0.962 0.992
SARL-ICD (Ours) 0.286 0.611 0.723 0.573 0.974 0.992
MIMIC-III-full CAML 0.087 0.534 0.709 0.560 0.895 0.986
LAAT 0.090 0.574 0.738 0.588 0.895 0.985
MSMN 0.103 0.584 0.751 0.598 0.948 0.992
PLM-ICD 0.104 0.598 0.771 0.613 0.926 0.989
PLM-CA 0.119 0.600 0.766 0.614 0.913 0.987
PLM-LLM 0.103 0.613 0.779 0.615 0.952 0.988
SARL-ICD (Ours) 0.123 0.618 0.786 0.631 0.954 0.990
MIMIC-IV ICD-9 PLM-CA 0.324 0.633 0.707 0.540 0.971 0.994
SARL-ICD (Ours) 0.356 0.644 0.719 0.550 0.974 0.995

Ablation Study

Ablation experiments conducted on MIMIC-IV ICD-10 examine structural components, while section truncation analyses on MIMIC-III-full validate the mechanism of JMoCo.

Evaluation Configuration Macro-F1 Micro-F1 P@8 P@15 Description
MIMIC-IV ICD-10 Component Ablation
PLM-CA (baseline) 0.261 0.596 0.709 0.561 Unmodified cross-attention baseline
+ Label-Text (unaugmented raw text) 0.265 0.596 0.707 0.558 Interaction with concise official ICD phrases
+ Label-Text (LLM-augmented descriptions) 0.270 0.597 0.709 0.561 Interaction with rich Qwen explanations
+ PT (JMoCo pretrained weights only) 0.272 0.596 0.709 0.561 Encoder initialized with JMoCo
+ Label-Text (augmented) + PT 0.277 0.599 0.713 0.563 Jointly enhanced PLM-CA branch
PLM-ICD (baseline) 0.211 0.585 0.699 0.550 Unmodified label-wise attention baseline
+ Label-Text (LLM-augmented descriptions) 0.263 0.596 0.715 0.564 Substantial +5.2% Macro gain on PLM-ICD
+ PT (JMoCo pretrained weights only) 0.260 0.596 0.714 0.563 Shortcut-free representations jump +4.9%
+ Label-Text (augmented) + PT 0.270 0.598 0.706 0.566 Jointly enhanced PLM-ICD branch
Linear Weighted Fusion (PLM-CA + PLM-ICD) 0.283 0.609 0.715 0.570 Uniform static probability weighting
SARL-ICD (MoE Adaptive Fusion Full) 0.286 0.611 0.723 0.573 Optimal dynamic routing performance
MIMIC-III-full JMoCo Section & Design Ablation (PLM-CA / PLM-ICD)
Narrative sections only (Ours JMoCo) 0.122 / 0.118 0.606 / 0.601 — — Best configuration: filters repetitive templates
Full discharge summary pretraining 0.120 / 0.112 0.603 / 0.600 — — Retaining templates causes noticeable Macro drop
Hospital-course only pretraining 0.117 / 0.103 0.603 / 0.596 — — Heavy template area drops PLM-ICD by 1.5%
Discharge-diagnosis only pretraining 0.120 / 0.110 0.602 / 0.598 — — Lacks rich patient-specific narrative contexts
w/o ICD Hierarchy Expansion 0.118 / 0.112 0.602 / 0.600 — — Loses semantic overlap among sibling codes
w/o Similarity Weighting 0.120 / 0.109 0.602 / 0.601 — — Unweighted InfoNCE weakens margin separation

Key Findings

  • Narrative truncation directly prevents contrastive shortcut learning: When pretraining on untruncated notes, correlation between embedding similarity and true ICD code similarity is remarkably poor (Pearson \(r=0.0115\), Spearman \(\rho=-0.0802\)), indicating the encoder primarily groups notes by hospital section boilerplate. Restricting pretraining to Narrative History raises Spearman \(\rho\) to \(+0.2209\). In zero-shot ICD code retrieval without any task fine-tuning (Table 8), narrative-only JMoCo boosts frequent code P@5 from 0.002 to 0.070, proving that representations capture actual disease phenotypes.
  • Enriched label semantics rescue rare and tail codes: On the complex MIMIC-IV ICD-10 benchmark, SARL-ICD boosts Macro-F1 to 28.6% (a relative gain of 35.5% over PLM-ICD's 21.1% and 9.6% over PLM-CA's 26.1%). Because rare codes lack extensive training examples, injecting explicit LLM-generated clinical indications and procedural nuances provides the necessary discriminative anchor.
  • MoE unlocks architectural synergy beyond individual limits: In the qualitative case study (Table 9), PLM-CA and PLM-ICD each correctly predict 12/15 ground truth codes, but on disjoint label sets (e.g., PLM-CA identifies oral candidiasis 112.0, while PLM-ICD identifies hearing loss 389.8). MoE adaptive routing correctly outputs 14/15 codes, uniquely recovering pancreatic neoplasm 157.9 which neither individual base model detected.

Highlights & Insights

  • Reframing document filtering as shortcut suppression: Text truncation in clinical NLP has historically been treated as a regrettable heuristic forced by context window constraints. This work proves that strategically discarding repetitive structural sections in self-supervised pretraining actively prevents representation collapse on administrative formatting.
  • Lightweight, decoupled cross-modal label injection: By using frozen Sentence-BERT embeddings of LLM-elaborated descriptions interacting via simple token-wise cosine pooling, SARL-ICD achieves substantial semantic enrichment without modifying transformer attention backbones or incurring prohibitive memory penalties.
  • Dynamic arbitration of inductive biases: Demonstrating that cross-attention and label-wise attention exhibit over 10% polarization across clinical categories provides empirical justification for MoE-based logit fusion in extreme multi-label classification.

Limitations & Future Work

  • Dependency on standardized ICD taxonomy: Constructing JMoCo positive pairs relies strictly on existing hierarchical tree relationships in ICD-9/10, making extension to ontology-free medical coding or cross-coding harmonization challenging.
  • Unverified automated LLM evaluation: While DeepSeek-Chat evaluation showed high faithfulness (2.98/3.0) and clinical accuracy (2.98/3.0) for the Qwen-generated descriptions, the expansions have not undergone rigorous blind review by certified medical coding professionals.
  • Dual-encoder inference footprint: Deploying full SARL-ICD necessitates running both PLM-CA and PLM-ICD transformer backbones during inference, elevating per-document latency to 56.1 ms (compared to 24.7 ms for PLM-ICD alone) and fine-tuning memory to 12.9 GB. Distilling the MoE ensemble into a unified backbone represents a promising direction for edge clinical deployment.
  • vs PLM-ICD & PLM-CA: PLM-ICD relies on label-wise attention over concise code representations, whereas PLM-CA uses learnable cross-attention queries across the full document. SARL-ICD provides both models with cleaner, shortcut-free initializations via JMoCo, augments them with LLM-expanded label features, and adaptively fuses their predictions to surpass both individual baselines.
  • vs MulSupCon & HSAP: Previous contrastive approaches either construct discrete positive sets per label (MulSupCon), which scales poorly to thousands of ICD codes, or depend on external UMLS knowledge graph alignments (HSAP). JMoCo formulates continuous affinity weights via hierarchical Jaccard expansion, maintaining extreme computational efficiency.
  • vs PLM-LLM: PLM-LLM relies on GPT-4 synonym generation aligned purely at the label representation level, resulting in severe degradation on tail codes (MIMIC-III Macro-F1 of only 10.3%). SARL-ICD integrates dense LLM descriptions directly into token-level document interaction, elevating Macro-F1 to 12.3%.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Formulates structural shortcut learning in clinical NLP and proposes narrative-truncated MoCo paired with MoE routing.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous validation across MIMIC-III and MIMIC-IV, backed by representation correlation analyses, section ablations, zero-shot retrieval, and latency benchmarking.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative connecting theoretical shortcut hypotheses to concrete empirical demonstrations and case studies.
  • Value: ⭐⭐⭐⭐⭐ Provides a robust, highly extensible blueprint for extreme multi-label classification in structured enterprise documents.