SARL-ICD: Structure-aware Contrastive Representation Learning with LLM-augmented Label Descriptions for Automatic ICD Coding¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/ZhuLab-Fudan/SARL-ICD
Area: Medical NLP
Keywords: Automatic ICD Coding, Structure-aware Contrastive Learning, Shortcut Learning, LLM Label Augmentation, Mixture-of-Experts Fusion
TL;DR¶
To tackle representation shortcut learning caused by templated semi-structured discharge summaries and the underutilization of rich label semantics, SARL-ICD introduces narrative-truncated Jaccard contrastive pretraining (JMoCo), lightweight LLM-augmented label-text interaction, and adaptive MoE fusion across complementary coding backbones, establishing new state-of-the-art results across MIMIC-III and MIMIC-IV.
Background & Motivation¶
Automatic International Classification of Diseases (ICD) coding is a foundational extreme multi-label classification challenge that maps free-form clinical discharge summaries into high-dimensional diagnostic and procedure billing spaces. Recent approaches built upon pretrained language models (PLMs) and label-specific attention mechanisms—most notably PLM-ICD and PLM-CA—have achieved remarkable empirical performance. Nevertheless, clinical discharge summaries possess highly repetitive, semi-structured organization: standardized sections including the "hospital course", "physical examination", and "discharge instructions" adhere to rigid linguistic boilerplates across vast patient cohorts. When conventional contrastive objectives or contextual encoders digest the entire document indiscriminately, models effortlessly overfit to these superficial structural regularities rather than learning discriminative, patient-specific pathophysiology. Consequently, two clinically distinct patient episodes frequently map too closely in the latent representation space merely because they share common documentation templates.
Concurrently, standardized ICD code systems inherently encode rich clinical hierarchies and semantic descriptions, yet existing frameworks severely underutilize this label-side knowledge. Although previous attempts have incorporated code descriptions via shallow Word2Vec embeddings, knowledge graphs, or architecture-specific prompt-tuning, they remain constrained by sparse official phrasing, lacking comprehensive clinical terminology, and often being rigidly coupled to specific model architectures without plug-and-play adaptability. Furthermore, an underlying architectural tension exists among dominant coding paradigms: cross-attention encoders like PLM-CA excel at extracting diffuse procedural and temporal evidence across lengthy clinical notes, whereas label-wise attention architectures like PLM-ICD better capture fine-grained semantic correspondence for chronic systemic diseases. Prior work has largely treated these architectures in isolation, leaving their complementary inductive biases unexploited.
This paper tackles these challenges by decoupling patient-specific clinical narratives from templated formatting, enriching label descriptions through large language models, and dynamically harmonizing heterogeneous attention inductive biases. Core idea: eliminate structural shortcut learning by truncating discharge summaries to narrative history sections under an ancestor-augmented Jaccard-guided contrastive pretraining scheme (JMoCo), while seamlessly injecting LLM-augmented dense label descriptions and dynamically arbitrating cross-attention versus label-attention predictions via a lightweight Mixture-of-Experts gating module.
Method¶
Overall Architecture¶
The SARL-ICD pipeline consists of three sequentially aligned phases: First, during offline structure-aware contrastive pretraining, the framework constructs positive document pairs based on augmented Jaccard similarity across ancestor-expanded ICD hierarchies, combined with an automated keyword-based document segmenter that discards templated hospital courses to train encoders exclusively on informative narrative histories via MoCo. Second, during downstream fine-tuning on full-length documents, an architecture-agnostic label-text interaction module computes local cosine similarities between token states and frozen Sentence-BERT embeddings of LLM-expanded (Qwen-7B-Chat) clinical descriptions, producing concatenated label-aware representations. Third, a lightweight Mixture-of-Experts (MoE) fusion module consumes the prediction logits produced by enhanced PLM-CA and PLM-ICD backbones to dynamically synthesize category-specific strengths into final multi-label probabilities.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
subgraph S1["Structure-Aware Contrastive Pretraining JMoCo"]
direction TB
A["Raw Clinical Discharge Notes"] --> B["1. Hierarchy-Augmented Jaccard Similarity & Narrative Truncation<br/>Extract Chief Complaint/HPI/PMH + Expand ICD Ancestor Trees"]
B --> C["2. Similarity-Weighted Momentum Contrast JMoCo<br/>Dynamic Negative Memory Queue + Jaccard Soft-Weighted InfoNCE"]
end
subgraph S2["Downstream Fine-Tuning & Adaptive Fusion"]
direction TB
D["Full Document Tokens H"] --> E["3. LLM-Augmented Lightweight Label-Text Interaction<br/>Qwen Clinical Expansions + Adaptive Pooled Cosine Concatenation"]
F["Target ICD Codes"] --> E
E --> G["Dual Fine-Tuned Backbones<br/>PLM-CA-L Cross-Attention + PLM-ICD-L Label-Wise Attention"]
G --> H1["4. Cross-Architecture Mixture-of-Experts Fusion<br/>Dynamic Category Inductive Bias Routing + Top-k Sparse Gating"]
end
C -.->|Initialize Backbone Weights| G
H1 --> I["Final Multi-Label ICD Predictions"]
Key Designs¶
1. Hierarchy-Augmented Jaccard Similarity & Narrative Truncation: Eliminating Surface Mismatch and Structural Shortcuts
Directly measuring clinical affinity between medical notes via naive Jaccard overlap of raw ICD code sets fails due to literal string mismatches between closely related conditions (for instance, code 251.1 for "Other specified hypoglycemia" and 251.2 for "Hypoglycemia, unspecified" share zero lexical overlap despite strong pathophysiological alignment). The authors overcome this by traversing the official ICD hierarchical tree to expand each note's ground-truth codes with all parent chapters and higher-level category ancestors \(\tilde{Y}_i = Y_i \cup \bigcup_{y \in Y_i} \text{Ancestors}(y)\). Document pairs exceeding a calibrated threshold on the expanded Jaccard coefficient \(s_{ij} = |\tilde{Y}_i \cap \tilde{Y}_j| / |\tilde{Y}_i \cup \tilde{Y}_j|\) are systematically extracted as clinically grounded positive pairs.
Equally crucial is the surgical suppression of document-level shortcut learning. Recognizing that hospital course narratives and discharge dispositions follow rigid, templated linguistic regularities across patients, an automated rule-based parser segments clinical documents into Narrative History, Hospital Course, and Discharge Information. Contrastive pretraining is strictly restricted to the Narrative History block (chief complaint, history of present illness, past medical history, family/social history, admission physical exam), entirely stripping out templated hospital course text. This deliberate pretraining-to-finetuning distribution shift forces the encoder to build patient-specific clinical representations without being contaminated by superficial structural cues, while allowing full-length documents to supply complete evidence during downstream fine-tuning.
2. Similarity-Weighted Momentum Contrast JMoCo: Structuring Latent Spaces with Continuous Pathological Gradients
Once positive pairs and truncated texts are extracted, standard binary contrastive formulations treat all positive pairs uniformly, neglecting variations in clinical affinity. JMoCo adopts a MoCo-style momentum contrastive paradigm featuring a dynamic FIFO negative sample queue, while embedding the augmented Jaccard score \(s_{ij}\) directly as a continuous soft sample weight in the InfoNCE objective.
By assigning higher loss contributions to document pairs with deeper pathological convergence, the momentum and query encoders jointly align clinical representations smoothly without requiring manual fine-grained supervision. Latent alignment evaluation demonstrates that this design drives the Spearman correlation between pairwise document embedding cosine similarity and true ICD label similarity from negative correlation up to \(+0.2209\), validating that the latent space captures clinical semantics rather than documentation syntax.
3. LLM-Augmented Lightweight Label-Text Interaction: Decoupled and Plug-and-Play Semantic Injection
Official ICD code descriptors are notoriously brief and jargon-laden (e.g., "00.96 Infusion of 4-Factor Prothrombin Complex Concentrate"), lacking the diagnostic reasoning, colloquial abbreviations, and anatomical nuances documented by treating physicians. To inject external medical knowledge, the authors utilize Qwen-7B-Chat to expand every code description into a 2-3 sentence clinical overview detailing technical execution, primary indications, and anatomical sites or complications, subsequently encoding them with Sentence-BERT into frozen dense label vectors \(e_i \in \mathbb{R}^d\).
To prevent architectural lock-in, the interaction module functions as an agnostic parallel branch: it computes token-level cosine similarity matrices \(S_{ij} = (e_i \cdot h_j) / (\|e_i\| \|h_j\|)\) across document representations \(H \in \mathbb{R}^{L \times d}\), applies sequence-wise adaptive average pooling, and projects the outcome into a fixed-dimensional interaction vector \(F \in \mathbb{R}^{N_c \times d_{\text{interact}}}\). Concatenating this feature with the base model's label-aware contextual state prior to the final classification head delivers rich semantic injection without altering backbone transformer topologies.
4. Cross-Architecture Mixture-of-Experts Fusion: Dynamically Exploiting Heterogeneous Inductive Biases
Granular root-category analysis reveals strong inductive bias divergence across leading architectures: cross-attention (PLM-CA) relies on global receptive fields and dominates procedural or time-sensitive conditions such as perinatal conditions, pregnancy complications, and congenital anomalies (achieving win rates of 80% to 100%), whereas label-wise attention (PLM-ICD) grounds predictions in label prototypes, substantially outperforming cross-attention on chronic systemic categories like circulatory, endocrine, and respiratory disorders (exceeding 60% win rates).
To exploit these orthogonal strengths, an MoE gating framework takes the concatenated logit outputs of enhanced PLM-CA and PLM-ICD models as input. Parameterized by a two-layer MLP with Top-k sparse gating, the module dynamically assigns code- and patient-specific weights across learned expert sub-networks, ensuring that each prediction adaptively balances the optimal attention paradigm for superior discrimination.
Loss & Training¶
SARL-ICD employs a modular, two-stage optimization strategy:
-
JMoCo Contrastive Pretraining: Query encoder \(f_q\) and momentum key encoder \(f_k\) (momentum \(m = 0.999\), queue size \(K = 4096\), temperature \(\tau = 0.07\)) are trained on truncated narrative histories using AdamW (learning rate 2e-5, weight decay 0.01) over 3 epochs. The pairwise loss and similarity-weighted objective are: $\(\ell_{ij} = \log \left( 1 + \sum_{n \in \mathcal{N}} \exp \left( \frac{q_i^\top (k_n - k_j)}{\tau} \right) \right), \quad \mathcal{L}_{\text{pretrain}} = \frac{1}{|\mathcal{P}|} \sum_{(i,j) \in \mathcal{P}} s_{ij} \cdot \ell_{ij}\)$
-
Downstream Fine-Tuning and MoE Optimization: The pretrained RoBERTa-PM backbone initializes PLM-CA-L and PLM-ICD-L, fine-tuned across 20 epochs with learning rate 5e-5 and batch size 16 on full clinical discharge summaries (chunked sequence length 4096). Subsequently, backbone parameters are frozen, and the MoE gating network (MIMIC-III: 4 experts, Top-3 routing; MIMIC-IV: 6 experts, Top-4 routing) is optimized using binary cross-entropy (BCE) loss on the combined logits.
Key Experimental Results¶
Main Results¶
SARL-ICD was comprehensively benchmarked against classical CNNs (CAML), structured label attention (LAAT), pretrained transformers (PLM-ICD, PLM-CA), and synonym-based contrastive baselines (MSMN, PLM-LLM) on MIMIC-IV ICD-10, MIMIC-III-full, and MIMIC-IV ICD-9:
| Dataset | Model | Macro-F1 | Micro-F1 | P@8 | P@15 | Macro-AUC | Micro-AUC |
|---|---|---|---|---|---|---|---|
| MIMIC-IV ICD-10 | CAML | 0.210 | 0.584 | 0.668 | 0.522 | 0.910 | 0.982 |
| LAAT | 0.222 | 0.574 | 0.684 | 0.539 | 0.944 | 0.989 | |
| PLM-ICD | 0.211 | 0.585 | 0.699 | 0.550 | 0.966 | 0.992 | |
| PLM-CA | 0.261 | 0.596 | 0.709 | 0.561 | 0.962 | 0.992 | |
| SARL-ICD (Ours) | 0.286 | 0.611 | 0.723 | 0.573 | 0.974 | 0.992 | |
| MIMIC-III-full | CAML | 0.087 | 0.534 | 0.709 | 0.560 | 0.895 | 0.986 |
| LAAT | 0.090 | 0.574 | 0.738 | 0.588 | 0.895 | 0.985 | |
| MSMN | 0.103 | 0.584 | 0.751 | 0.598 | 0.948 | 0.992 | |
| PLM-ICD | 0.104 | 0.598 | 0.771 | 0.613 | 0.926 | 0.989 | |
| PLM-CA | 0.119 | 0.600 | 0.766 | 0.614 | 0.913 | 0.987 | |
| PLM-LLM | 0.103 | 0.613 | 0.779 | 0.615 | 0.952 | 0.988 | |
| SARL-ICD (Ours) | 0.123 | 0.618 | 0.786 | 0.631 | 0.954 | 0.990 | |
| MIMIC-IV ICD-9 | CAML | 0.191 | 0.584 | 0.661 | 0.502 | 0.907 | 0.988 |
| LAAT | 0.293 | 0.612 | 0.688 | 0.527 | 0.958 | 0.991 | |
| PLM-ICD | 0.298 | 0.626 | 0.700 | 0.535 | 0.972 | 0.994 | |
| PLM-CA | 0.324 | 0.633 | 0.707 | 0.540 | 0.971 | 0.994 | |
| SARL-ICD (Ours) | 0.356 | 0.644 | 0.719 | 0.550 | 0.974 | 0.995 |
Ablation Study¶
Stepwise ablations on MIMIC-IV ICD-10 systematically validate the performance gains contributed by pretrained weights (PT), label-text interaction (Label-Text), and MoE fusion:
| Config | Macro-F1 | Micro-F1 | P@8 | P@15 | Note |
|---|---|---|---|---|---|
| PLM-CA Baseline | 0.261 | 0.596 | 0.709 | 0.561 | unaugmented baseline |
| + Label-Text (raw descriptions) | 0.265 | 0.596 | 0.707 | 0.558 | adding unaugmented official descriptions |
| + Label-Text (LLM augmented) | 0.270 | 0.597 | 0.709 | 0.561 | LLM-generated expansions yield clear gains |
| + JMoCo pretraining (PT) | 0.272 | 0.596 | 0.709 | 0.561 | pretrained weights improve macro discrimination |
| + Label-Text + PT (raw desc) | 0.273 | 0.598 | 0.711 | 0.561 | joint pretraining and raw text interaction |
| + Label-Text + PT (LLM augmented) | 0.277 | 0.599 | 0.713 | 0.563 | synergistic pretraining and LLM interaction |
| PLM-ICD Baseline | 0.211 | 0.585 | 0.699 | 0.550 | unaugmented baseline |
| + Label-Text (raw descriptions) | 0.255 | 0.588 | 0.705 | 0.555 | raw descriptions boost Macro-F1 by +4.4% |
| + Label-Text (LLM augmented) | 0.263 | 0.596 | 0.715 | 0.564 | LLM expansions provide further boost |
| + JMoCo pretraining (PT) | 0.260 | 0.596 | 0.714 | 0.563 | JMoCo weights boost PLM-ICD substantially |
| + Label-Text + PT (raw desc) | 0.233 | 0.587 | 0.706 | 0.543 | joint combination with raw text |
| + Label-Text + PT (LLM augmented) | 0.270 | 0.598 | 0.716 | 0.566 | combined modules yield +28.0% relative Macro-F1 |
| Linear Weighted (PLM-CA + PLM-ICD) | 0.283 | 0.609 | 0.715 | 0.570 | static fixed-weight ensemble baseline |
| SARL-ICD (MoE Adaptive Fusion) | 0.286 | 0.611 | 0.723 | 0.573 | dynamic gating attains the highest performance |
Targeted component and section ablation on MIMIC-III-full confirms the importance of pretraining designs:
| Pretraining Strategy & Section Selection | PLM-CA Micro-F1 | PLM-CA Macro-F1 | PLM-ICD Micro-F1 | PLM-ICD Macro-F1 |
|---|---|---|---|---|
| Full JMoCo | 0.606 | 0.122 | 0.601 | 0.118 |
| w/o Hierarchy Expansion | 0.602 | 0.118 | 0.600 | 0.112 |
| w/o Similarity Weighting | 0.602 | 0.120 | 0.601 | 0.109 |
| w/o Narrative Truncation | 0.603 | 0.120 | 0.600 | 0.112 |
| Input: Full discharge summary | 0.603 | 0.120 | 0.600 | 0.112 |
| Input: Narrative sections only | 0.606 | 0.122 | 0.601 | 0.118 |
| Input: Hospital-course only | 0.603 | 0.117 | 0.596 | 0.103 |
| Input: Discharge diagnosis only | 0.602 | 0.120 | 0.598 | 0.110 |
Key Findings¶
- Simultaneous Macro and Micro Superiority: Prior state-of-the-art methods exhibited a stark performance tradeoff: PLM-CA captured rare codes (Macro-F1 11.9%, Micro-F1 60.0%), while PLM-LLM favored dominant codes (Micro-F1 61.3%, Macro-F1 10.3%). SARL-ICD effectively eliminates this tradeoff, leading simultaneously on both Macro-F1 (12.3%) and Micro-F1 (61.8%) on MIMIC-III-full, while posting a 9.6% relative gain in Macro-F1 over PLM-CA on MIMIC-IV ICD-10.
- Section Selection Governs Representation Quality: Pretraining on the full discharge summary degrades performance compared to narrative-only inputs. Crucially, pretraining on the highly templated "hospital-course" section causes PLM-ICD's Macro-F1 to collapse to 0.103. In zero-shot retrieval (Table 8), narrative JMoCo achieves P@5 of 0.070 on frequent labels, compared to only 0.002 for full-text pretraining, empirically verifying that templated sections emit competitive shortcut noise that misguides representation learning.
- Symbiotic Role of Interaction Features: As revealed in Appendix H (Table 14), evaluating the label-text interaction feature in isolation (F-only) yields a low Macro-F1 of 0.089, proving that it serves as an indispensable complementary prior rather than a standalone classifier.
Highlights & Insights¶
- Structure-Aware Document Truncation Suppresses Contrastive Shortcuts: By recognizing that repetitive administrative phrasing contaminates contrastive similarity, the authors strategically train on narrative histories while evaluating on full texts. This deliberate distribution shift prevents models from exploiting template heuristics.
- Decoupled and Architecture-Agnostic LLM Semantic Injection: Expanding sparse coding phrases into detailed pathological descriptions and encoding them via pooled cosine similarity layers provides an elegant, modular interface that elevates diverse transformer architectures without architectural redesigns.
- Empirical Characterization and MoE Exploitation of Dual Attention Biases: The paper quantifies how cross-attention and label-wise attention specialize across surgical/temporal versus chronic metabolic categories, demonstrating that a lightweight MoE gating layer can dynamically unlock the best of both worlds.
Limitations & Future Work¶
- Dependency on Official ICD Hierarchies: JMoCo's positive pair mining relies on standardized parent-child graphs inherent to ICD-9 and ICD-10, making extension to uncurated local hospital vocabularies difficult without auxiliary ontologies like UMLS.
- LLM Quality and Low-Resource Generalization: Label expansions generated by Qwen-7B, while highly rated by DeepSeek judges, may suffer from hallucination or lack domain precision when applied to rare long-tail syndromes or non-English EHR systems.
- Increased Multi-Branch Inference Footprint: Deploying two concurrent transformer backbones (PLM-CA and PLM-ICD) alongside the MoE gating network increases inference latency to 56.1 ms per note and demands ~12.9 GB GPU memory, creating deployment overhead for real-time edge environments.
Related Work & Insights¶
- vs PLM-ICD & PLM-CA (Huang et al., 2022; Edin et al., 2024): Previous backbones feed entire clinical records directly into single attention mechanisms, leaving them vulnerable to semi-structured shortcut learning while locking them into single-model biases. SARL-ICD purifies the underlying encoder via JMoCo and dynamically fuses their complementary outputs through MoE.
- vs PLM-LLM (Wu et al., 2025b): PLM-LLM applies GPT-4 to generate synonyms for label-side contrastive alignment, but ignores the semi-structured template redundancy within patient notes. SARL-ICD tackles both sides by combining LLM label expansion with structure-aware text truncation.
- vs HSAP (Yang et al., 2022): HSAP relies on complex UMLS graph hierarchies and hierarchical contrastive objectives tailored specifically for Longformer. SARL-ICD achieves stronger alignment using intrinsic ICD ancestor trees and a modular cosine interaction mechanism that easily ports to any encoder.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ Pinpoints the semi-structured shortcut dilemma in clinical contrastive learning and introduces an intuitive narrative truncation and dual-backbone MoE fusion scheme.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorously tested across MIMIC-III-full, MIMIC-IV ICD-9, and MIMIC-IV ICD-10 with in-depth section ablations, zero-shot label retrieval, and per-category win rate analyses.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally articulate narrative structure, clear technical formulation, and comprehensive appendix detailing case studies and compute costs.
- Value: ⭐⭐⭐⭐☆ Sets a new performance benchmark for extreme multi-label clinical NLP while offering actionable insights into mitigating shortcut learning in semi-structured document modeling.