Skip to content

FOCAL: Fine-Grained Optimal-Transport-Driven Contrastive Alignment of Language and ECGs with Waveform Enhancement

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/CuCl-2/FOCAL
Area: Reasoning
Keywords: Multimodal ECG Alignment, Optimal Transport, Semantic Soft Targets, Waveform Enhancement, Cross-Modal Representation

TL;DR

To overcome the neglect of localized waveform patterns in global ECG-report alignment and recover missing waveform descriptors omitted in ~55% of clinical reports, FOCAL introduces a Semi-Unbalanced Optimal Transport (Semi-UOT) routing mechanism, a semantic similarity matrix to cancel out tag-level false negatives, and an offline coarse-model-verified LLM enrichment pipeline, achieving a new state-of-the-art average zero-shot Macro AUC of 81.1% across six benchmarks.

Background & Motivation

Electrocardiograms (ECGs) represent the principal non-invasive diagnostic modality for detecting cardiac arrhythmias and cardiovascular disorders in clinical practice. Recent advancements in self-supervised multimodal representation learning have demonstrated remarkable promise by pulling paired 12-lead ECG signals and clinical diagnostic reports together in a shared latent space. Nevertheless, foundational and recent multimodal architectures (such as MERL, MELP, and D-BETA) predominantly enforce alignment at the macroscopic, whole-signal and whole-report level. This global formulation fundamentally fails to capture the intricate, fine-grained relationships between specific localized waveform anomalies—such as ST-segment elevation, T-wave inversion, or lead-specific RSR' patterns—and distinct clinical diagnostic tags, leaving the models sub-optimal on complex diagnostic tasks that require fine-grained morphology comprehension.

Realizing fine-grained multimodal grounding for ECGs faces three formidable hurdles. First, clinical datasets lack segment-level annotations linking individual report findings to exact temporal intervals and lead positions. While traditional Optimal Transport (OT) could theoretically route discrete distributions, standard OT enforces strict, uniform marginal constraints on both modalities. This balanced formulation directly contradicts clinical reality: while every diagnostic tag in a clinical report must be grounded, ECG pathological manifestations are temporally and spatially sparse, meaning abundant background segments (such as normal baselines) carry zero diagnostic utility and should not be forced to absorb semantic mass. Second, decomposing reports into individual diagnostic tags within a contrastive batch drastically amplifies the false-negative problem: different patients frequently share identical common conditions (such as sinus rhythm or myocardial infarction), and standard binary contrastive loss functions (such as SigLIP) penalize non-paired samples by erroneously pushing semantically identical tags apart. Third, approximately 55% of routine ECG reports in large clinical databases (e.g., MIMIC-ECG) omit explicit waveform descriptions entirely. Utilizing Large Language Models (LLMs) to infer missing waveform manifestations from high-level diagnoses is heavily confounded by LLM hallucinations and the non-bijective clinical reality where a single disease can manifest diverse, non-unique waveform morphologies.

To systematically conquer these bottlenecks, the authors propose a principled curriculum that decouples sparse signal dynamics from clinical text semantics and filters generative hallucinations through a cross-modal verification loop. Core idea: formulate localized tag-patch routing as a Semi-Unbalanced Optimal Transport problem that strictly grounds all clinical tags while relaxing ECG patch marginals, guide contrastive learning with a text-derived continuous semantic similarity matrix to eliminate false negatives, and construct an offline coarse-to-fine LLM waveform enrichment pipeline filtered by a coarse multimodal verifier.

Method

Overall Architecture

The end-to-end framework of FOCAL is composed of three interconnected modules: a spatial-temporal ECG patch encoder paired with an independent tag-level clinical text encoder; a parameter-free Semi-Unbalanced Optimal Transport (Semi-UOT) routing module that aggregates patch embeddings into tag-specific ECG representations; and a semantic-guided soft sigmoid contrastive loss function. Operating prior to the final pre-training phase, an offline coarse-to-fine LLM enrichment pipeline queries an LLM for potential waveform markers and subjects them to cross-modal similarity verification against a pre-trained Coarse FOCAL checkpoint, curating a high-fidelity enriched textual corpus.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    subgraph DataPrep["Offline Data Pipeline: Waveform Enrichment"]
        direction TB
        A1["Raw Clinical Reports<br/>55% lack waveform descriptions"] --> A2["LLM Candidate Waveform Generation<br/>Qwen3-8B CoT Prompting"]
        A2 --> A3["Coarse Model Verification & Filtering<br/>Coarse FOCAL threshold 0.95 removes hallucinations"]
        A3 --> A4["Fine-Grained Enriched Reports<br/>High-fidelity disease + morphology text"]
    end

    subgraph Encoding["Multimodal Encoders"]
        direction TB
        B1["12-Lead ECG Signal Xi<br/>Divided into N = L × Ntime Patches"] --> B2["Spatial-Temporal ViT Encoder<br/>Add Lead/Time Embeddings → Hi"]
        A4 --> B3["BioClinicalBERT<br/>Comma-split tags encoded independently → Zi"]
    end

    subgraph CoreAlignment["Core Alignment & Contrastive Objective"]
        direction TB
        B2 & B3 --> C1["Semi-Unbalanced Optimal Transport (Semi-UOT)<br/>Strict text marginals + Relaxed ECG KL penalty"]
        C1 --> C2["Tag-Specific ECG Representations<br/>Transport plan weighted aggregation → e_j^i"]
        C2 & B3 --> C3["Semantic-Guided Soft Sigmoid Contrastive Loss<br/>Online text similarity matrix S neutralizes false negatives"]
    end

    DataPrep --> Encoding
    Encoding --> CoreAlignment

Key Designs

1. Semi-Unbalanced Optimal Transport (Semi-UOT): Resolving the structural asymmetry between localized waveform sparsity and clinical tag necessity

Standard balanced Optimal Transport enforces strict uniform marginal distributions over both domains. However, ECG manifestations are temporally localized and spatially isolated across 12 leads, leaving extensive background patches that contain no pathological evidence. Compelling balanced transport inadvertently disperses clinical tag semantic mass over irrelevant normal baselines, corrupting fine-grained features. Semi-UOT resolves this fundamental asymmetry by enforcing full coverage of clinical text tags while allowing uninformative ECG segments to receive negligible transport mass.

Given \(m_i\) independent tag embeddings \(Z_i = [z_1^i, \dots, z_{m_i}^i] \in \mathbb{R}^{m_i \times d}\) and \(N\) ECG patch tokens \(H_i = [h_1^i, \dots, h_N^i] \in \mathbb{R}^{N \times d}\), a cosine distance cost matrix \(C \in \mathbb{R}^{m_i \times N}\) is formulated where: $\(C_{j,p} = 1 - \frac{(z_j^i)^\top h_p^i}{\|z_j^i\|_2 \|h_p^i\|_2}\)$ The optimal transport plan \(T^* \in \mathbb{R}_+^{m_i \times N}\) is optimized with entropic regularization and a Kullback-Leibler (KL) divergence penalty on the ECG side: $\(T^* = \arg\min_{T \mathbf{1}_N = a} \sum_{j=1}^{m_i} \sum_{p=1}^N T_{j,p} C_{j,p} - \epsilon H(T) + \tau \text{KL}(T^\top \mathbf{1}_{m_i} \| b)\)$ where \(a = \frac{1}{m_i} \mathbf{1}_{m_i}\) and \(b = \frac{1}{N} \mathbf{1}_N\). The hard constraint \(T \mathbf{1}_N = a\) guarantees that all diagnostic tags are grounded, while the soft KL divergence term governed by relaxation factor \(\tau\) lets the model naturally disregard irrelevant background patches. Solved efficiently via the unbalanced Sinkhorn-Knopp algorithm, the normalized plan \(\hat{T}_{j,p} = m_i \cdot T_{j,p}^*\) produces the tag-specific ECG representation: $\(e_j^i = \sum_{p=1}^N \hat{T}_{j,p} h_p^i\)$ Because the iterative Sinkhorn algorithm contains strictly differentiable matrix primitives, the routing is entirely parameter-free and allows gradient backpropagation end-to-end to both encoders.

2. Semantic-Guided Soft Targets: Mitigating false-negative collisions inherent in tag-level contrastive alignment

Decomposing clinical reports into \(K = \sum_{i=1}^B m_i\) individual tags across a batch of \(B\) reports drastically enlarges the contrastive negative pool. However, identical or semantically congruent diagnoses (such as "Atrial Fibrillation" or "ST Elevation") occur routinely across unrelated patients. Traditional contrastive losses (such as InfoNCE or standard SigLIP) treat all non-paired elements as hard negatives, creating conflicting repulsive gradients between semantically equivalent clinical conditions and degrading representation quality.

FOCAL resolves this by deriving a continuous semantic similarity matrix \(S \in [0, 1]^{K \times K}\) directly from the text tag embeddings, where \(S_{k,l} = \text{sim}(z_k, z_l)\). Replacing hard binary labels with continuous soft targets, the semantic-guided soft sigmoid loss is formulated as: $\(\mathcal{L} = \frac{1}{K^2} \sum_{k=1}^K \sum_{l=1}^K \left[ S_{k,l} \log(1 + \exp(-x_{k,l})) + (1 - S_{k,l}) \log(1 + \exp(x_{k,l})) \right]\)$ where \(x_{k,l} = t \cdot \text{sim}(e_k, z_l)\) is the temperature-scaled cosine similarity between the \(k\)-th tag-specific ECG embedding and the \(l\)-th text tag embedding, and \(t\) is a learnable scaling parameter initialized to \(\log 10\). When non-paired tags share identical medical semantics (\(S_{k,l} \approx 1\)), they act as positive soft targets, effectively neutralizing erroneous repulsive forces at the gradient level.

3. Coarse-to-Fine Waveform Enrichment: Filtering LLM hallucinations and non-bijective clinical mappings

Approximately 55% of routine ECG reports in MIMIC-ECG lack explicit waveform morphology descriptions. While LLMs can hypothesize potential waveform abnormalities from diagnostic labels, the non-bijective nature of cardiology (a single disease may produce heterogeneous waveform variants) combined with LLM hallucinations introduces destructive false positives if fed directly into contrastive training.

The framework deploys a robust three-stage coarse-to-fine curriculum: - Step 1: Pre-train a Coarse FOCAL model using contrastive learning solely on the original, unaugmented ECG-report pairs. - Step 2: Query an LLM (Qwen3-8B accelerated by vLLM) with a zero-shot prompt requesting candidate waveform manifestations for each clinical report. Next, compute the cross-modal similarity between each generated candidate \(f_j\) and the actual ECG signal using the Coarse FOCAL model, retaining only candidates exceeding a high-confidence threshold of \(\tau_{\text{filter}} = 0.95\). - Step 3: Append the verified waveform features to the original reports and continue training Coarse FOCAL on this enriched corpus to establish the final FOCAL model. This offline process incurs zero computational overhead during downstream inference.

Loss & Training

  • Network Architecture: 120 non-overlapping patches fed to a randomly initialized Spatial-Temporal ViT for ECG encoding; BioClinicalBERT for tag text encoding; shared projection dimension \(d\).
  • Optimization Strategy: AdamW optimizer with initial learning rate \(2 \times 10^{-5}\), weight decay \(1 \times 10^{-4}\), batch size 100, cosine annealing scheduler with 10% warmup.
  • Curriculum Schedule: Coarse FOCAL is pre-trained for 10 epochs on original MIMIC-ECG pairs; after feature generation and filtering at threshold 0.95, fine FOCAL is trained for an additional 3 epochs on the augmented corpus.
  • Hyperparameters: Semi-UOT entropy parameter \(\epsilon = 0.1\), KL divergence penalty \(\tau = 1.0\).

Key Experimental Results

Main Results

Zero-shot multi-label classification performance was evaluated across six benchmarks: PTB-XL (Superclass 5 classes, Subclass 23 classes, Form 19 classes, Rhythm 12 classes), CPSC2018 (9 classes), and CSN (38 classes), evaluated via Macro AUC (%).

Method PTBXL-Super PTBXL-Sub PTBXL-Form PTBXL-Rhythm CPSC2018 CSN Average
MERL (Liu et al., 2024b) 74.2 75.7 65.9 78.5 82.8 74.4 75.3
MELP (Wang et al., 2025) 76.2 81.2 69.1 85.4 84.2 77.6 79.0
D-BETA (Hung et al., 2025) 76.2 75.9 66.1 88.6 80.1 76.3 77.1
FOCAL (Ours) 77.7 83.5 70.3 90.8 85.0 79.2 81.1
Margin over best baseline +1.5 +2.3 +1.2 +2.2 +0.8 +1.6 +2.1

In linear probing evaluations with frozen ECG backbones across 1%, 10%, and 100% fine-tuning regimes, FOCAL consistently established top performance over 10 contrastive and generative SSL baselines (e.g., Barlow Twins, MoCo-v3, TS-TCC, ST-MEM), confirming superior feature transferability.

Ablation Study

Ablation results on architectural choices, loss formulation, and training curriculum on average zero-shot Macro AUC (Table 3 and Table 10 from the paper):

Configuration / Variant Avg. Zero-Shot Macro AUC (%) Drop (Δ) Key Finding / Mechanism
FOCAL (Default Full Model) 81.1 - Full Semi-UOT + Soft Targets + Verified Enrichment
w/ Cross-Attention (replacing OT) 78.3 -2.8 Attention over-smooths weights, losing sparse localized grounding
w/ Standard Balanced OT (Strict marginals) 79.2 -1.9 Uniform constraints force noise assignment to uninformative baselines
w/o Fine-Grained Alignment (Global-only) 76.0 -5.1 Macro alignment fails to differentiate local morphological anomalies
w/o Semantic Similarity Matrix (Hard Negatives) 77.6 -3.5 False negative repulsive gradients severely corrupt latent geometry
w/o LLM Enrichment (Coarse FOCAL, 10 ep) 79.6 -1.5 Omission of waveform descriptors leaves local morphology unguided
Coarse FOCAL + 3 extra epochs (original text) 79.5 -1.6 Proves gains stem strictly from enriched features rather than training steps
Global only (w/o both Fine-Grained & Enrichment) 74.8 -6.3 Pure macro baseline; highlights the +5.1% and +1.5% orthogonal gains

In human verification experiments on 200 clinician-annotated records, unverified LLM features yielded low retrieval AUCs (71.42% to 84.25%), whereas Coarse FOCAL verification lifted feature AUCs to 83.42%–93.25% (ST elevation: 71.42 → 83.42; Long QT: 84.25 → 93.25; Abnormal QRS: 76.68 → 91.68). Feeding these enriched reports to the external MERL baseline yielded an immediate +2.2% average zero-shot gain across all six benchmarks, proving model-agnostic transferability.

Key Findings

  • Fine-grained alignment is the primary driver of performance: The \(2 \times 2\) ablation confirms that the Semi-UOT fine-grained architecture delivers a +5.1% absolute improvement over global alignment, while waveform enrichment delivers a consistent, orthogonal +1.5% boost.
  • Greatest improvements on localized morphology and rhythm categories: FOCAL achieved its most pronounced advantages on PTBXL-Rhythm (90.8% vs. MELP's 85.4%, a +5.4% gain) and PTBXL-Form (70.3%), directly demonstrating the efficacy of grounding localized segments to precise tags.
  • Hyperparameter robustness: Performance remains within 80.6%–81.1% across wide parameter intervals (\(\epsilon \in [0.05, 0.2]\) and \(\tau \in [0.5, 2.0]\)), but sharply declines when \(\tau \to \infty\) (enforcing standard balanced OT) or \(\epsilon \to 0\) (unstable exact OT).
  • Inference efficiency: Whereas MERL incurs an inference-time overhead of 1.17× due to live LLM querying during test time, FOCAL shifts LLM calls entirely offline, executing only parallelizable Sinkhorn-Knopp operations at test time with 1.00× relative latency.

Highlights & Insights

  • Physiologically grounded formulation via Semi-UOT: By translating the domain reality that clinical tags must hold while ECG signals are temporally sparse into unbalanced optimal transport constraints, FOCAL provides an elegant mathematical solution to multimodal grounding without ad-hoc heuristic thresholding.
  • Continuous semantic soft targets neutralize cross-sample false negatives: Generating an online semantic similarity matrix from pre-trained clinical language embeddings provides smooth, continuous supervision that converts identical inter-report tags into constructive soft targets rather than harmful negative distractors.
  • Coarse model acts as a cross-modal verification filter: Leveraging the initially trained coarse model to filter LLM-generated waveform descriptors resolves the challenging non-bijective mapping in cardiology, yielding an open-source, high-quality data asset that universally benefits external model architectures.

Limitations & Future Work

  • Author-admitted limitation: Querying LLMs over large-scale clinical cohorts (e.g., nearly 800,000 reports in MIMIC-ECG) entails substantial upfront computational and time investments during offline preprocessing, though amortized as a one-time data curation effort.
  • Remaining boundary issues: In cases with severe baseline wandering, extreme low voltage, or high noise artifacts across all 12 leads, the initial cost matrix computation may experience bias, potentially hindering optimal transport convergence.
  • Future directions: The coarse-to-fine verification paradigm and semi-unbalanced routing can be naturally extended to other sparse physiological time-series modalities, such as multi-channel electroencephalograms (EEGs) and electromyograms (EMGs).
  • vs MERL (Liu et al., 2024b): MERL relies on unimodal contrastive alignment and online LLM reasoning prompts (CKEPE) during inference, incurring 1.17× relative test latency. FOCAL performs fine-grained alignment at the patch-tag level and handles all LLM enrichment offline, outperforming MERL by +5.8% Macro AUC while maintaining 1.00× test speed.
  • vs MELP (Wang et al., 2025): MELP captures multi-scale ECG structures across tokens, beats, and rhythms via fixed hierarchical heuristics. FOCAL relies on optimal transport to dynamically discover the optimal mapping between arbitrary localized waveforms and clinical tags without rigid pre-segmented boundaries.
  • vs MedFILIP (Liang et al., 2025) & fVLM (Shui et al., 2025): While fine-grained alignment has been applied to spatial anatomical regions in radiological images, FOCAL pioneers fine-grained multi-lead spatial-temporal ECG alignment and provides a robust solution to the severe label false-negative collisions inherent in clinical reports.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ [Pioneering integration of Semi-Unbalanced Optimal Transport and semantic soft targets to solve physiological sparsity and false negatives in ECGs]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Rigorous evaluation across 6 datasets in zero-shot, linear probing, cross-domain distribution shifts, and human clinician validation]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical derivations, insightful motivation, and transparent visual activation analyses]
  • Value: ⭐⭐⭐⭐⭐ [Delivers a 1.00× real-time inference architecture alongside an open-source, verified fine-grained multimodal clinical dataset]