Revitalizing Medical Time Series with Vision-Informed Retrieval: A Vision-Language Perspective¶
Conference: NeurIPS 2026
arXiv: 2609.34652
Code: https://github.com/yu-guoqi/ViRe
Area: Time Series
Keywords: Medical Time Series, Vision-Language Models, Cross-Modal Retrieval, EEG/ECG Classification, Cross-Attention
TL;DR¶
Addressing the limitation of existing medical time series models that treat signals purely as numerical sequences while ignoring visual waveform morphology, ViRe extracts morphology-aware Vision Queries from rendered waveform plots using a frozen CLIP model to retrieve clinical evidence from numerical features via cross-attention, achieving an average 6.42% relative improvement across six public benchmarks.
Background & Motivation¶
Medical time series (MedTS), notably electrocardiography (ECG) and electroencephalography (EEG), serve as foundational diagnostic modalities for identifying cardiac arrhythmias, detecting epileptic seizures, monitoring sleep architecture, and stratifying cardiovascular risks. While machine learning transitioned from traditional handcrafted time-frequency feature engineering to modern convolutional networks, recurrent models, and long-range Transformer architectures, prevailing deep learning models remain grounded in a reductive premise: they treat MedTS strictly as one-dimensional multi-channel numerical sequences or discrete data matrices, optimizing representations solely over amplitude trajectories and point-wise numerical correlations.
This numerics-oriented paradigm creates a fundamental disconnect with real-world clinical diagnostic reasoning. In clinical workflows, cardiologists and neurologists do not scrutinize raw numerical matrices; instead, they inspect waveform tracings (such as P-QRS-T complexes, ST-segment deviations, epileptiform discharges, and cross-lead synchronous rhythms) to detect global shapes and localized pathological deviations, summarizing these visual cues into structured diagnostic reports. Conventional sequence models lack the inductive bias associated with waveform visual geometry and morphology, leaving them sensitive to device drift, scaling shifts, and transient baseline wander, which hinders them from forming diagnostic evidence aligned with clinical expert reasoning.
However, training a visual perception network from scratch to understand waveform morphology is data-hungry and prone to overfitting under severe cross-patient and cross-device domain shifts. In contrast, large-scale vision-language models (VLMs) such as CLIP have learned structured visual representations under natural language supervision that naturally align with human-describable morphological concepts, capturing trends, spikes, peaks, and periodic fluctuations. Core idea: deterministically render multi-channel medical time series into stacked waveform plots, extract high-level morphology-aware priors as Vision Queries using a frozen CLIP vision encoder, and deploy asymmetric dual-path cross-attention to retrieve pathological evidence from numerical temporal and channel token representations.
Method¶
Overall Architecture¶
ViRe bridges numerical time-series modeling with visual waveform interpretation through an asymmetric cross-modal retrieval framework. The architecture maintains two complementary numerical representation pathways (temporal patch dynamics and full-trajectory channel correlations) while feeding a deterministically rendered multi-lead waveform plot through a frozen CLIP ViT vision encoder to obtain a global morphological query.
The complete workflow comprises four primary stages: dual-axis numerical tokenization (temporal slicing and channel-trajectory projection), deterministic waveform visualization and visual prior extraction, asymmetric cross-modal retrieval via dual cross-attention encoders, and dual-summary fusion for disease diagnosis. In this design, the frozen visual branch acts exclusively as a retrieval Query generator without serving as an independent classification pathway, ensuring that all Keys and Values originate strictly from original numerical tokens while being selectively aggregated under the guidance of morphological priors.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
IN["Multi-Channel Medical Time Series<br/>X โ R^(T ร C)"] --> N_T["Temporal Axis Tokenization<br/>Non-overlapping Patch Mapping U"]
IN --> N_C["Channel Trajectory Tokenization<br/>Full Lead Projection V"]
IN --> V_OP["Deterministic Waveform Rendering<br/>Stacked Multi-Panel Image I"]
N_T --> E_T["Temporal Encoder<br/>Temporal Features U-bar"]
N_C --> E_C["Channel Encoder<br/>Lead Features V-bar"]
V_OP --> CLIP["Frozen CLIP Vision Encoder<br/>Morphology Embedding Z"]
CLIP --> ALIGN["Dimension Alignment Layer<br/>Global Vision Query Q โ R^(1 ร D)"]
ALIGN --> CA_T["Temporal Cross-Attention Retrieval<br/>CrossAttn(Q, U-bar, U-bar)"]
ALIGN --> CA_C["Channel Cross-Attention Retrieval<br/>CrossAttn(Q, V-bar, V-bar)"]
E_T --> CA_T
E_C --> CA_C
CA_T --> FUSE["Dual Retrieval Summary Addition<br/>O = U~ + V~"]
CA_C --> FUSE
FUSE --> OUT["Linear Classification Head<br/>Diagnostic Logits Y-hat"]
Key Designs¶
1. Dual-Axis Complementary Numerical Tokenization: Decoupling Temporal Evolution and Spatial Channel Semantics To thoroughly capture both intra-channel temporal evolutions and inter-channel spatial correlations, ViRe performs tokenization along two orthogonal axes on the input signal \(X \in \mathbb{R}^{T \times C}\). Along the temporal axis, the multi-channel sequence is divided into non-overlapping temporal segments of length \(L\); each segment is flattened and linearly projected with a learnable positional embedding to form temporal tokens \(U \in \mathbb{R}^{P \times D}\) (where \(P = \lceil T/L \rceil\)), capturing short-term waveform shape and long-range temporal trends. Along the channel axis, the full temporal trajectory of each individual lead over the entire duration \(T\) is embedded into a single channel token with channel positional embeddings, constructing \(V \in \mathbb{R}^{C \times D}\) to retain lead-specific physical semantics. Both token sets are processed by independent 6-layer Transformer encoders to produce context-rich temporal features \(\bar{U} \in \mathbb{R}^{P \times D}\) and channel features \(\bar{V} \in \mathbb{R}^{C \times D}\).
2. Deterministic Waveform Visualization and Frozen Vision Query Extraction: Morphological Priors with Zero Added Backbone Parameters To introduce visual morphological priors without task-specific vision fine-tuning, ViRe incorporates a deterministic visualization operator \(\mathcal{V}(X)\). The operator processes all \(C\) channels individually, plotting each lead against its sampling timestamps on an independent canvas while stripping non-clinical visual artifacts such as ticks, coordinate grids, spines, and legends. The resulting individual RGB panels are vertically concatenated in a canonical order into a compact composite image \(I \in \mathbb{R}^{H \times W \times 3}\). Passing \(I\) through a frozen pre-trained CLIP ViT vision encoder produces a high-level representation \(Z \in \mathbb{R}^{\tilde{D}}\). A lightweight linear dimension-alignment layer projects \(Z\) into the shared numerical latent dimension: $\(\bar{Z} = Z W_z + b_z, \quad W_z \in \mathbb{R}^{\tilde{D} \times D}, \; b_z \in \mathbb{R}^D\)$ Reshaped to \(Q \in \mathbb{R}^{1 \times D}\), \(\bar{Z}\) serves as the global Vision Query. Leveraging CLIP's pre-trained alignment between visual features and descriptive concepts, \(Q\) encapsulates holistic morphological concepts (e.g., sharp spikes, baseline deviations, and abnormal synchrony) while keeping the vision backbone entirely frozen during training.
3. Asymmetric Dual-Path Cross-Modal Retrieval: Content-Dependent Adaptive Pooling of Numerical Evidence Rather than performing naive element-wise addition or feature concatenation, ViRe uses the global Vision Query \(Q\) to retrieve relevant numerical evidence via cross-attention. Using \(Q\) as the shared query, temporal representations \(\bar{U}\) and channel representations \(\bar{V}\) act as Keys and Values: $\(\widetilde{U} = \mathrm{CrossAttn}(Q, \bar{U}, \bar{U}) \in \mathbb{R}^{1 \times D}, \qquad \widetilde{V} = \mathrm{CrossAttn}(Q, \bar{V}, \bar{V}) \in \mathbb{R}^{1 \times D}\)$ Mathematically, this cross-attention operation realizes a sample-adaptive pooling rule jointly determined by the visual query and numerical keys: $\(\alpha(Q, K) = \mathrm{softmax}\left(\frac{Q W_Q (K W_K)^\top}{\sqrt{d}}\right), \qquad \mathrm{Attn}(Q, K) = \sum_{s=1}^S \alpha_s(Q, K) k_s W_V\)$ When the Vision Query detects morphological abnormalities (such as an extended QRS complex or epileptic spike discharges), the attention distribution dynamically concentrates on the relevant time intervals and pathological channels. The resulting summaries are fused via element-wise addition and projected to class logits: $\(O = \widetilde{U} + \widetilde{V}, \qquad \widehat{Y} = O W_y + b_y\)$
Loss & Training¶
The framework is optimized end-to-end using standard cross-entropy loss: $\(\mathcal{L} = -\sum_{k=1}^K Y_k \log \widehat{Y}_k\)$ During training, a consistent data augmentation strategy is applied where one operation is sampled uniformly at random from six candidates: temporal flipping, channel shuffling, temporal masking, frequency masking, sensor jittering, and dropout. Both the numerical inputs and the waveform renderings undergo synchronized transformations to prevent the visual branch from exploiting rendering artifacts. Augmentations are completely disabled during validation and testing. Furthermore, because the rendering is deterministic and the vision encoder is frozen, CLIP features can be precomputed and cached offline, keeping single-epoch training overhead virtually identical to pure numerical Transformer baselines.
Key Experimental Results¶
Main Results¶
ViRe was benchmarked against ten competitive time series architectures (including Autoformer, FEDformer, Informer, iTransformer, MTST, PatchTST, Reformer, and Medformer) across six public EEG and ECG benchmarks under strict subject-independent splits. The table below presents the average score (Avg, computed as the arithmetic mean across Accuracy, Precision, Recall, F1, AUROC, and AUPRC):
| Dataset | Modality / Classes | Best Baseline Model | Best Baseline Avg | ViRe (Ours) | Relative Gain |
|---|---|---|---|---|---|
| APAVA | EEG (2 classes) | Medformer | 79.74 | 92.79 | +16.37% (+13.05) |
| ADFTD | EEG (3 classes) | Medformer | 54.63 | 59.83 | +9.52% (+5.20) |
| TDBrain | EEG (2 classes) | Medformer | 91.91 | 95.54 | +3.95% (+3.63) |
| PTB | ECG (2 classes) | iTransformer | 84.95 | 89.08 | +4.86% (+4.13) |
| PTB-XL | ECG (5 classes) | PatchTST | 69.90 | 69.47 | -0.61% (-0.43) |
| MIMIC | ECG (2 classes) | Reformer | 87.90 | 90.68 | +3.16% (+2.78) |
| Mean | Across 6 Benchmarks | Medformer | 77.90 | 82.90 | +6.42% (+5.00) |
Ablation Study¶
The table below presents comprehensive ablations across retrieval query types, modality fusion strategies, and vision backbones evaluated on ADFTD, APAVA, PTB, and MIMIC, reporting Accuracy, F1-Score, and average relative improvement over the baseline without retrieval (w/o):
| Ablation Dimension | Configuration | ADFTD (Acc / F1) | APAVA (Acc / F1) | PTB (Acc / F1) | MIMIC (Acc / F1) | Avg. Relative Gain |
|---|---|---|---|---|---|---|
| Query Type | Without Retrieval (w/o) | 54.79 / 51.73 | 83.86 / 82.45 | 81.45 / 74.58 | 84.92 / 84.81 | Baseline (0.00%) |
| All-Zero Query (Zero) | 54.72 / 51.23 | 84.34 / 83.73 | 84.62 / 80.21 | 86.13 / 86.06 | +1.92% | |
| Gaussian Noise Query | 54.46 / 51.54 | 82.89 / 81.65 | 82.83 / 77.34 | 86.50 / 86.44 | +0.76% | |
| Vision Query (ViRe) | 57.83 / 54.02 | 91.43 / 91.19 | 88.26 / 85.81 | 88.61 / 88.54 | +7.72% | |
| Fusion Strategy | Element-wise Addition (Add) | 56.67 / 53.53 | 85.95 / 85.45 | 84.81 / 81.26 | 87.57 / 87.49 | +4.05% |
| Concatenation (Concat) | 56.86 / 53.55 | 84.23 / 83.02 | 83.60 / 79.13 | 87.88 / 87.81 | +3.02% | |
| Cross-Attention Retrieval | 57.83 / 54.02 | 91.43 / 91.19 | 88.26 / 85.81 | 88.61 / 88.54 | +7.72% | |
| Vision Backbone | Random Init ViT | 37.39 / 27.34 | 80.62 / 77.18 | 80.65 / 73.50 | 83.67 / 83.58 | -11.81% |
| ImageNet Pre-trained ViT | 52.21 / 50.55 | 84.48 / 84.86 | 82.06 / 75.72 | 86.52 / 86.42 | +0.34% | |
| CLIP Vision Encoder | 57.83 / 54.02 | 91.43 / 91.19 | 88.26 / 85.81 | 88.61 / 88.54 | +7.72% |
Key Findings¶
- Vision Queries are the primary driver of performance gains: Uninformative queries (all-zero or Gaussian noise) produce marginal gains (+0.76% to +1.92%), whereas the VLM-derived Vision Query delivers a +7.72% gain, confirming that the improvement stems from semantic morphological guidance rather than added cross-attention parameters.
- Language-supervised vision priors drastically outperform generic visual features: A randomly initialized ViT severely degrades accuracy (-11.81%), and ImageNet pre-training provides only a modest +0.34% improvement. In contrast, CLIP-Vision achieves +7.72%, demonstrating that natural-language supervision organizes visual representations into concept-aligned spaces effective for biomedical waveform interpretation.
- Strongest advantages on data-constrained and subtle pathologies: ViRe demonstrates its largest improvements on smaller cohorts like APAVA (+16.37%) and difficult multi-class settings like ADFTD (+9.52%). On PTB-XL, fine-grained diagnostic gains reach +19.78 F1 points on left ventricular hypertrophy (LVH) and +15.32 on myocardial hypertrophy (HYP).
- Attention aligns tightly with clinical morphology: Cross-attention exhibits significant concentration on high-curvature timestamps (Morphology-Aware Attention Ratio, MAR = 1.81 on PTB) and clinical QRS complexes (Clinical Alignment Ratio, CAR = 1.65). Visual shuffling across different patients reduces macro-F1 by 15.61 points, verifying that retrieval focuses on genuine, patient-specific morphological markers.
Highlights & Insights¶
- Repurposing foundation vision models as lightweight retrieval priors: Rather than treating heavy vision transformers as end-to-end classifiers, ViRe frames them as frozen Query generators for numerical sequence retrieval, preserving the numerical precision of time series while infusing global geometric priors.
- Theoretical generalization over mean pooling: The authors establish that standard scaled dot-product cross-attention mathematically subsumes uniform mean pooling as a parameter subspace, guaranteeing that ViRe's hypothesis space contains standard numerical pooling architectures with bounded empirical risk.
- Plug-and-play cross-modal enhancement: By rendering raw time series deterministically and pre-caching CLIP features offline, ViRe avoids joint multimodal end-to-end training and eliminates the need for expensive paired biomedical text annotations.
Limitations & Future Work¶
- Author-acknowledged limitations: The visualization pipeline relies on a fixed vertical stacking layout across channels. For high-density recordings (e.g., 128-lead EEG), vertical concatenation within standard image dimensions may lead to resolution compression and visual overlap.
- Additional observed limitations: The current design focuses on windowed or heartbeat-segmented recordings and has not been tested on continuous long-duration monitoring (e.g., 24-hour ambulatory Holter ECG). Furthermore, on massive and regular datasets like PTB-XL, highly tuned numerical patch models (PatchTST) remain marginally ahead, indicating diminishing returns for visual retrieval in large-scale periodic regimes.
- Future directions: Potential improvements include exploring adaptive layout rendering (e.g., dynamic lead placement or color-coded frequency overlays) and incorporating the CLIP text encoder to inject clinical diagnostic prompts for tri-modal alignment.
Related Work & Insights¶
- vs Medformer (NeurIPS 2024): While Medformer captures cross-granularity patch dynamics purely within the numerical domain, ViRe introduces visual morphological priors and cross-attention retrieval, achieving significant gains on APAVA and ADFTD.
- vs TS-CLIP (EMNLP 2025) / ViTime (TMLR 2025): TS-CLIP aligns time series directly with text for zero-shot tasks, whereas ViTime casts the visual image domain as the primary modeling space. ViRe uniquely uses frozen vision features as asymmetric queries to guide retrieval over raw numerical sequences, balancing morphological intuition with numerical granularity.
Rating¶
- Novelty: โญโญโญโญโญ Formulating frozen VLM waveform representations as retrieval queries for numerical time series is an insightful and original perspective.
- Experimental Thoroughness: โญโญโญโญโญ Thoroughly evaluated across 6 public EEG/ECG benchmarks against 10 strong baselines, backed by rigorous clustering, attention alignment, and concept-decoding analyses.
- Writing Quality: โญโญโญโญโญ Exceptionally clear narrative, rigorous theoretical inclusion proofs, and self-contained methodological explanations.
- Value: โญโญโญโญโญ Bridges computer vision foundation models and physiological signal analysis, offering a reproducible and computation-friendly blueprint.