Progressive Memory Transformer: Memory-Aware Attention for Time-Series¶
Conference: NeurIPS 2026
arXiv: 2609.31351
Code: https://github.com/dsb-ifi/pmt
Area: Time Series
Keywords: Time-Series Representation Learning, Progressive Memory, Multi-Scale Contrastive Learning, Self-Supervised Learning, Time-Series Forecasting
TL;DR¶
PMT introduces Progressive Memory Transformer, which equips sliding windows with writable, window-aligned memory slots that expose mid-range motifs as first-class representations alongside tokens and sequence summaries, achieving state-of-the-art low-label classification across seven benchmarks (1% labels beating prior 5% baselines) and superior long-horizon forecasting under a three-scale contrastive learning framework.
Background & Motivation¶
Time-series signals inherently exhibit structural dynamics across multiple temporal scales simultaneously: high-frequency variations within short windows, recurring motifs and regimes at intermediate scales, and global semantics governing entire sequence trajectories. Correspondingly, downstream tasks operate at varied granularitiesโlocal forecasting relies on fine-grained token dynamics, motif detection demands intermediate temporal representations, and sequence classification targets holistic global properties. However, prevailing self-supervised learning (SSL) paradigms for time-series remain largely anchored to instance-level global contrastive objectives or localized temporal neighborhood discrimination, placing the supervisory burden almost exclusively on global readout representations.
This global-centric convention overlooks the inherent hierarchical structure of time-series data, resulting in the destructive compression or discarding of informative intermediate dynamics. Conventional architectures aggregate intermediate hidden states via aggressive max-pooling or sequence-wide averaging, treating them merely as transient computational variables. Even recurrent architectures and segment-caching transformers (such as Transformer-XL) treat their recurrent hidden states or read-only caches as internal computational devices rather than representation interfaces exposed to direct, scale-matched supervision. Meanwhile, recent memory-augmented transformers introduce persistent memory banks that remain decoupled from sliding temporal windows, rendering them ill-suited for localized mid-range supervision.
To address the absence of direct mid-range representation and multi-scale supervision, this paper introduces an explicit three-scale contrastive framework. For such a hierarchy to operate effectively, the underlying backbone must expose representations at every level. Core idea: Progressive Memory Transformer (PMT) augments a transformer with sample-specific, window-aligned writable memory slots that propagate horizontally across sliding windows and vertically across layers, exposing mid-range motif representations as first-class citizens directly supervised alongside token-level Gaussian contrastive learning and sequence-level instance contrastive learning.
Method¶
Overall Architecture¶
The PMT processing pipeline consists of four integrated stages: 1D convolutional patchification, Progressive Memory Attention (PMA) block stacking, attentive overlap aggregation, and global readout encoding: 1. 1D Convolutional Patchification: For an input time-series waveform \(x \in \mathbb{R}^{L \times C}\) of length \(L\) with \(C\) channels, a 1D convolution with kernel size and stride \(k\) tokenizes the raw signal into \(K\) patch tokens with embedding dimension \(D\); 2. Progressive Memory Attention (PMA) Blocks: The \(K\) patch tokens are unfolded into overlapping sliding windows of length \(W\) and stride \(S\) (overlap ratio \(O = \lceil W/S \rceil\)). Each PMA block receives horizontal memory slots from the previous window and vertical memory plus token representations from the preceding layer, applying adaptive gates and asymmetric attention to update local tokens and memory slots in place; 3. Attentive Overlap Aggregator: Because adjacent windows overlap, each global token position is computed \(O\) times within each block; a single-query cross-attention pooling module aggregates these overlapping copies to preserve a constant sequence length across network depth; 4. Neighborhood Masked Encoding & Multi-Scale Contrastive Supervision: The refined token stream is processed through a shallow transformer encoder with neighborhood masking (restricting patch tokens to local contexts) while a dedicated \([CLS]\) token attends globally to aggregate sequence summaries. The network is end-to-end trained with three independent objectives: Hierarchical Gaussian Contrastive Loss (HGCL) on fine-grained tokens, PMA Contrastive Loss (PCL) on mid-range memory slots, and Instance Contrastive Loss (ICL) on the global \([CLS]\) summary.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Multichannel Waveform<br/>1D Conv Tokenization into K Patch Tokens"] --> B["Progressive Memory Attention (PMA)<br/>Horizontal Memory Carry + Vertical Layer Fusion"]
B --> C["Attentive Overlap Aggregator<br/>Merge O-fold Overlaps to Constant Length"]
C --> D["Dual Global Feature Refinement<br/>Neighborhood-Masked Tokens + Global CLS Summary"]
D --> E["Three-Scale Independent Supervision<br/>Token-Level HGCL + Mid-Range PCL + Global ICL"]
Key Designs¶
1. Progressive Memory Attention: Dual-Axis Horizontal Propagation and Vertical Layer Fusion
To resolve the trade-off in standard transformers between quadratic compute on long contexts and abrupt context truncation under local windowing, PMA equips every sliding window with \(n_m\) writable memory slots. At layer block \(b\) and temporal window \(w\), PMA integrates horizontal memory \(M_{w-1,b}\) from the preceding window at the same layer, vertical memory \(\bar{M}_{w,b-1}\) from the previous layer, and previous-layer tokens \(\bar{W}_{w,b-1}\). To handle regime transitions and prevent stale context contamination, two adaptive gates regulate the inputs: memory gate \(G_M\) softly blends inherited memory with a learned reset state \(M_r\) (analogous to LSTM reset gates), while token gate \(G_W\) blends processed tokens with raw patch representations \(W_{w,0}\) to retain low-level morphology. The gated representations are concatenated and processed by a single Multi-Head FlashAttention layer: $\([M_{w,b}, W_{w,b}^{\text{out}}] = \mathrm{SelfAttn}\left(\left[M_{w-1,b} \,\|\, \bar{M}_{w,b-1} \,\|\, \bar{W}_{w,b-1}\right]\right)\)$ Under asymmetric attention masking, memory slot queries remain unmasked to read and write across both memory and tokens, whereas token queries only attend to causal tokens and horizontal memory \(M_{w-1,b}\), strictly preventing information leakage from deeper vertical summaries \(\bar{M}_{w,b-1}\). Horizontal memory expands receptive field by stride \(S\) per window while depth adds \(W-S\) nominal tokens per layer, allowing deep layers to compose full-sequence context without re-scanning historical tokens.
2. Attentive Overlap Aggregator: Distortion-Free Fusion of Overlapping Sliding Windows
When sliding windows overlap (\(S < W\)), adjacent windows share positions by an overlap factor \(O = \lceil W/S \rceil\). Stacking blocks without consolidation causes token sequence lengths to explode exponentially with depth. To maintain a constant token sequence length across blocks without sacrificing multi-view information, PMT introduces a lightweight Attentive Overlap Aggregator at the end of each PMA block. For any global position \(t\) with \(O_t\) overlapping representations \(\{W_t^{(r)}\}_{r=1}^{O_t}\), the aggregator utilizes a single per-position query \(q_h\) (produced via depthwise 1D convolution and a two-layer MLP) to attend over the overlapping keys and values: $\(A_t = \gamma \, \mathrm{LN}\left(O^{-1/2} \mathrm{Concat}_{h=1}^{H} \sum_{r=1}^{O_t} \alpha_{h,r} V_{h,r}\right), \quad \alpha_{h,r} = \mathrm{softmax}_r \left(\frac{\langle q_h, K_{h,r} \rangle}{\sqrt{d_h}} + b_{h,r}\right)\)$ The aggregated representation \(A_t\) is combined with the masked mean skip connection \(S_t = \frac{1}{O_t} \sum_{r=1}^{O_t} W_t^{(r)}\) through a learnable SkipGate. The aggregator accounts for only 1%โ3% of per-window processing time in streaming mode and under 1% in vectorized training, eliminating representation bloat at virtually zero overhead.
3. Three-Scale Contrastive Learning: Joint Supervision from Fine Tokens to Mid-Range Motifs
To enforce discriminative representations across all three structural levels, PMT generates weak views (channel-wise z-scaling and low-variance Gaussian noise to preserve fine details) and strong views (time-warping and magnitude-warping to disrupt short-range correlations while preserving coarse semantics), applying three complementary losses: - Fine-Grained Token Continuity (HGCL): Operates directly in token space by establishing soft-positive Gaussian weights \(q_{k,j}^t = \exp\left(-\frac{(k-j)^2}{2\sigma_p^2}\right)\) derived from window geometry rather than costly dynamic time warping (DTW), encouraging smooth local temporal continuity while treating out-of-sequence tokens as hard negatives; - Mid-Range Motif Consistency (PCL): Applies InfoNCE directly to the \(\ell_2\)-normalized memory slots emerging from the final PMA block without an intermediate projection head. Pairing corresponding memory slots at the same window index across stochastic views while treating all slots from other sequences in the batch as negatives forces memory slots to act as robust motif detectors; - Global Instance Alignment (ICL): Supervised via a standard two-layer MLP projection head on the \([CLS]\) token, contrasting global sequence representations against other batch instances.
Loss & Training¶
The total pretraining objective is a weighted combination of the three multi-scale losses: $\(\mathcal{L}_{\text{total}} = \lambda_{\text{ICL}} \mathcal{L}_{\text{ICL}} + \lambda_{\text{HGCL}} \mathcal{L}_{\text{HGCL}} + \lambda_{\text{PCL}} \mathcal{L}_{\text{PCL}}\)$ Formulations of each individual component are as follows: 1. Mid-Range PCL Loss: Symmetric InfoNCE across views 1 and 2 over valid window memory slots (sampling 512 negatives per anchor from out-of-sample sequences): $\(\mathcal{L}_{\text{PCL}} = -\mathbb{E}_{m_a} \log \frac{\exp(\langle m_a, m_p \rangle / \tau)}{\exp(\langle m_a, m_p \rangle / \tau) + \sum_{m_n \in \mathcal{N}_a} \exp(\langle m_a, m_n \rangle / \tau)}\)$ 2. Fine-Grained HGCL Loss: Weighted combination of token-within-window continuity \(\overline{\mathcal{L}_t}\) and window-across-sequence consistency \(\overline{\mathcal{L}_w}\) using bucket-form InfoNCE: $\(\mathcal{L}_{\text{HGCL}} = \alpha \overline{\mathcal{L}_t} + \beta \overline{\mathcal{L}_w}\)$ 3. Global ICL Loss: Symmetrized instance discrimination over projected \([CLS]\) representations \(g(c)\): $\(\mathcal{L}_{\text{ICL}} = -\frac{1}{\mathcal{B}} \sum_{i=1}^{\mathcal{B}} \log \frac{\exp(\langle g(c_i^1), g(c_i^2) \rangle / \tau)}{\sum_{(i', v) \neq (i, 1)} \exp(\langle g(c_i^1), g(c_{i'}^v) \rangle / \tau)}\)$ Training employs AdamW with a cosine annealing learning rate scheduler, peaking at \(1\times 10^{-4}\) after a 5% warmup and decaying to \(1\times 10^{-6}\) with batch size 256. Ablations show that uniform weighting (\(\lambda_{\text{all}} = 1\)) lands within ~2pp of exhaustively tuned per-dataset configurations, providing a robust default configuration.
Key Experimental Results¶
Main Results¶
Linear SVM probe classification results on official test splits after freezing self-supervised backbones across seven UCR/UEA/UCI benchmarks on 1% and 5% labeled subsets:
| Dataset | Label Split | PMT (Top-1 / mF1) | TS2Vec + SoftCLT | TS-TCC + SoftCLT | Relative Gain (Top-1) |
|---|---|---|---|---|---|
| HAR | 1% labels | 92.9 / 93.2 | 91.0 / 91.0 | 82.9 / 82.8 | +1.9% |
| Epilepsy | 1% labels | 96.6 / 94.7 | 96.3 / 94.1 | 95.6 / 95.6 | +0.3% |
| Wafer | 1% labels | 98.7 / 96.5 | 95.3 / 88.1 | 96.5 / 96.5 | +2.2% |
| FordA | 1% labels | 88.2 / 88.2 | 87.1 / 87.1 | 81.5 / 81.2 | +1.1% |
| FordB | 1% labels | 76.3 / 76.2 | 67.9 / 67.9 | 74.8 / 74.8 | +1.5% |
| POC | 1% labels | 68.2 / 65.7 | 63.6 / 62.8 | 65.4 / 64.6 | +2.8% |
| ElectricDevices | 1% labels | 64.1 / 58.1 | 62.0 / 53.0 | 64.6 / 63.2 | -0.5% |
| 7-Dataset Average | 1% labels | 83.6 / 81.8 | 80.5 / 77.7 | 80.2 / 79.8 | +3.1% |
| HAR | 5% labels | 95.5 / 95.9 | 92.1 / 92.1 | 92.6 / 92.6 | +2.9% |
| Wafer | 5% labels | 99.0 / 97.2 | 98.8 / 96.8 | 98.2 / 98.2 | +0.2% |
| POC | 5% labels | 72.7 / 70.6 | 70.9 / 69.7 | 69.4 / 66.3 | +1.8% |
| 7-Dataset Average | 5% labels | 85.2 / 83.6 | 84.5 / 82.7 | 85.9 / 84.2 | Top performance |
In time-series forecasting on ETTh1 and Electricity across multiple prediction horizons (\(H \in \{24, 48, 168, 336, 720\}\)), PMT attains lower average MSE across all four dataset and mode combinations compared to SoftCLT, with performance gains magnifying at longer horizons (\(H \ge 168\), e.g., multivariate Electricity MSE at \(H=720\) dropping from 0.517 to 0.421).
Ablation Study¶
1. Multi-Scale Loss Component Ablation (Averaged across 1% and 5% splits on FordA, HAR, POC, and Wafer):
| Configuration | Top-1 Accuracy (%) | Macro-F1 (%) | Note |
|---|---|---|---|
| All losses active (\(\lambda_{\text{all}} = 1\)) | 87.49 | 86.06 | Complete multi-scale framework |
| w/o ICL (\(\lambda_{\text{ICL}}=0\)) | 83.33 | 81.23 | Removing global contrast degrades Top-1 by 4.16% |
| w/o HGCL (\(\lambda_{\text{HGCL}}=0\)) | 84.69 | 82.37 | Removing token continuity drops Top-1 by 2.80% |
| w/o PCL (\(\lambda_{\text{PCL}}=0\)) | 87.33 | 86.01 | Removing mid-range contrast drops Top-1 by 0.16% |
| ICL only | 81.37 | 79.59 | -6.12% compared to full model |
| HGCL only | 82.51 | 79.17 | -4.98% compared to full model |
| PCL only | 81.58 | 79.25 | -5.91% compared to full model |
2. State Propagation & Motif Preservation: FordA Cue-Retention EOS Probe
To verify whether intermediate states truly propagate mid-range evidence rather than solving classification via localized shortcuts, Hann-tapered burst perturbations were injected into upstream windows of test sequences. Logistic regression probes evaluated whether cue evidence survived propagation to the final end-of-sequence (EOS) state:
| Model Architecture & State Type | EOS AUC | EOS Top-1 Accuracy | EOS Macro-F1 | Mechanism Note |
|---|---|---|---|---|
| PMT (Full Model) | 0.994 | 0.921 | 0.920 | Writable memory slots + PCL contrastive supervision |
| PMT (\(\lambda_{\text{PCL}}=0\)) | 0.983 | 0.831 | 0.824 | Writable memory architecture without PCL loss |
| xLSTM (Recurrent hidden state) | 0.956 | 0.820 | 0.817 | Closed recurrent state, no scale-matched loss |
| Transformer-XL (Segment cache) | 0.933 | 0.795 | 0.790 | Read-only cache concatenation |
| TS2Vec (+SoftCLT) | 0.649 | 0.638 | 0.604 | Hierarchical max-pooling degrades mid-range cue |
Key Findings¶
- Writable Memory Prevents Motif Information Loss: The cue retention probe demonstrates that hierarchical pooling architectures like TS2Vec discard intermediate perturbations (0.649 EOS AUC). In contrast, PMT's writable memory naturally preserves ranking signals (0.983 AUC), and PCL supervision sharply refines feature boundaries, boosting threshold-based Top-1 accuracy by +9.0% and Macro-F1 by +9.6%.
- Objective Complementarity: Single-loss variants lag behind the full framework by 5%โ6%, confirming that token-level continuity (HGCL), mid-range motif consistency (PCL), and global sequence alignment (ICL) serve mutually non-redundant supervisory functions.
- Horizon-Dependent Forecasting Gains: Short-horizon forecasts are adequately handled by local token representations alone, whereas long-horizon predictions (\(H \ge 168\)) benefit significantly from context accumulated in PMT's persistent memory slots.
Highlights & Insights¶
- Transforming Intermediate States into First-Class Representations: Rather than discarding hidden activations via pooling or hiding them within recurrent black-box transitions, PMT turns intermediate window states into directly supervisable representation interfaces.
- Geometry-Derived Gaussian Contrast (HGCL): Deriving soft positive weights directly from sliding window scale and stride provides continuous local temporal smoothing at minimal computation, completely avoiding the expensive quadratic dynamic time warping used in prior work.
- Lightweight Constant-Length Overlap Aggregator: Applying single-query cross-attention across overlapping window rows maintains a strict sequence length across stacked layers at only 1%โ3% processing overhead.
Limitations & Future Work¶
- Sensitivity to Tokenizer Stride: The input relies on a fixed-stride 1D convolution. Expanding stride from 10% to 100% (non-overlapping) causes a 25% drop on FordA, revealing sensitivity to patch stride on short-duration events and highlighting the need for adaptive tokenizers.
- Batch Size Dependency: Due to its reliance on InfoNCE negative mining (especially ICL), performance depends on moderately large batch sizes (256), limiting application on micro-scale datasets with only tens of sequences unless momentum queues are incorporated.
Related Work & Insights¶
- vs TS2Vec / SoftCLT: TS2Vec relies on hierarchical max-pooling that compresses local details away, while SoftCLT introduces computationally heavy DTW distance computations. PMT retains the full token stream while explicitly exposing writable memory slots supervised without DTW.
- vs Transformer-XL / xLSTM: Transformer-XL provides read-only cache concatenation without slot-level rewrites; xLSTM's recurrent state is a closed computational pipeline. PMT's memory slots are sample-specific, window-aligned, writable, and contrastively grounded as a multi-scale representation.
Rating¶
- Novelty: โญโญโญโญโญ Proposes a window-aligned writable memory mechanism and a three-scale contrastive framework for time-series.
- Experimental Thoroughness: โญโญโญโญโญ Comprehensive evaluation across 7 classification benchmarks, Hann burst cue retention probes, long-horizon forecasting, and detailed ablations.
- Writing Quality: โญโญโญโญโญ Clear motivation, rigorous receptive-field derivations, complete algorithmic formulations, and insightful qualitative analyses.
- Value: โญโญโญโญโญ Provides an architectural blueprint for multi-scale representation learning in time-series foundation models.