TimeTok: Granularity-Controllable Time-Series Generation via Hierarchical Tokenization¶
Conference: NeurIPS 2026
arXiv: 2605.01418
Area: Time Series
Keywords: Time-Series Generation, Granularity-Controllable Generation, Hierarchical Tokenizer, Flow Matching, Autoregressive Model
TL;DR¶
TimeTok introduces a unified framework for Granularity-Controllable Time-Series Generation (GC-TSG) by encoding time series into coarse-to-fine discrete tokens, combining flow-matching decoding with block-wise autoregressive modeling to achieve both unconditional generation and conditional refinement from coarse sketches.
Background & Motivation¶
Time-series data are inherently multiscale, naturally spanning broad temporal granularities from slow-moving macro trends to high-frequency micro fluctuations. However, in mission-critical applications such as healthcare diagnostics and financial quantitative trading, collecting high-fidelity temporal observations is often constrained by hardware costs, bandwidth limitations, and privacy boundaries. Consequently, real-world systems suffer from a severe granularity mismatch between the coarse-grained observations readily available and the high-resolution dynamic sequences required for reliable downstream analysis. For instance, clinical-grade electrocardiogram (ECG) monitors sample at up to 1000 Hz, while consumer-grade smartwatches collect signals below 60 Hz; similarly, financial market tick data are sampled in milliseconds but analyzed at task-dependent resolutions spanning daily or monthly rebalancing.
Existing time-series generative models (such as TimeGAN, TimeVQVAE, Diffusion-TS, and ImagenTime) are strictly formulated around a single fixed temporal granularity inherited from training data. They lack a native, controllable interface for answering two practical questions: given a low-fidelity observation or a user-drawn coarse trend sketch, what fine-grained trajectories are statistically plausible? And conversely, in the absence of conditions, can we synthesize signals directly at an arbitrary target temporal resolution? While modeling the full refinement trajectory directly in continuous raw space might seem conceptually viable, ancestral sampling across raw series requires conditioning on the entire history of continuous trajectories, causing extreme computational burden and severe training instability.
To overcome this dilemma, this work elevates temporal granularity from an arbitrary preprocessing nuisance into an explicit, generative control variable, formalizing the task of Granularity-Controllable Time-Series Generation (GC-TSG). The core angle of attack is to semantically align multi-scale continuous temporal dynamics with discrete token prefixes, turning token budget into an intrinsic physical dial of temporal resolution. Core idea: encode time series into an ordered sequence of discrete tokens whose prefixes strictly dictate coarse-to-fine granularities, pair them with a token-budget-conditioned flow-matching decoder, and model block-wise transitions with a VAR Transformer to unify generation from scratch and sketch refinement within a single architecture.
Method¶
Overall Architecture¶
TimeTok establishes a discrete representation where token sequence length is semantically aligned with temporal granularity, facilitating progressive autoregressive refinement in latent space. The pipeline comprises three cooperative modules: an information-reduction operator family that produces multi-scale targets during tokenizer pretraining; a hierarchical tokenizer built with patch projection, register tokens, finite scalar quantization (FSQ), and a conditional flow-matching decoder; and a VAR Transformer trained over frozen discrete tokens to predict successive granularity blocks. During inference, an automatic level detection module infers the source granularity of coarse inputs, enabling conditional coarse-to-fine decoding as well as unconditional synthesis from scratch ([BOS]).
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Time Series or Coarse Sketch<br/>(or [BOS] for unconditional generation)"] --> B["Semantically Ordered Hierarchical Tokenizer<br/>Patch Projection + Register Tokens + FSQ + Flow Matching"]
B --> C["Granularity-Controllable Inference & Detection<br/>Complexity-simplicity tradeoff selects level i via L2 steepest drop"]
C --> D["Granularity Block Autoregressive Modeling<br/>VAR Transformer predicts next-level token blocks from coarse prefix"]
D --> E["Flow Matching Decoding<br/>Continuous time series x^(j) at target granularity j"]
Key Designs¶
1. Semantically Ordered Hierarchical Tokenizer: Mapping Temporal Granularities to Variable-Length Discrete Prefixes
Conventional time-series discrete representation models (such as TOTEM or TimeVQVAE) map sequences into unordered discrete tokens with fixed lengths, where prefixes carry no monotonic semantic meaning. To enforce an explicit coarse-to-fine hierarchy, TimeTok couples register token discretization with a budget-aware conditional flow-matching decoder.
Given an input series \(\mathbf{x} \in \mathbb{R}^T\), the sequence is partitioned into \(W\) non-overlapping patches and projected into patch embeddings \(\mathbf{E} \in \mathbb{R}^{W \times h}\). The model appends \(M\) learnable register tokens \(\mathbf{R} \in \mathbb{R}^{M \times h}\) to form an augmented sequence \(\mathbf{E} \oplus \mathbf{R}\), which is processed by a Transformer encoder. The output states corresponding to the register tokens are discretized using Finite Scalar Quantization (FSQ) configured with levels \([4,4,4,4,4,4]\), yielding a 6-dimensional discrete code per token and an effective vocabulary size of \(4^6 = 4096\), producing quantized tokens \(\mathbf{Q} = [\mathbf{q}_1, \dots, \mathbf{q}_M]\).
To align token ordering with temporal granularity, the authors establish \(L=8\) granularity levels, allocating \(n_\ell = 2^{\ell-1}\) tokens for level \(\ell \in \{1,\dots,L\}\), with \(n_L = 128 = M\). The decoder is formulated via Conditional Flow Matching, targeting continuous representations smoothed by Gaussian operators \(\mathbf{x}^{(\ell)} = g_{\sigma(\ell)} * \mathbf{x}\) with bandwidth \(\sigma(\ell) = \sigma_{\max} \cdot \frac{L-\ell}{L-1}\). During training, the velocity predictor \(\mathbf{v}_\theta\) is conditioned strictly on the truncated leading \(n_\ell\) tokens \(\mathbf{Q}_{1:n_\ell}\), minimizing the coarse-to-fine flow-matching objective: $\(\mathcal{L}_{\mathrm{CTF}}(\theta)=\mathbb{E}_{\mathbf{x},\,\boldsymbol{\epsilon},\,\tau,\,\ell}\Big[\,\big\|\mathbf{v}_{\theta}(\mathbf{x}_{\tau}^{(\ell)}\!\mid\!\mathbf{Q}_{1:n_{\ell}},\tau)-(\mathbf{x}^{(\ell)}-\boldsymbol{\epsilon})\big\|_{2}^{2}\,\Big]\)$ where \(\tau \sim \mathcal{U}[0,1]\) and \(\mathbf{x}_\tau^{(\ell)} = (1-\tau)\boldsymbol{\epsilon} + \tau \mathbf{x}^{(\ell)}\). Truncating the conditioning prefix forces leading tokens to compress global macro shapes, leaving high-frequency dynamic residuals to trailing tokens.
2. Granularity Block Autoregressive Modeling: Efficient Latent Coarse-to-Fine Refinement
Directly decomposing likelihood across raw continuous trajectories \(\log p_\theta(\mathbf{x}) = \sum_{\ell=1}^L \log p_\theta(\mathbf{x}^{(\ell)} \mid \mathbf{x}^{(<\ell)})\) requires models to condition on entire sequences of previous trajectories, which is computationally expensive and prone to error accumulation. TimeTok transfers this transition process into the compact discrete token space using a VAR Transformer.
Once the tokenizer is frozen, the joint token distribution factorizes block-wise across granularity levels: $\(\log p_\phi(\mathbf{Q}_{1:n_L}) = \sum_{\ell=1}^{L} \log p_\phi(\mathbf{Q}_{n_{\ell-1}+1:n_\ell} \mid \mathbf{Q}_{1:n_{\ell-1}})\)$ Rather than predicting token-by-token, the VAR Transformer generates entire blocks of tokens \(\mathbf{Q}_{n_{\ell-1}+1:n_\ell}\) corresponding to level \(\ell\) in parallel, conditioned on all preceding coarser tokens \(\mathbf{Q}_{1:n_{\ell-1}}\). Optimization proceeds via standard cross-entropy: $\(\mathcal{L}(\phi) = -\mathbb{E}_{\mathbf{Q}}\left[\sum_{\ell=1}^L \log p_\phi(\mathbf{Q}_{n_{\ell-1}+1:n_\ell} \mid \mathbf{Q}_{1:n_{\ell-1}})\right]\)$ Operating over only \(M=128\) tokens allows TimeTok to bypass the compounding errors and slow sampling associated with continuous autoregressive modeling.
3. Granularity-Controllable Inference & Automatic Level Detection: Controllable Synthesis from Arbitrary Inputs
In real-world applications, input conditioning series frequently lack explicit granularity metadata (e.g., rough hand-drawn curves or multi-source sensor streams). TimeTok incorporates an automated level detection mechanism coupled with controllable generation.
Given an unannotated condition \(\mathbf{x}_{\mathrm{in}}\), the encoder maps it to full tokens \(\mathbf{Q}\), which are subsequently decoded at each level \(\ell \in \{1,\dots,8\}\) using prefix \(\mathbf{Q}_{1:n_\ell}\). The model computes the \(L_2\) distance between \(\mathbf{x}_{\mathrm{in}}\) and each reconstruction, selecting the level \(i\) exhibiting the steepest relative drop in reconstruction error (complexity-simplicity tradeoff). The prefix \(\mathbf{Q}_{1:n_i}\) is then supplied to the VAR Transformer to autoregressively sample missing token blocks up to target level \(j\) (\(j > i\)): $\(\hat{\mathbf{Q}}_{n_{\ell-1}+1:n_\ell} \sim p_\phi(\cdot \mid \mathbf{Q}_{1:n_i}, \hat{\mathbf{Q}}_{n_i+1:n_{\ell-1}}), \quad \ell = i+1, \dots, j\)$ Finally, the concatenated sequence \(\mathbf{Q}_{1:n_i} \oplus \hat{\mathbf{Q}}_{n_i+1:n_j}\) is passed through the flow-matching decoder to generate continuous trajectory \(\hat{\mathbf{x}}^{(j)}\). Unconditional generation corresponds to conditioning on a special [BOS] token, while setting \(j=L\) recovers standard time-series synthesis.
Loss & Training¶
TimeTok implements a modular two-stage training scheme: - Stage 1 (Tokenizer Training): Jointly optimizes the Transformer encoder, FSQ quantization grid, and flow-matching decoder. Each training step samples a granularity level \(\ell \sim \mathcal{U}\{1,\dots,L\}\), constructs \(\mathbf{x}^{(\ell)}\) using Gaussian kernel filtering, and trains \(\mathbf{v}_\theta\) via flow-matching loss \(\mathcal{L}_{\mathrm{CTF}}(\theta)\) using AdamW (learning rate \(1\times 10^{-4}\) with cosine decay). - Stage 2 (VAR Transformer Training): Freezes the tokenizer and extracts discrete tokens offline. The VAR Transformer is trained on token blocks with cross-entropy loss \(\mathcal{L}(\phi)\). Class conditioning is incorporated via Adaptive Layer Normalization (AdaLN).
Key Experimental Results¶
Main Results¶
Under standard unconditional generation, models are evaluated using FID to assess distribution matching, alongside Train-Synthetic-Test-Real (TSTR) classification accuracy/F1 or forecasting MSE.
| Dataset | Task (Length) | Metric | TimeTok (Ours) | ImagenTime (Prev. SOTA) | TOTEM | DiffusionTS | TimeVQVAE | TimeGAN |
|---|---|---|---|---|---|---|---|---|
| ECG5000 | Classification (140) | FID ↓ | 0.004±0.001 | 0.020±0.010 | 0.022±0.006 | 0.041±0.009 | 0.033±0.011 | 0.034±0.002 |
| ECG5000 | Classification (140) | ACC ↑ | 0.948±0.006 | 0.940±0.007 | 0.923±0.004 | 0.896±0.015 | 0.925±0.004 | 0.815±0.042 |
| ECG5000 | Classification (140) | F1 ↑ | 0.564±0.047 | 0.477±0.104 | 0.379±0.001 | 0.552±0.028 | 0.380±0.001 | 0.348±0.018 |
| ItalyPowerDemand | Classification (24) | FID ↓ | 0.007±0.001 | 0.043±0.006 | 0.016±0.004 | 0.023±0.005 | 0.060±0.011 | 0.026±0.008 |
| ItalyPowerDemand | Classification (24) | ACC ↑ | 0.982±0.000 | 0.978±0.002 | 0.979±0.007 | 0.979±0.003 | 0.979±0.003 | 0.958±0.003 |
| Nasdaq | Forecasting (244) | FID ↓ | 0.091±0.017 | 0.142±0.032 | 0.164±0.008 | 0.208±0.015 | 0.136±0.015 | 0.316±0.050 |
| Nasdaq | Forecasting (244) | MSE ↓ | 0.045±0.001 | 0.046±0.000 | 0.046±0.001 | 0.056±0.001 | 0.048±0.001 | 0.061±0.001 |
| ETTh1 | Forecasting (144) | FID ↓ | 0.070±0.006 | 0.074±0.007 | 0.102±0.015 | 0.083±0.011 | 0.107±0.009 | 1.068±0.243 |
| ETTh1 | Forecasting (144) | MSE ↓ | 0.103±0.000 | 0.099±0.000 | 0.102±0.001 | 0.101±0.001 | 0.102±0.001 | 0.104±0.001 |
In the Granularity-Controllable Time-Series Generation (GC-TSG) conditional refinement benchmark, models generate fine-grained samples at level \(j\) conditioned on coarse inputs at level \(i\), evaluated via \(\text{CRPS}_{\text{sum}}\) and coarse consistency (C-Cons RMSE):
| Dataset | GC-TSG Setting | Metric | TimeTok (Ours) | TimeGAN | TimeVQVAE | TOTEM | DiffusionTS | RNN (PF) |
|---|---|---|---|---|---|---|---|---|
| ECG5000 | All-Pairs Avg (\(i \to j\)) | \(\text{CRPS}_{\text{sum}}\) ↓ | 1.587±0.867 | 1.737±0.737 | 3.693±0.858 | 2.639±1.286 | 2.672±1.551 | 3.797±1.067 |
| ECG5000 | All-Pairs Avg (\(i \to j\)) | C-Cons (RMSE) ↓ | 0.010±0.006 | 0.011±0.005 | 0.044±0.013 | 0.035±0.019 | 0.031±0.019 | 0.034±0.011 |
| Nasdaq | All-Pairs Avg (\(i \to j\)) | \(\text{CRPS}_{\text{sum}}\) ↓ | 15.87±12.48 | 39.11±22.14 | 35.32±34.30 | 35.97±22.54 | 101.5±63.26 | 48.20±22.14 |
| Nasdaq | All-Pairs Avg (\(i \to j\)) | C-Cons (RMSE) ↓ | 0.096±0.090 | 0.196±0.105 | 0.282±0.258 | 0.294±0.247 | 0.771±0.538 | 0.370±0.215 |
Ablation Study¶
Table A: Discrete Token Hierarchy vs. Raw-Space AR Modeling
| Configuration | ECG5000 C2F (\(1\to 8\)) \(\text{CRPS}_{\text{sum}}\) ↓ | ECG5000 C-Cons ↓ | Nasdaq C2F (\(1\to 8\)) \(\text{CRPS}_{\text{sum}}\) ↓ | Nasdaq C-Cons ↓ | ECG5000 L8 Unconditional FID ↓ |
|---|---|---|---|---|---|
| Raw-Space AR Transformer | 4.827±1.768 | 0.027±0.014 | 32.57±25.82 | 0.160±0.136 | 3.938±0.921 |
| TimeTok (Discrete Token Space) | 3.042±0.900 | 0.011±0.007 | 17.72±12.63 | 0.089±0.085 | 0.006±0.001 |
Table B: Robustness Across Information-Reduction Operators (\(\text{CRPS}_{\text{sum}}\) in GC-TSG)
| Operator Type (\(\mathcal{R}_\ell\)) | TimeTok (ECG5000) | TimeGAN (ECG5000) | TimeVQVAE (ECG5000) | TimeTok (Nasdaq) | TimeGAN (Nasdaq) | TimeVQVAE (Nasdaq) |
|---|---|---|---|---|---|---|
| Wavelet Transform | 2.354±0.916 | 3.084±1.111 | 6.475±0.952 | 17.29±13.05 | 39.80±22.00 | 35.68±34.24 |
| Moving Average | 2.080±0.905 | 2.702±1.199 | 3.884±0.852 | 17.13±12.91 | 39.66±21.99 | 35.75±34.99 |
| Downsampling | 2.126±0.915 | 2.686±1.126 | 3.496±0.998 | 16.50±12.55 | 39.51±22.15 | 34.63±34.14 |
| Gaussian Smoothing (Training) | 1.587±0.867 | 1.737±0.737 | 3.693±0.858 | 15.87±12.48 | 39.11±22.14 | 35.32±34.30 |
Key Findings¶
- Structural Superiority of Discrete Token Latents: Directly deploying an autoregressive Transformer on concatenated 8-level raw continuous sequences fails to match distribution metrics, yielding an unconditional FID on ECG5000 of 3.938 compared to TimeTok's 0.006. This confirms that discrete hierarchical tokenization prevents compounding autoregressive errors.
- High Efficiency of Exponential Token Budgets: Comparing an exponential token allocation (\(n_\ell = 2^{\ell-1}\), starting from 1 token up to 128) against uniform binning (\(n_\ell = 16\cdot \ell\)) shows identical training loss and validation reconstruction curves, proving that coarse temporal dynamics require very few tokens to model adequately.
- Robust Out-of-Distribution Operator Generalization: Even though TimeTok is trained exclusively using Gaussian kernel filtering, it maintains state-of-the-art conditional CRPS across wavelet transforms, moving averages, and decimation downsampling, demonstrating high robustness to diverse real-world sketch inputs.
- Bridging Hardware Sampling Mismatches: Classifiers trained on hospital-grade ECG signals exhibit an accuracy collapse from 0.959 to 0.314 when fed consumer-grade downsampled signals. Feeding refined signals generated by TimeTok restores classification accuracy to 0.918.
Highlights & Insights¶
- Token Prefix Length as a Physical Granularity Dial: Instead of treating multiscale tokens merely as intermediate scaffolding for full-resolution image generation, TimeTok endows token budgets with physical temporal semantics, enabling explicit user-facing resolution control.
- Unified Modeling Across the Generative Spectrum: A single set of weights unifies unconditional generation from scratch, target-resolution generation, and conditional trajectory refinement from rough sketches.
- Foundational Tokenizer Extension via DFA: By incorporating detrended fluctuation analysis (DFA) scaling exponents to objectively bin heterogeneous datasets like UTSD into granularity levels, TimeTok demonstrates strong cross-dataset pretraining transferability.
Limitations & Future Work¶
- Operator Symmetry Assumptions: The framework relies on smoothing kernels to establish training hierarchies, which may smooth out sharp phase shifts or extreme financial jump events.
- Multivariate Cross-Channel Dependencies: Current empirical validations center on univariate or channel-independent time series; scaling to hundreds of correlated sensor channels with varying sampling rates remains to be explored.
- Inference Latency in High-Frequency Regimes: Combining block-wise Transformer sampling with numerical ODE flow-matching integration incurs higher inference latency than single-step GANs, posing challenges for sub-millisecond execution.
Related Work & Insights¶
- vs. TimeGAN / Diffusion-TS / ImagenTime: Standard time-series generators only model trajectories at a fixed training resolution; TimeTok establishes GC-TSG to allow bidirectional cross-scale conditioning and user-controllable target resolutions.
- vs. TOTEM / WaveToken / TimeVQVAE: Existing tokenizers produce unordered token sets with fixed sequence lengths; TimeTok establishes a monotonic semantic hierarchy where token count controls temporal resolution.
- vs. Vision VAR / FlexTok: While VAR uses intermediate scales primarily to boost final-resolution synthesis, TimeTok treats each granularity level as a standalone, physically meaningful temporal observation.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates the GC-TSG problem and introduces hierarchical tokenization paired with flow matching for temporal resolution control.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Thoroughly evaluates standard generation, conditional refinement, operator generalization, and foundational pretraining.
- Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical framing, clear narrative arc, and crisp motivation.
- Value: ⭐⭐⭐⭐⭐ Highly relevant for sensor alignment, multi-resolution foundation tokenizers, and user-driven sketch generation.