TimeSense: Decoupling Modality Learning for Robust Time Series Reasoning¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/NetManAIOps/TimeSense
Area: LLM Reasoning
Keywords: Time Series Reasoning, Multimodal LLM, Modality Decoupling, Temporal Reconstruction, EvalTS Benchmark
TL;DR¶
To resolve the multimodal imbalance and temporal degradation caused by text-dominated optimization in multimodal LLMs, TimeSense introduces a fusion-then-decoupling architecture that decouples temporal representations after cross-modal fusion and applies dense reconstruction supervision, substantially outperforming prior baselines across the newly introduced diagnostic benchmark EvalTS and multiple external benchmarks.
Background & Motivation¶
Time series data serve as the cornerstone for critical real-world applications across energy grids, healthcare telemetry, intelligent transportation, meteorology, and IT operations. Inspired by the breakthrough success of Vision-Language Models (VLMs), recent efforts have actively sought to extend Large Language Models (LLMs) into multimodal time series systems. Frameworks such as ChatTime, ChatTS, and ITFormer attempt to bridge sequential numerical signals with natural language instructions, aspiring to unlock zero-shot temporal perception, anomaly diagnosis, and multi-hop causal reasoning through a flexible linguistic interface.
However, existing time series multimodal large language models (MLLMs) suffer from a fundamental optimization imbalance termed unimodal bias. Because training is governed almost exclusively by autoregressive text generation objectives (cross-entropy), supervisory gradients are applied almost entirely in the discrete language space. Numerically grounded answers typically occupy only a minuscule fraction of the output token sequence, rendering the cross-entropy loss largely insensitive to subtle temporal misalignments. Consequently, textual gradients heavily overwhelm temporal gradients (\(\mathbb{E}[\|\nabla_{T_{ts}}\mathcal{L}_{txt}\|] \ll \mathbb{E}[\|\nabla_{T_{txt}}\mathcal{L}_{txt}\|]\)), and temporal gradients rapidly decay toward zero (\(\mathbb{E}[\|\nabla_{T_{ts}}\mathcal{L}_{txt}\|] \to 0\)) as soon as the model settles on language priors and surface-level phrasing templates. This text-dominated optimization forces the model to fit only sparse, task-correlated local fragments while discarding intermediate temporal transitions, subtle trend variations, and long-range dependencies essential for complex reasoning.
Compounding this architectural issue, current evaluation benchmarks such as TimeMQA, BEDTime, and TSQA remain predominantly restricted to one-dimensional sequences and constrained multiple-choice or true/false formats. When sequence segment counts or input dimensions increase, existing models suffer catastrophic performance collapses, yet existing benchmarks fail to offer diagnostic insights across progressive cognitive stages. To overcome both limitations, this paper develops EvalTS—an open-generation diagnostic benchmark spanning atomic perception, molecular comparison, and compositional reasoning—and rethinks multimodal representation learning from the ground up. Core idea: break text-dominated optimization by enabling deep cross-modal fusion within a shared Transformer backbone, subsequently decoupling the temporal hidden states to apply dense, text-independent reconstruction supervision, which theoretically eliminates the gradient-vanishing stationary points of the temporal modality.
Method¶
Overall Architecture¶
TimeSense follows an end-to-end "fusion-then-decoupling" pipeline. First, raw multivariate time series are augmented with explicit normalized coordinates, segmented into non-overlapping patches, and mapped via an MLP encoder into temporal tokens. These tokens are prepended alongside textual instruction tokens into a shared Transformer LLM backbone for unconstrained multimodal attention. After deep contextual interaction, the output hidden states corresponding to temporal tokens are explicitly decoupled from the text stream. The decoupled temporal states are routed into a dedicated, lightweight MLP reconstruction decoder trained via mean squared error (MSE) backpropagation, while the textual hidden states pass through the standard language modeling head to generate free-form reasoning outputs.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multivariate Time Series Input X in R^(D x L)"] --> B["Position-Sensitive Patch Embedding<br/>Coordinate Attachment + Patch Flatten + Shared MLP"]
C["Text Instructions and Prompts C"] --> D["Text Tokenizer and Embedding Layer"]
B --> E["Shared Transformer LLM Backbone<br/>Deep Cross-Modal Self-Attention Fusion"]
D --> E
E --> F["Modality Hidden State Decoupling<br/>Split Hidden Sequence into H_(ts) and H_(txt)"]
F -->|Temporal Stream| G["Disentangled Temporal Decoding & Dense Reconstruction<br/>Dedicated MLP Decoder + MSE Loss L_ts"]
F -->|Linguistic Stream| H["Autoregressive Text Generation & Reasoning<br/>Standard LM Head + Cross-Entropy Loss L_txt"]
G --> I["Two-Stage Coordinated Optimization<br/>Stage 1 Temporal Warm-up + Stage 2 Joint Finetuning"]
H --> I
Key Designs¶
1. Position-Sensitive Patch Embedding: Preventing permutation invariance while capturing fine-grained local dynamics
Standard projection techniques for sequence modeling often rely implicitly on transformer attention bias or naive linear projection, which frequently causes models to lose exact temporal indices over extended horizons, leading to localization jitter in sudden spikes or change points. To eliminate permutation invariance, TimeSense explicitly associates each discrete timestep \(t \in \{0, \dots, L-1\}\) with a normalized temporal coordinate \(p_t = \frac{t}{L-1}\). For channel \(d\), the scalar observation is paired with its coordinate into an augmented vector \(\tilde{x}^{(d)}_t = [x^{(d)}_t; p_t] \in \mathbb{R}^2\). The sequence is partitioned into non-overlapping patches of length \(P\), yielding flattened patch vectors \(\hat{x}^{(d)}_n = \text{vec}(\tilde{X}^{(d)}_n) \in \mathbb{R}^{2P}\). A shared learnable MLP encoder \(f_\phi: \mathbb{R}^{2P} \to \mathbb{R}^H\) projects each patch into a temporal token:
$\(T^{(d,n)}_{ts} = f_\phi(\hat{x}^{(d)}_n) \in \mathbb{R}^H\)$
Flattening across channels and patch indices produces \(T_{ts}^{\text{flat}} \in \mathbb{R}^{(DN) \times H}\), inserted into the text stream between <ts> and <ts/> delimiters. This explicit geometric anchoring ensures that token representations remain strictly sensitive to temporal shifts and exact coordinate localization.
2. Modality Hidden State Decoupling: Isolating temporal representations from single-channel linguistic erosion
During deep Transformer processing, temporal and textual representations intermingle across multi-head self-attention layers. If supervision is computed solely on autoregressive text tokens at the tail, error backpropagation tends to take linguistic shortcuts, causing the temporal hidden states to collapse into sparse representations that fire only on isolated salient cues (\(H^{(ts)}(X) \approx H^{(ts)}(\Pi_S X)\), where \(S\) denotes a sparse task-correlated coordinate subset). To prevent this erosion, TimeSense decouples the final-layer hidden sequence \(H \in \mathbb{R}^{T \times H}\) based on the predefined boundary indices of the time series tokens: $\(H^{(ts)} = H_{1:DN} \in \mathbb{R}^{(DN) \times H}, \quad H^{(txt)} = H_{DN+1:T}\)$ By isolating \(H^{(ts)}\), the framework creates an independent optimization conduit for the temporal modality, guaranteeing that rich temporal structural signals are preserved regardless of whether the prompt is sparse or the textual target is concise.
3. Disentangled Temporal Decoding & Dense Reconstruction: Provable gradient preservation against unimodal bias
To enforce that \(H^{(ts)}\) preserves globally comprehensive temporal properties rather than overfitting to sparse task fragments, TimeSense directs \(H^{(ts)}\) into a dedicated MLP decoder \(g_\psi: \mathbb{R}^H \to \mathbb{R}^P\). The decoder reconstructs each numerical patch \(\hat{X}^{(d,n)}_{rec} = g_\psi(H^{(ts)}_{(d,n)})\), and an inverse-patching operation recovers the full multivariate sequence \(\hat{X}_{rec} \in \mathbb{R}^{D \times L}\). The temporal reconstruction loss is formulated as: $\(\mathcal{L}_{ts} = \|\hat{X}_{rec} - X\|_2^2\)$ The authors formalize two foundational theoretical proofs in Appendix D. First, if a representation is subset-only (\(H^{(ts)}(x) = g(\Pi_S x)\)), any decoder Dec incurs a strictly non-vanishing minimax reconstruction error lower bound: $\(\sup_{x \in \mathcal{X}} \|\text{Dec}(H^{(ts)}(x)) - x\|_2 \ge c_S = \frac{1}{2} \sup_{x, x' \in \mathcal{X}, \Pi_S x = \Pi_S x'} \|x - x'\|_2 > 0\)$ Hence, sparse representations cannot drive the reconstruction loss to zero. Second, under the assumption of a locally non-degenerate reconstruction head (\(\sigma_{min}(J_{\text{Dec}_\psi}(u)) \ge \mu > 0\)), when the language gradient vanishes (\(\nabla_{T_{ts}}\mathcal{L}_{txt} = 0\)), the joint gradient satisfies \(\nabla_\phi \mathcal{L}_{joint} = \lambda \nabla_{T_{ts}} \mathcal{L}_{ts} \frac{\partial T_{ts}}{\partial \phi} \neq 0\). Thus, as long as reconstruction error persists, the reconstruction objective forces non-zero gradients into the temporal encoder, effectively neutralizing unimodal optimization bias.
Loss & Training¶
TimeSense employs a two-stage training scheme that establishes a solid temporal representation foundation before joint instruction alignment:
- Stage 1 Temporal Warm-up Pretraining: The LLM backbone parameters are frozen, and only the temporal encoder \(f_\phi\) and decoder \(g_\psi\) are trained for 300 steps using purely the reconstruction objective \(\min_{\phi, \psi} \mathcal{L}_{ts}\). This aligns the patch embedding manifold and prevents uncalibrated random representations from destabilizing the LLM's language space.
- Stage 2 Joint Multimodal Finetuning: The entire network is unfrozen, and the model is jointly fine-tuned for 1,200 steps using the multi-objective loss: $\(\min_{\Theta} \mathcal{L}_{joint} = \mathcal{L}_{txt} + \lambda \mathcal{L}_{ts}\)$ with the balance hyperparameter set to \(\lambda = 1.0\). Crucially, the reconstruction decoder \(g_\psi\) acts solely as a training-time regularizer and is discarded at inference time, adding zero parameters, latency, or memory overhead during deployment. All models are trained across 8 \(\times\) A800 GPUs using DeepSpeed and LLaMA-Factory with a per-device batch size of 1, gradient accumulation of 32, learning rate of \(10^{-5}\), and warmup ratio of 0.02.
Key Experimental Results¶
Main Results¶
TimeSense is evaluated across the newly constructed EvalTS benchmark (encompassing Atomic, Molecular, and Compositional levels) as well as four external time series QA datasets (TSQA-A, TSQA-B, TimeSeriesExam, and BEDTime). Under strictly controlled conditions where all SFT baselines share identical training data and step budgets:
| Model | Modality | TSQA-A (MCQ) | TSQA-B (MCQ) | TSQA-B (T/F) | TimeSeriesExam (MCQ) | TimeSeriesExam (T/F) | BEDTime (MCQ) | EvalTS (A) | EvalTS (M) | EvalTS (C) |
|---|---|---|---|---|---|---|---|---|---|---|
| GPT5 | Text | 0.587 | 0.683 | 0.838 | 0.534 | 0.691 | 0.538 | 0.695 | 0.506 | 0.351 |
| GPT5-VL | Text+Vision | 0.781 | 0.667 | 0.775 | 0.164 | 0.108 | 0.706 | 0.655 | 0.484 | 0.202 |
| Qwen2.5-Instruct | Text | 0.413 | 0.350 | 0.675 | 0.284 | 0.179 | 0.122 | 0.606 | 0.481 | 0.305 |
| Qwen2.5-VL | Text+Vision | 0.521 | 0.575 | 0.750 | 0.491 | 0.559 | 0.594 | 0.550 | 0.613 | 0.313 |
| ChatTS+SFT | Text+TS | 0.748 | 0.517 | 0.688 | 0.543 | 0.607 | 0.508 | 0.728 | 0.521 | 0.567 |
| TimeMQA+SFT | Text+TS | 0.377 | 0.625 | 0.775 | 0.000 | 0.000 | 0.173 | 0.409 | 0.286 | 0.200 |
| ITFormer+SFT | Text+TS | 0.392 | 0.433 | 0.650 | 0.388 | 0.512 | 0.305 | 0.282 | 0.356 | 0.154 |
| TimeSense (Ours) | Text+TS | 0.781 | 0.692 | 0.825 | 0.716 | 0.726 | 0.817 | 0.930 | 0.854 | 0.713 |
On EvalTS subtasks, TimeSense achieves commanding leads across both univariate and multivariate configurations: Extreme (1.000 / 1.000), Spike (0.960 / 0.900), Trend (1.000 / 0.994), Segment (0.940), and Root Cause Analysis (0.708). On Trend-based segmentation, TimeSense improves by 121% over the leading proprietary baseline GPT5 (Segment: 0.940 vs 0.345).
Ablation Study¶
The ablation experiments on EvalTS validate the critical contribution of each structural component:
| Variant Config | Atomic-Uni | Atomic-Multi | Molecular | Compositional | Notes & Insights |
|---|---|---|---|---|---|
| TimeSense (Full) | 0.945 | 0.914 | 0.854 | 0.713 | Full architecture with Stage 1 warm-up and dense reconstruction |
| w/o Stage 1 | 0.835 | 0.584 | 0.495 | 0.615 | Omitting warm-up collapses multi-channel alignment (-36.1% on Atomic-Multi) |
| w/o Rec (Reconstruction) | 0.921 | 0.799 | 0.439 | 0.566 | Removing reconstruction halves Molecular relational reasoning (-48.6%) |
| w/o All | 0.618 | 0.475 | 0.402 | 0.372 | Stripping temporal inductive bias entirely degrades to weak textual SFT |
Additional sensitivity studies reveal: 1. Reconstruction Weight \(\lambda\): Testing \(\lambda \in [0.1, 5.0]\) indicates that \(\lambda = 1.0\) strikes the optimal balance (Spike 0.960, Segment 0.940). A tiny weight (\(\lambda=0.1\)) under-supervises temporal features (Segment drops to 0.615), whereas an excessive weight (\(\lambda=5.0\)) creates competition against instruction tuning (Segment drops to 0.843). 2. Patch Length Sensitivity: A patch length of \(P=8\) delivers the best global performance (Spike 0.960, Segment 0.940, Relative 0.856, AD 0.718). Excessively small patches (\(P=4\)) impair relational reasoning (Relative plummets to 0.437), while excessively large patches (\(P=32\)) compromise fine-grained localization (Spike collapses to 0.320 and Segment to 0.199). 3. Dimensional Scalability: On the Spike localization task across dimensions \(D \in \{1, 2, 3, 4\}\), TimeSense maintains near-constant accuracy (0.960, 0.896, 0.919, 0.886), whereas baselines degrade sharply (e.g., ChatTS plummets from 0.887 to 0.290, and TimeMQA drops to 0.381).
Key Findings¶
- Stage 1 Warm-up is the keystone for multi-channel stability: Without Stage 1 pretraining, the model's accuracy on multivariate atomic tasks plummets from 0.914 to 0.584, proving that raw, unaligned temporal representations destabilize the pre-trained LLM's cross-channel attention during initial optimization.
- Dense reconstruction bridges the chasm between atomic perception and relational reasoning: Omitting reconstruction (
w/o Rec) produces only a minor drop on basic single-series tasks (0.945 \(\to\) 0.921) but cuts performance in half on Molecular tasks (0.854 \(\to\) 0.439). This highlights that while language loss alone can memorize isolated salient points, dense reconstruction is mandatory to retain intermediate transition dynamics and relational slopes across multiple segments. - Vision-based sequence rendering cannot substitute for native temporal tokens: While VLMs (GPT5-VL, Qwen2.5-VL) demonstrate reasonable coarse trend recognition, their accuracy on exact extremum localization lags significantly behind (e.g., Extreme accuracy of 0.425 and 0.297), severely constrained by image rasterization limits and coordinate quantization errors.
Highlights & Insights¶
- Train-Time Regularization with Zero Inference Overhead: The auxiliary reconstruction decoder \(g_\psi\) operates exclusively during training to propagate gradient signals into the decoupled temporal representations. At inference time, it is removed entirely, preserving standard LLM inference latency and memory footprints.
- Theoretically Proven Defense Against Unimodal Bias: The paper moves beyond empirical tinkering by providing mathematical proofs that subset-only representations suffer an irreducible reconstruction lower bound and that joint reconstruction objectives guarantee non-zero temporal gradients, formalizing the dynamics of multimodal collapse.
- Rigorous Open-Generation Evaluation Protocol: EvalTS avoids multiple-choice bias by using Qwen2.5-32B-Instruct strictly as a semantic entity parser followed by deterministic rule scripts for scoring, achieving a 99.8% (499/500) human-audit agreement rate.
Limitations & Future Work¶
- Synthetic Gap vs. Real-World Industrial Noise: Although ChronGen enables controlled, fine-grained diagnostic variations, synthetic series do not fully capture non-uniform sampling frequencies, extended missing-data intervals, sensor packet drops, and real-world non-stationary dynamics.
- Architectural Scope Limited to a Single Family: The primary experiments center on the Qwen2.5-14B-Instruct backbone. Evaluating scalability to larger foundations (e.g., 70B+ parameters) and alternative sequence backbones (such as state-space models like Mamba) remains an open challenge.
- Absence of Calibrated Uncertainty in High-Stakes Deployment: In safety-critical sectors like clinical patient monitoring or power grid fault clearing, uncalibrated LLM hallucinations remain hazardous. Incorporating conformal prediction and strict uncertainty calibration is necessary prior to actual industrial deployment.
Related Work & Insights¶
- vs. ChatTS (Xie et al., 2025): ChatTS relies on synthetic textual descriptions to align time series attributes with language tokens, but its temporal representations remain subservient to the text-generation loss. TimeSense introduces post-fusion decoupling and explicit temporal reconstruction, widening the performance gap substantially on complex molecular reasoning (EvalTS Molecular: 0.854 vs 0.402).
- vs. ChatTime / ITFormer (Wang et al., 2025a,b): Earlier works treat time series as an isolated foreign dialect or rely on frozen time encoders prepended to frozen LLMs, sacrificing bi-directional cross-modal interaction. TimeSense achieves deep attention fusion while preventing multimodal collapse via auxiliary reconstruction.
- Cross-Domain Takeaway: In multimodal systems where one modality (text) overwhelmingly dominates supervisory gradients, post-fusion representation decoupling combined with dense autoencoding reconstruction provides an elegant, domain-agnostic mechanism to prevent perceptual representation collapse.
Rating¶
- Novelty: ⭐⭐⭐⭐ [Introduces an elegant fusion-then-decoupling framework with dense temporal reconstruction to eliminate unimodal optimization bias, backed by theoretical proofs]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Introduces the comprehensive 3-tier EvalTS benchmark, tests across 4 external datasets, and includes extensive ablations on dimensions, sequence lengths, and hyperparameters]
- Writing Quality: ⭐⭐⭐⭐⭐ [Exceptional narrative clarity, rigorous mathematical proofs, clean diagrammatic workflows, and well-structured empirical analyses]
- Value: ⭐⭐⭐⭐⭐ [Provides a definitive blueprint for resolving representation collapse in time series MLLMs and releases an open-generation diagnostic benchmark for future temporal reasoning research]