Skip to content

Subliminal Clocks: Latent Time Modelling in Diffusion Language Models

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Diffusion Language Models, Latent Time Representation, Mechanistic Interpretability, Mean Activation Vectors, Causal Steering

TL;DR

This paper demonstrates through probing and causal activation steering that absorbing-state Diffusion Language Models (DLMs) lacking explicit timestep conditioning spontaneously encode a continuous, ordered, and low-dimensional manifold of denoising progress (\(\tau\)) within their residual streams, allowing systematic modulation of output entropy and confidence.

Background & Motivation

Masked discrete Diffusion Language Models (DLMs, such as LLaDA and Dream) have emerged as a compelling alternative paradigm to standard autoregressive Large Language Models, demonstrating remarkable promise in controllable generation, non-autoregressive parallel decoding, and arbitrary text infilling. In contrast to conventional continuous diffusion models where a continuous timestep scalar embedding \(t\) is explicitly conditioned upon at every denoising iteration, absorbing-state DLMs receive neither a timestep embedding nor an explicit noise level input during training or inference. The network simply operates on partially masked sequences containing [MASK] tokens, iteratively recovering clean text through an unmasking policy. While extensive prior research has concentrated on enhancing sampling speed, noise schedules, or output quality, the internal representational dynamics and operational mechanics underlying the discrete denoising process remain largely uncharted.

This setup raises a fundamental mechanistic question: do diffusion models that lack explicit temporal conditioning internally represent their denoising progression? Theoretically, in absorbing discrete diffusion, the reverse denoising process only needs to restore masked positions, and the objective can be reformulated such that continuous diffusion time is closely tied to a continuous relaxation of the mask ratio. This establishes the unmasked token fraction \(\tau\) as a natural observable proxy for denoising progress. However, whether the neural network actually internalizes this sequence-level statistic into its residual stream, how causally relevant such an internal representation is to downstream generations, and what topological geometry it manifests across Transformer layers and intermediate sub-layer components have remained entirely open questions.

To resolve these questions, this paper establishes a systematic three-stage empirical investigation: first training non-linear MLP probes to decode denoising time directly from isolated token activations across layers; second, isolating global denoising directions via cross-sample mean activation vectors and applying causal activation steering during inference to modulate model entropy, confidence, and output divergence; and third, characterizing the low-dimensional manifold topology, self-correction capabilities, and opposing sub-layer geometric representations within the network depth. Core idea: Absorbing discrete diffusion language models spontaneously synthesize a structured, low-dimensional, and causally operative latent manifold of denoising progress \(\tau\) within their residual streams, continually recomputing and self-correcting this internal clock via opposing representations across self-attention and MLP blocks.

Method

Overall Architecture

The paper presents an interpretability pipeline spanning probe-based decoding, causal vector steering, and geometric characterization. Formally, for a response sequence of length \(L\) at denoising step \(t\), letting \(U_R(x_t)\) denote the unmasked token positions, empirical denoising progress is measured by \(\tau_t := |U_R(x_t)| / L \in [0, 1]\). The methodology first trains 5-layer residual MLP probes on hidden states across depth to assess the decodability of \(\tau_t\) from single-token representations. It then aggregates activations across examples into 100 denoising-progress bins to extract mean activation vectors \(\mu_{t,l}\), executing inference-time activation steering to alter the model's perception of time. Finally, Principal Component Analysis (PCA) and pairwise cross-layer/intra-layer cosine similarity metrics are applied to reveal the parabolic subspace geometry and the cross-depth dynamic error-correction loop.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["DLM Hidden States $h_{j}^{t,l}$"] --> B["Residual MLP Probe Decoding<br/>Assess decodability of progress τ from single token activations"]
    B --> C["Mean Activation Vector Extraction<br/>Average over samples to isolate clean denoising direction μ"]
    C --> D["Causal Activation Steering<br/>Swap mean vectors & perturb along low-dimensional PCA subspaces"]
    D --> E["Cross-Layer Self-Correction & Sub-Layer Geometry<br/>Measure error attenuation and anti-correlated Attention-MLP directions"]
    E --> F["Downstream Output Modulation<br/>Predictable shifts in entropy, confidence, and token probabilities"]

Key Designs

1. Residual MLP Probe Decoding: Assessing token-level encoding of sequence-level progress

To determine whether individual token hidden states internally carry sequence-level denoising progress without accessing surrounding context, the authors deploy a dedicated 5-layer residual neural network probe \(\phi_l: \mathbb{R}^d \to (0, 1)\) at each layer \(l\). Each probe module stacks Linear-LayerNorm-GELU blocks with width capped at 1024 dimensions, incorporates an internal residual skip-connection, and applies a terminal Sigmoid activation to bound predictions strictly within \((0, 1)\) matching the support of \(\tau_t\). The probes are trained exclusively on contextualized hidden states \(h_{j}^{t,l,n}\) using a mean squared regression loss: $$ \mathcal{L}{\text{MLP}} = \mathbb{E}))^2 \right] $$ To verify whether temporal awareness is localized or distributed, probes are trained across three token selections: exclusively on } \left[ (\tau_t - \phi_l(h_{j}^{t,l,n[MASK] tokens, exclusively on non-[MASK] tokens, and across all tokens simultaneously. Across all layers of LLaDA and Dream, the MLP probes achieve high determination coefficients (\(R^2 > 0.5\), reaching up to 0.95 in middle layers), confirming that latent progress information is broadly diffused across the entire residual stream rather than confined to masked positions.

2. Mean Activation Vector Extraction: Suppressing instance variance to isolate the global time signal

While high probe accuracy proves information availability, probe non-linearities might theoretically capture entangled or complex features. To extract a clean, linear direction representing pure denoising progress, the authors compute mean activation vectors across responses. Partitioning normalized denoising time into 100 discrete progress bins \(t \in [100]\), the mean activation vector for progress bin \(t\) at layer \(l\) is defined as: $$ \mu_{t,l} := \mathbb{E}{n,j} \left[ h \right] $$ Averaging across hundreds of distinct prompt-response instances and token positions cancels out sequence-specific semantics and token variance, isolating the shared systematic trajectory of denoising. This yields 3,200 mean vectors for LLaDA (32 layers) and 2,800 for Dream (28 layers). Applying the pre-trained, frozen MLP probes directly to these mean vectors (}^{t,l,n\(\phi_l(\mu_{t,l})\)) yields nearly perfect monotonic alignment with actual bin indices (Pearson correlation of 0.976 and Spearman rank correlation of 0.980 for LLaDA; 0.962 and 0.974 for Dream), proving that the mean vectors isolate the identical coherent latent direction identified by the probes.

3. Causal Activation Steering: Manipulating downstream inference via low-dimensional projection

To demonstrate that the recovered direction plays a causal role in generation rather than serving as an epiphenomenal trace, the authors intervene on activations during inference. At denoising step \(t\), activations at layer \(l\) are shifted by replacing the current bin mean with a target bin mean \(\hat{t}\): $$ \tilde{h}{j}^{t,l} := h $$ The perturbation vector }^{t,l} - \mu_{t,l} + \mu_{\hat{t},l\(\Delta_{t \to \hat{t}}^l = \mu_{\hat{t},l} - \mu_{t,l}\) naturally eliminates the time-invariant base mean \(\bar{\mu}_l := \mathbb{E}_t[\mu_{t,l}]\). To establish whether this causal effect is concentrated in a tiny subspace, \(\Delta_{t \to \hat{t}}^l\) is decomposed via PCA into the subspace \(P_{\parallel,l}\) spanned by the top-\(k\) principal components and its orthogonal complement \(P_{\perp,l}\), with both rescaled to equal norm: $$ \Delta_{\parallel,l}^{t \to \hat{t}} := P_{\parallel,l} \Delta_{t \to \hat{t}}^l \frac{|\Delta_{t \to \hat{t}}^l|}{|P_{\parallel,l} \Delta_{t \to \hat{t}}^l|}, \quad \Delta_{\perp,l}^{t \to \hat{t}} := P_{\perp,l} \Delta_{t \to \hat{t}}^l \frac{|\Delta_{t \to \hat{t}}^l|}{|P_{\perp,l} \Delta_{t \to \hat{t}}^l|} $$ Steering along the parallel subspace produces systematic behavioral shifts: steering forward (\(\hat{t} > t\)) reduces prediction entropy and increases model confidence (maximum token probability), while steering backward (\(\hat{t} < t\)) increases entropy and decreases confidence. Conversely, steering along orthogonal directions \(P_{\perp,l}\) or random Gaussian controls produces incoherent, negligible variations, proving that the top principal components govern the functional denoising clock.

4. Cross-Layer Self-Correction & Sub-Layer Geometry: Dynamic error attenuation and Attention-MLP antagonism

Analyzing interventions injected at different network depths reveals an intrinsic self-correcting computation. When perturbations are introduced in shallow layers (e.g., layer 6), tracking probe outputs across subsequent layers demonstrates that the discrepancy between steered and clean representations progressively attenuates, drifting back to the clean trajectory before reaching the output blocks. When dissecting intra-layer components, the authors measure pairwise cosine similarities across sub-layers and discover that activations from self-attention and the post-\(W_{\text{out}}\) output of the MLP exhibit an extreme anti-correlation (cosine similarity approaching -1.0). While both modules agree on the scalar distance from the temporal origin, they point in diametrically opposite geometric directions, indicating that the Transformer layers maintain and refine the time signal through an alternating counter-balancing mechanism.

Loss & Training

Discrete masked diffusion language models are optimized using weighted cross-entropy over masked positions. Under supervised fine-tuning (SFT), given prompt \(p_0\), ground-truth response \(r_0\), noise level \(s \sim \mathcal{U}(0, 1]\), and corrupted response \(r_s\) with masked index set \(\mathcal{M}(r_s) := \{j : r_j^s = \text{[MASK]}\}\), the objective minimizes: $$ \mathcal{L}{\text{SFT}} = - \mathbb{E} \log p_\theta(r_j^0 \mid p_0, r_s) \right] $$ For probe training, the AdamW optimizer is employed with a learning rate of } \left[ \frac{1}{s} \sum_{j \in \mathcal{M}(r_s)\(\alpha = 10^{-3}\) and weight decay of \(6 \times 10^{-6}\). Probes are trained for 20 epochs across 300 training sequences and 100 validation sequences. Sequence lengths \(L\) and sampling steps \(T\) are dynamically varied across epochs to ensure broad coverage across the \((0, 1)\) interval, updating gradients batched across complete rollout trajectories.

Key Experimental Results

Main Results: Probing Accuracy Across Network Depth

The performance of 5-layer residual MLP probes is benchmarked against linear probes across network layers and token subsets to evaluate the decodability of denoising progress \(\tau\).

Model Probe Architecture Token Subset Shallow Layers (Layer 3) Middle Layers (Layer 15) Deepest Layers (Layer 28/32)
LLaDA (32L) MLP Probe ALL Tokens \(R^2 = 0.87\) \(R^2 = 0.94\) \(R^2 = 0.75\)
LLaDA (32L) MLP Probe [MASK] Tokens \(R^2 = 0.86\) \(R^2 = 0.94\) \(R^2 = 0.60\)
LLaDA (32L) MLP Probe no-[MASK] Tokens \(R^2 = 0.86\) \(R^2 = 0.90\) \(R^2 = 0.70\)
LLaDA (32L) Linear Probe ALL Tokens \(R^2 = 0.89\) \(R^2 = 0.83\) \(R^2 = -0.22\)
LLaDA (32L) Linear Probe [MASK] Tokens \(R^2 = 0.93\) \(R^2 = 0.86\) \(R^2 = -0.81\)
Dream (28L) MLP Probe ALL Tokens \(R^2 = 0.98\) \(R^2 = 0.91\) \(R^2 = 0.85\)
Dream (28L) Linear Probe ALL Tokens \(R^2 = 0.96\) \(R^2 = 0.80\) \(R^2 = -0.26\)

Ablation Study: Downstream Reasoning Task Performance Under Mean-Steering

To examine whether activation steering causes degenerate catastrophic collapse or controlled shifts, accuracy was measured across reasoning and code generation tasks under diverse target steering bins \(\hat{t} \in \{1, 25, 50, 75, 100\}\) applied at Layer 29 (LLaDA) and Layer 25 (Dream).

Model Benchmark Baseline (No Steering) Steer \(\hat{t}=1\) Steer \(\hat{t}=25\) Steer \(\hat{t}=50\) Steer \(\hat{t}=75\) Steer \(\hat{t}=100\)
LLaDA-1.5 GSM8K 84.3 81.6 81.0 83.1 84.1 81.0
LLaDA-1.5 HumanEval 44.5 41.6 47.4 46.0 45.3 46.7
LLaDA-1.5 StrategyQA 69.5 63.1 69.5 66.6 65.3 69.2
Dream-7B GSM8K 68.8 51.6 61.5 59.5 56.9 23.1
Dream-7B HumanEval 35.0 28.0 32.9 35.0 29.4 23.8
Dream-7B StrategyQA 68.3 67.2 66.3 67.2 68.5 68.3

Key Findings

  • Non-Linear Representation in Deep Layers: Linear probes perform exceptionally well in early-to-mid layers (\(R^2 > 0.85\)), but completely collapse to negative coefficients in the final layers (\(R^2 = -0.22\) for LLaDA, \(-0.26\) for Dream). In stark contrast, residual MLP probes retain high accuracy (\(R^2 = 0.75\) and \(0.85\)). This demonstrates that temporal information does not vanish at the end of the network, but transforms from an easily accessible linear direction into a higher-order non-linear manifold.
  • Ultra-Low Dimensional Parabolic Manifold: Principal Component Analysis reveals that over 90% of the variance among mean activation vectors across layers is captured by fewer than 3 principal components. When plotted in 2D and 3D PCA coordinates, the time vectors trace a clean, continuous parabolic trajectory from unmasked fraction 0 to 1, demonstrating that the network allocates a compact 2D geometric arc to track diffusion time.
  • Premature EOS Collapse in Dream: At extreme steering \(\hat{t}=100\), Dream's GSM8K accuracy plummets from 68.8 to 23.1. Error inspection indicates this is triggered by premature unmasking of the [EOS] token at early spatial positions, caused by over-inflating the network's internal belief that denoising is near completion. This confirms that the internal clock is functionally coupled with token-level termination priors.

Highlights & Insights

  • Emergent Latent Clocks Without Conditioning: The paper provides clear empirical proof that neural language models trained solely on masked denoising objectives without time inputs spontaneously synthesize an internal continuous clock in their residual streams.
  • Sufficiency of 2D PCA Subspaces for Steering: Causal intervention restricted solely to the top-2 principal components (\(k=2\)) almost completely mirrors the impact of the full unprojected activation vector, while perturbations in the orthogonal subspace fail to modulate entropy or confidence, demonstrating the precise geometric concentration of the causal mechanism.
  • Self-Healing Depth Dynamics: Injected early-layer perturbations are systematically suppressed and corrected by deeper Transformer blocks, highlighting that the network continually recalibrates its internal state representations as computation flows through depth.

Limitations & Future Work

  • Scope Confined to Pure Masked Diffusion: Experiments focus exclusively on absorbing-state DLMs (LLaDA and Dream) trained via masked cross-entropy, leaving open whether similar time manifolds emerge in hybrid block-diffusion architectures or Gaussian discrete models.
  • Circuit-Level Head Identification Unresolved: While the paper documents the macroscopic anti-correlation between self-attention and MLP blocks, the microscopic attention heads and specific neuron circuits responsible for computing or reading the clock remain unidentified.
  • Inference Optimization Not Yet Realized: The findings suggest potential applications for dynamic adaptive stepping, early-exit generation, or adaptive remasking, but concrete algorithmic speedups remain to be developed in future engineering efforts.
  • vs Gurnee et al. (2025) (Counting Task Geometry): Prior work analyzed mean activation vectors to track explicit character counting in autoregressive models; this paper extends the mean vector paradigm to non-autoregressive, time-agnostic diffusion models to track implicit generative time.
  • vs Wang et al. (2026) (DLM-Scope SAE Steering): While DLM-Scope applied Sparse Autoencoders to extract and steer discrete semantic concepts, this paper isolates a global meta-cognitive variable—denoising progress—operating at a foundational architectural level.
  • vs Zhou et al. (2026) (Phase-Dependent DLM Interventions): Prior research suggested concepts are only steerable at discrete temporal stages; this paper reveals that the latent time signal \(\tau\) persists throughout the entire diffusion process and is repeatedly recalibrated across depth.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ First definitive demonstration and geometric characterization of emergent latent time representations in time-agnostic diffusion language models.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous combination of probe comparisons, causal subspace steering, layer-wise error drift analysis, and downstream benchmark ablations.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative progression connecting theoretical foundations, empirical probing, causal validation, and geometric visualization.
  • Value: ⭐⭐⭐⭐⭐ Foundational mechanistic insight that deepens theoretical comprehension of discrete diffusion and points toward adaptive non-uniform sampling schemes.