Observable Patterns Are Not Explanations: A Causal-Geometric Analysis of Latent Reasoning Models¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/darpanaswal/InterpretingLatents
Area: Interpretability
Keywords: latent reasoning models, mechanistic interpretability, causal interventions, gradient subspace, geometric analysis
TL;DR¶
Challenging the widespread assumption that observable latent-state patterns (such as BFS tree search and scratchpad computation) constitute evidence for internal reasoning mechanisms, this paper uses matched controls, causal tracing, and gradient-subspace interventions to show that these patterns are epiphenomena rather than mechanisms, proving that latent-thought influence is graded and concentrated within low-dimensional geometric subspaces.
Background & Motivation¶
Chain-of-thought (CoT) prompting has become the standard paradigm for eliciting reasoning in large language models, yet it incurs severe inference computation costs and is fundamentally constrained by the discreteness of natural-language token generation. To address these bottlenecks, latent reasoning models (LRMs, such as COCONUT and CODI) perform intermediate reasoning in continuous hidden representations by passing recurrent states or continuous soft tokens. However, bypassing explicit natural-language generation also removes the primary surface artifact available for monitoring, auditing, and safety oversight, turning internal reasoning dynamics into an opaque black box.
Early mechanistic interpretability efforts attempted to decode latent thoughts using observational probes, interpreting observable features as proof of internal reasoning algorithms. For instance, logit-lens and depth probes were taken as evidence that models execute breadth-first search (BFS) over superposed reasoning candidates, or maintain an ordered scratchpad storing operands and intermediate results. Yet, these observational findings face severe methodological vulnerabilities: do these observed patterns uniquely emerge from the proposed recurrent or curriculum mechanisms, or do they appear identically in non-LRM controls? Furthermore, does an observable geometric pattern actually exert a causal effect on downstream predictions, or is it merely an epiphenomenon correlated with successful execution? Without resolving these questions, interpretability risks suffering from the "dead salmon effect," mistaking probe-induced visual artifacts for true model algorithms.
To move beyond superficial observational attributions, this paper advocates treating latent thoughts not as "hidden explanations" but as "extended hidden computation states" whose functional role must be grounded in causal influence on behavior. Core idea: Observable patterns in latent space do not establish reasoning mechanisms; claims of internal reasoning require matched non-recurrent controls and causal interventions, showing that latent-thought influence is a graded quantity concentrated within low-dimensional loss-sensitive gradient subspaces whose dynamics track behavioral impact.
Method¶
Overall Architecture¶
The paper introduces an end-to-end framework integrating matched control baselines, causal tracing, gradient-subspace localization, and geometric dynamical analysis. First, it constructs strictly matched controls lacking recurrence (Pause-as-thought) or curriculum progression (\(\text{COCONUT}_u\)), alongside standard non-latent baselines, to test whether observable signatures uniquely stem from LRM architectures. Second, it performs test-time thought ablation and per-position causal tracing to measure the behavioural necessity and sufficiency of latent tokens. Third, it isolates the loss-sensitive directions via singular value decomposition on the gradient matrix, scaling these subspaces to probe graded causal influence. Finally, it analyzes the dynamical evolution and principal-angle stability of both full latent representations and causally active subspaces across reasoning steps.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input prompt & latent sequence<br/>(LRMs & matched control baselines)"] --> B["Stage 1: Observational Pattern Probing<br/>BFS frontier probing & Logit-Lens decoding"]
B --> C["Stage 2: Causal Effect Measurement<br/>Test-time ablation & per-position causal tracing"]
C --> D["Stage 3: Gradient-Subspace Localization<br/>Loss-gradient SVD extraction & alpha-scaling"]
D --> E["Stage 4: Dynamics & Geometric Stability<br/>Markov dynamical modeling & principal angle cosine"]
E --> F["Causal-Geometric Conclusion:<br/>Patterns are not explanations; influence is low-dimensional & dynamic"]
Key Designs¶
1. Matched Control Baseline Suite: Disentangling Architecture from Probing Artifacts To isolate whether observable patterns genuinely emerge from latent recurrence or curriculum design, the authors construct rigorous control models evaluated on the same tasks. Alongside target LRMs (COCONUT, which uses curriculum replacement of tokens with latents, and CODI, which distills rationales into latents), two primary controls are introduced: Pause-as-thought (PaT), which preserves the format and training curriculum of COCONUT but replaces recursive state-passing with static learned pause tokens (eliminating internal recurrence); and \(\text{COCONUT}_u\), which perturbs COCONUT's staged curriculum by uniformly sampling stages with probability \(u=0.3\). In addition, Base GPT-2 and Explicit-CoT GPT-2 serve as observational controls lacking latent training. If BFS or scratchpad patterns were truly driven by latent recurrence, PaT should fail to exhibit them while \(\text{COCONUT}_u\) should retain them.
2. Per-Position Causal Tracing: Dissecting Behavioral Influence Across Sites To evaluate whether latent thoughts causally drive model predictions or are bypassed by prompt shortcuts, the authors extend causal tracing to latent reasoning pipelines. Given residual stream activation site \(s = (\ell, p)\) at layer \(\ell\) and sequence position \(p\), three forward passes are executed: a clean pass, a corrupted pass (\(cp\)) run on a partner prompt yielding an alternative target answer, and a patched pass (\(pp\)) injecting the clean activation at site \(s\) into the corrupted run. At the answer boundary, greedy token generation is evaluated against the clean distribution via forward Kullback-Leibler divergence over content window \(W\): $\(IE(s) = 1 - \frac{KL(P_{\text{clean}} \parallel P_{pp})}{KL(P_{\text{clean}} \parallel P_{cp})}\)$ where the indirect effect \(IE(s) \le 1\) quantifies behavioral recovery (1 being full recovery, 0 being no effect). By evaluating prompt positions, individual thought tokens \(T_1, \dots, T_K\), all-thought block \(T_{\text{full}}\), and answer boundaries, this method decouples true computational influence from superficial representation presence.
3. Gradient-Subspace Localization: Uncovering Graded Non-Binary Influence Rather than treating thought utilization as a binary shortcut-versus-reasoning dichotomy, this design isolates the low-dimensional directions within the latent state most sensitive to the final task loss. For instance \(i\) at thought step \(t\) with latent vector \(h_{i,t} \in \mathbb{R}^D\) and loss \(L_i\), the gradient \(g_{i,t} = \nabla_{h_{i,t}} L_i\) is computed. Singular value decomposition on the gradient matrix \(G_t = [g_{1,t}; \dots; g_{N,t}]\) yields right singular vectors capturing 95% cumulative energy, forming orthonormal basis \(B_t \in \mathbb{R}^{D \times k_t}\). The latent state is decomposed into the gradient projection and orthogonal complement, scaled by intervention factor \(\alpha \in \{0, 0.5, 1, 1.5, 2, 5, 10, 25, 50, 100\}\): $\(h_{i,t} \leftarrow h_{i,t} + (\alpha - 1) B_t B_t^\top h_{i,t}\)$ Setting \(\alpha = 0\) performs targeted subspace ablation, while \(\alpha > 1\) amplifies the gradient component, benchmarked against a rank-matched random orthonormal control \(B_t^{\text{rand}}\) generated via QR decomposition on Gaussian noise. Measuring prediction flip rates reveals that latent influence behaves as a graded, steerable continuum rather than an all-or-nothing switch.
4. Dynamical Markov Modeling and Geometric Subspace Stability To understand how causally influential latent trajectories evolve across discrete computation steps, the authors model step-wise transitions and geometric persistence. On both full latent representations and gradient-projected states \(h_{i,t}^{\parallel} = B_t B_t^\top h_{i,t}\), pairs \(X_{i,t} = [h_{i,t-1}; \dots; h_{i,t-n}]\) and \(Y_{i,t} = h_{i,t}\) are formed for Markov orders \(n \in \{1, \dots, 5\}\). Shared transition functions under ridge regression and a two-layer MLP with GELU activations are fitted on training splits, comparing test \(R^2\) against an identity baseline (\(\hat{h}_t = h_{t-1}\)) and a training mean baseline. Furthermore, geometric alignment across adjacent time steps is quantified via the mean squared principal angles: $\(s_t = \text{mean} \, \sigma(B_t^\top B_{t+1})^2\)$ where \(s_t \to 1\) indicates static, invariant causal directions, while \(s_t \to 0\) signifies orthogonal step-wise rotations. This links the geometric stability of the causal subspace directly to behavioral influence.
Key Experimental Results¶
Main Results¶
The framework is evaluated on multi-step graph navigation (ProsQA) and multi-step arithmetic reasoning (GSM8K) across GPT-2-small and Llama-3.2-1B-Instruct backbones, testing performance under full configuration versus complete thought removal.
| Task / Setting | Model Family | Full Model Accuracy (%) | Thoughts Removed Accuracy (%) | Accuracy Shift (%) |
|---|---|---|---|---|
| Graph-Hopping (GPT-2) | Base GPT-2 (B) | 2.4 ± 1.3 | – | – |
| Explicit-CoT (CoT) | 79.0 ± 3.6 | – | – | |
| Pause-as-thought (PaT) | 95.4 ± 1.8 | 95.6 ± 1.7 | +0.2 | |
| COCONUT (C) | 98.0 ± 1.3 | 97.8 ± 1.3 | -0.2 | |
| \(\text{COCONUT}_u\) (Cu) | 96.0 ± 1.7 | 91.8 ± 2.4 | -4.2 | |
| CODI | 80.0 ± 3.5 | 80.0 ± 3.5 | 0.0 | |
| Arithmetic-Reasoning (GPT-2) | Base GPT-2 (B) | 1.4 ± 0.6 | – | – |
| Explicit-CoT (CoT) | 41.9 ± 2.7 | – | – | |
| Pause-as-thought (PaT) | 26.4 ± 2.4 | 21.4 ± 2.2 | -5.0 | |
| COCONUT (C) | 35.7 ± 2.5 | 7.7 ± 1.4 | -28.0 | |
| \(\text{COCONUT}_u\) (Cu) | 30.8 ± 2.5 | 39.1 ± 2.6 | +8.3 | |
| CODI | 41.8 ± 2.7 | 25.6 ± 2.4 | -16.2 | |
| Arithmetic-Reasoning (Llama-3.2) | Pause-as-thought (PaT) | 26.8 ± 2.4 | 3.9 ± 1.0 | -22.9 |
| COCONUT (C) | 33.2 ± 2.4 | 12.1 ± 1.7 | -21.1 | |
| \(\text{COCONUT}_u\) (Cu) | 51.7 ± 2.6 | 49.4 ± 2.7 | -2.3 | |
| CODI | 44.6 ± 2.6 | 23.5 ± 2.3 | -21.1 |
Gradient-Subspace Diagnostics and Geometric Stability¶
| Task | Model | Mean Rank \(\bar{k}\) (95% Energy) | Adjacent Stability \(\text{Adj.} \cos^2\) | Off-diagonal Stability \(\text{Off-diag.} \cos^2\) | Norm Ratio \(\|h_c\|/\|h\|\) |
|---|---|---|---|---|---|
| Graph-Hopping (GPT-2) | PaT | 16.3 | 0.871 | 0.697 ± 0.026 | 0.353 |
| COCONUT (C) | 11.8 | 0.972 | 0.671 ± 0.024 | 0.026 | |
| \(\text{COCONUT}_u\) (Cu) | 15.5 | 0.486 | 0.197 ± 0.012 | 0.045 | |
| CODI | 32.8 | 0.898 | 0.587 ± 0.008 | 0.100 | |
| Arithmetic-Reasoning (GPT-2) | PaT | 141.6 | 0.711 | 0.623 ± 0.019 | 0.378 |
| COCONUT (C) | 127.0 | 0.512 | 0.394 ± 0.021 | 0.369 | |
| \(\text{COCONUT}_u\) (Cu) | 121.7 | 0.574 | 0.404 ± 0.014 | 0.297 | |
| CODI | 36.2 | 0.271 | 0.351 ± 0.016 | 0.306 | |
| Graph-Hopping (Llama-3.2) | COCONUT (C) | 2.0 | 0.863 | 0.469 ± 0.020 | 0.022 |
| Arithmetic-Reasoning (Llama-3.2) | COCONUT (C) | 302.5 | 0.934 | 0.590 ± 0.004 | 0.574 |
Key Findings¶
- Observable Patterns Replicate Without Recurrence: On graph-hopping, PaT (completely lacking recurrence) replicates COCONUT's BFS exploration profile (rising \(P(\text{correct})\) and falling entropy) just as strongly, while perturbing the curriculum in \(\text{COCONUT}_u\) inverts this pattern despite keeping recurrence. On arithmetic, standard CoT GPT-2 without latent training produces CODI-like step-wise logit-lens alternations, proving that observational signatures do not indicate unique LRM mechanisms.
- Causal Necessity is Task-Dependent: In graph-hopping, removing all latent thoughts leaves accuracy virtually unchanged for COCONUT (98.0% vs 97.8%), PaT, and CODI, with prompt tokens accounting for almost all causal recovery in causal tracing. In arithmetic reasoning, removing thoughts induces a severe collapse for COCONUT (dropping 28.0% on GPT-2 and 21.1% on Llama-3.2), showing that latent computation necessity is dictated by task requirements.
- Causal Effect Concentrates in Low-Rank Directions: In graph-hopping, while test-time ablation indicates no global accuracy drop, gradient-subspace scaling (\(\alpha \ge 50\)) induces near 100% flip rates compared to ~1% for random directions. In arithmetic reasoning, targeted ablation (\(\alpha = 0\)) of the gradient subspace degrades performance sharply, proving that behavioral influence resides in a compact, steerable subspace.
- Dynamic Evolution Underneath Static Latents: On graph-hopping, full latent representations appear static across steps (dominated by the identity copy baseline), but their projection into the gradient subspace exhibits structured dynamics well-captured by MLPs. On arithmetic, both the full state and the causal subspace rotate dynamically across steps (adjacent stability dropping to 0.27–0.57).
Highlights & Insights¶
- Methodological Wake-Up Call: Demonstrates that finding decodable tokens or geometrically appealing patterns in latent states does not imply that the model utilizes those patterns to compute answers, warning against false interpretability.
- Reframing the Shortcut Dichotomy: Shows that thought usage is not an all-or-nothing binary state; models possess overdetermined alternative circuits, and latent states exert a graded influence that can be localized and steered along specific gradient directions.
- Actionable Geometric Targets: By isolating the low-dimensional subspace responsible for causal behavioral effects, the paper provides concrete geometric targets for latent model distillation, compression, and alignment monitoring.
Limitations & Future Work¶
- Linear Subspace Assumption: The gradient subspace is extracted linearly via SVD; non-linear manifolds or sparse features (such as sparse autoencoders or dictionary learning) might reveal richer structure flattened by linear bases.
- Off-Distribution Intervention Artifacts: Large-scale activation patching and \(\alpha\)-scaling inevitably push activations off-manifold, which can induce secondary behavioral shifts.
- Scale and Multi-Task Scope: Experiments are evaluated on smaller models (GPT-2, Llama-3.2-1B) on single-task checkpoints with fixed length (\(K=6\)); scaling to frontier reasoning architectures and joint multi-task training remains open.
Related Work & Insights¶
- vs COCONUT (Hao et al., 2025): COCONUT interpreted entropy and candidate-mass dynamics as breadth-first search across superposed paths; this paper shows that non-recurrent PaT controls replicate these trends and test-time thought ablation leaves accuracy intact, disproving that BFS dynamics explain task success.
- vs CODI (Shen et al., 2025): CODI claimed self-distillation produces an internal computational scratchpad; this paper reveals that explicit-CoT baselines without latent training exhibit similar logit-lens alternations, indicating probe artifacts rather than a unique scratchpad mechanism.
- vs Binary Causal Analysis (Zhang et al., 2025a; Cui et al., 2026): While previous work treated latent tokens as either used or bypassed, this work demonstrates that causal influence is continuous, steerable, and localized within compact gradient subspaces.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Establishes a rigorous causal-first framework that dismantles widespread observational assumptions in latent reasoning models.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Exceptionally rigorous with multi-level matched controls, causal tracing, subspace interventions, and cross-model validation on GPT-2 and Llama-3.2.
- Writing Quality: ⭐⭐⭐⭐⭐ Beautifully structured, philosophically sound, and exceptionally clear in delineating observation from causation.
- Value: ⭐⭐⭐⭐⭐ Foundational contribution to mechanistic interpretability and safety oversight of continuous-thought architectures.