Skip to content

Thinking Without Speaking: Efficient Tree of Thoughts Reasoning via Latent Steering

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/xiaoshumiao123/ITOT
Area: LLM Reasoning
Keywords: latent steering, Tree of Thoughts (ToT), test-time compute, contrastive prototype, training-free reasoning

TL;DR

Addressing the severe latency and token blow-up of Tree of Thoughts (ToT) under the generate-then-score paradigm, this paper presents Implicit Tree of Thoughts (IToT)—a training-free framework that reframes discrete tree search as continuous optimization in latent hidden representations, steering reasoning paths via contrastive prototypes and CMA-ES to outperform explicit search baselines while operating below the token budget of vanilla Chain of Thought.

Background & Motivation

Tree of Thoughts (ToT) and its graph-structured extensions empower large language models to conquer complex mathematical reasoning, formal logic, and multi-hop planning tasks by exploring branched decision spaces beyond standard autoregressive generation. However, deploying tree-structured reasoning in real-world or latency-sensitive applications remains severely bottlenecked by extreme inference overhead. The architectural culprit is the decoupled "generate-then-score" paradigm: at every expansion node of the reasoning tree, the language model must explicitly decode multiple candidate thoughts into natural language text, followed by invoking external reward models, step-level process verifiers, or prompt-based self-evaluation routines to assess branch viability. Even modern efficiency optimizations—such as dynamic branch pruning, constrained generation, or state summarization—merely reduce the branch factor while leaving the prohibitive per-node textual decoding and evaluation latency completely intact.

The core tension lies in the mismatch between discrete search mechanisms and autoregressive representations. Explicit tree exploration aims to mitigate early hallucinations and suboptimal trajectory traps by exploring diverse solution hypotheses; however, emitting exhaustive candidate texts consumes thousands of superfluous tokens. Concurrently, mechanistic interpretability studies reveal that deep hidden representations in pretrained Transformers inherently capture abstract semantic validity and geometric signatures that systematically distinguish successful reasoning paths from confused, erroneous ones. Forcing the model to verbalize every transient exploratory dead-end into tokens before judging its quality introduces enormous computational redundancy when the internal activations already manifest intermediate correctness signals during forward propagation.

This work bypasses discrete textual generation altogether by internalizing the entire tree exploration, evaluation, and pruning loop into the continuous hidden-state space of the language model. Core idea: recast tree-structured reasoning as continuous latent optimization, where decoding windows natively dictate search depth, directional hidden-state perturbations act as alternative branches, and cosine alignment with offline contrastive prototypes enables intrinsic evaluation and beam steering without emitting intermediate candidate tokens.

Method

Overall Architecture

IToT operates across two coordinated phases: Offline Prototype Calibration and Inference-Time Latent Steering. In the offline phase, the framework queries the frozen base model on a tiny set of paired calibration prompts to produce correct reference solutions and high-temperature confused derivations, deriving a layer-wise contrastive prototype vector that captures the geometric direction from failure to success while canceling out query-specific lexical semantics. During online generation, autoregressive decoding is partitioned into fixed token windows representing search depth levels. At each controlled upper Transformer layer, a pool of candidate steering directions (branches) is maintained. Within each window, the top-ranked latent branch is smoothly injected into post-attention hidden activations via soft convex mixing. At window boundaries, Covariance Matrix Adaptation Evolution Strategy (CMA-ES) refines candidate directions against the contrastive prototype, followed by momentum smoothing and latent beam pruning. Consequently, evaluation and branch selection collapse directly into forward-pass computation, producing only the final, verified reasoning path in text.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input prompt & offline paired traces"] --> B["Stage 1: Offline prototype calibration<br/>Compute normalized residual vectors across upper layers"]
    B --> C["Stage 2: Windowed autoregressive decoding<br/>Partition token sequence into windows of size W"]
    C --> D["Stage 3: Continuous soft latent steering<br/>Inject highest-value branch direction into post-attention activations"]
    D --> E["Stage 4: CMA-ES latent branch expansion<br/>Derivative-free optimization guided by prototype cosine similarity"]
    E --> F["Stage 5: Momentum tracking & latent beam pruning<br/>Update directional history and retain Top-K steering candidates"]
    F -->|EOS not reached| C
    F -->|EOS reached| G["Output final clean reasoning text"]

Key Designs

1. Offline Prototype Calibration: Distilling a Geometric Compass of Logical Correctness

To eliminate explicit reward models and prompt-based text judging, this design extracts an intrinsic geometric quality proxy directly from the model's upper-layer representations. For a decoder-only LLM with \(L\) layers, interventions are restricted to upper Transformer blocks \(\mathcal{L}_{\text{high}} = \{l_{\text{start}}, \dots, L\}\) which govern high-level reasoning rather than lexical processing. Given a minimal calibration set \(\mathcal{D}_{\text{cal}} = \{(x_i, y_i^+, y_i^-)\}_{i=1}^N\), reference solutions \(y_i^+\) serve as successful traces, while failed traces \(y_i^-\) are synthesized by prompting the frozen model under high-temperature stochastic decoding on the identical input prompt \(x_i\). Because both traces share identical question context, the semantic topics cancel out when contrasting their terminal hidden states \(z^{(l)}(x_i, y) = h_T^{(l)}\). The layer-wise contrastive prototype is computed and \(\ell_2\)-normalized across the calibration instances: $\(p^{(l)} = \text{Norm}\left( \frac{1}{N} \sum_{i=1}^N \left( z^{(l)}(x_i, y_i^+) - z^{(l)}(x_i, y_i^-) \right) \right)\)$ This frozen prototype vector \(p^{(l)}\) serves as a lightweight, zero-token proxy during inference, enabling direct cosine alignment scoring in latent space.

2. Periodic Latent Branching and Selection: Mapping Discrete Search into Continuous Evolution

To avoid the combinatorial token blow-up of discrete candidate branching, IToT formulates branch expansion and selection as continuous derivative-free optimization. Autoregressive decoding is segmented into windows of \(W\) tokens. For each controlled layer \(l \in \mathcal{L}_{\text{high}}\), a pool of \(B\) latent branches is maintained: \(\mathcal{B}^{(l)} = \{(m_b^{(l)}, v_b^{(l)})\}_{b=1}^B\), where \(m_b^{(l)} \in \mathbb{R}^d\) is a unit steering direction and \(v_b^{(l)} \in \mathbb{R}\) tracks accumulated quality. At the conclusion of each window, CMA-ES performs local derivative-free refinement around current momentum vectors \(m_b^{(l)}\), sampling candidates from a Gaussian distribution and minimizing the negative cosine alignment with the prototype: $\(J^{(l)}(z_i) = - \cos(z_i, p^{(l)}) = - \frac{z_i}{\|z_i\|_2} \cdot p^{(l)}\)$ The best mutated candidate \(d_b^{(l)}\) updates the branch direction via momentum parameter \(\beta \in [0, 1]\) to stabilize optimization across extended reasoning horizons, while accumulating alignment score: $\(m_b^{(l)} \leftarrow \text{Norm}(\beta m_b^{(l)} + (1-\beta) d_b^{(l)}), \quad v_b^{(l)} \leftarrow v_b^{(l)} + \cos(d_b^{(l)}, p^{(l)})\)$ Branches are subsequently ranked by accumulated value \(v_b^{(l)}\), and only the top \(K\) branches (Beam Width \(B_w = K\)) survive into the subsequent window, reproducing ToT beam pruning entirely within representations.

3. Continuous Latent Steering: Preserving Language Fluency via Soft Convex Injection

Aggressive hidden-state overwriting easily derails the foundational syntax and linguistic fluency of autoregressive models. To guarantee intervention stability, IToT applies a continuous, norm-scaled soft convex steering mechanism. Between window update intervals, the model decodes autoregressively using the highest-scoring branch \(b^*(l) = \arg\max_b v_b^{(l)}\) at each layer. Interventions target the post-attention, layer-normalized representation \(h_t^{(l)}\). The unit steering direction is scaled dynamically by the \(\ell_2\) norm of the current token's hidden state to prevent manifold departure: $\(\tilde{h}_t^{(l)} = h_t^{(l)} + \lambda^{(l)} \|h_t^{(l)}\|_2 m_{b^*(l)}^{(l)}\)$ where \(\lambda^{(l)}\) specifies the layer-dependent steering intensity. Rather than replacing \(h_t^{(l)}\), the representation undergoes convex blending with mixing coefficient \(\alpha \in [0, 1]\): $\(h_t'^{(l)} = (1 - \alpha) h_t^{(l)} + \alpha \tilde{h}_t^{(l)}\)$ The adjusted representation \(h_t'^{(l)}\) propagates forward through subsequent blocks, continuously anchoring generation along the valid reasoning trajectory without generating a single auxiliary judgment token.

Loss & Training

IToT is strictly training-free and operates entirely at inference time with frozen weights. Optimization is executed on-the-fly via CMA-ES over directional vectors. The controlled layers \(\mathcal{L}_{\text{high}}\) encompass the final 8 Transformer blocks with linearly scaled intervention coefficients \(\lambda^{(l)} \in \{0.01, 0.02, \dots, 0.08\}\). Soft mixture coefficient \(\alpha = 0.5\). Offline calibration requires only \(N = 10\) paired examples. Standard inference hyperparameters include branch factor \(B_f = 4\), beam width \(B_w = 2\), update window size \(W = 128\), CMA-ES population size \(\lambda = 8\), generation iterations \(G = 4\), and momentum decay \(\beta = 0.9\).

Key Experimental Results

Main Results

Evaluation spans five diverse open-source LLMs (Qwen2.5-1.5B/3B-Instruct, DeepSeek-R1-Distill-Qwen-1.5B, Qwen2.5-Math-1.5B-Instruct, and Gemma-3-1B-IT) across six benchmarks encompassing mathematics (GSM8K, MATH500), commonsense reasoning (CommonsenseQA), multi-step logical deduction (ProsQA), general knowledge (MMLU), and reading comprehension (Belebele). Performance is measured via \(P@5\) (pass@5 across five generations), runtime per sample (seconds), and total token consumption.

Dataset Metric IToT (Ours) ToT (Tree Search) CoT (Single Chain) Gain (vs ToT / CoT)
GSM8K Accuracy / Runtime(s) / Tokens 0.853 / 46.06 / 2,509.89 0.745 / 46.63 / 22,926.30 0.871 / 52.37 / 2,357.34 +10.8 pts (vs ToT) / 89.1% token cut
MATH500 Accuracy / Runtime(s) / Tokens 0.701 / 69.80 / 3,358.39 0.636 / 173.10 / 30,446.82 0.620 / 80.89 / 3,294.66 +6.5 pts / +8.1 pts / 89.0% token cut
CommonsenseQA Accuracy / Runtime(s) / Tokens 0.757 / 38.90 / 1,874.71 0.699 / 55.49 / 6,733.52 0.734 / 47.20 / 2,453.71 +5.8 pts / +2.3 pts / 72.2% token cut
Belebele Accuracy / Runtime(s) / Tokens 0.775 / 52.05 / 1,544.50 0.719 / 71.31 / 11,576.48 0.763 / 33.50 / 1,517.68 +5.6 pts / +1.2 pts / 86.7% token cut
ProsQA Accuracy / Runtime(s) / Tokens 0.812 / 90.03 / 3,413.70 0.469 / 178.80 / 41,473.18 0.211 / 93.37 / 4,057.96 +34.3 pts / +60.1 pts / 91.8% token cut
MMLU Accuracy / Runtime(s) / Tokens 0.689 / 47.79 / 2,289.79 0.664 / 73.58 / 11,583.82 0.668 / 79.16 / 3,144.92 +2.5 pts / +2.1 pts / 80.2% token cut

Ablation Study

The ablation analysis conducted on GSM8K under standard parameters (\(T_{\max}=1024, W=128, B_f=4, B_w=2\)) isolates the contribution of each core design component:

Configuration Accuracy (Acc. ↑) Tokens / sample (↓) Time / sample (s ↓) Note & Core Takeaway
IToT (Full system) 0.871 301.01 5.358 Complete default framework achieving peak performance
w/o Tree 0.859 300.13 3.777 Eliminating multi-branch latent exploration drops accuracy by 1.2 pts
w/o Steering 0.870 298.78 5.841 Disabling latent guidance degrades accuracy and increases runtime
w/o Pruning 0.867 299.20 7.454 Retaining all unpruned branches inflates runtime by 39.1% without accuracy gain
w/o CMA 0.868 298.91 4.121 Removing derivative-free optimization impairs alignment with prototypes
w/o Momentum 0.867 300.86 5.378 Discarding historical update momentum causes directional trajectory instability
Single-layer 0.866 297.92 5.694 Intervening on only one layer weakens multi-layer semantic steering depth
w/o Calibration 0.866 299.22 5.910 Random or uncalibrated directions fail to provide a constructive logical gradient

Key Findings

  • Dramatic Superiority on Multi-Hop Graph Reasoning (ProsQA): On ProsQA, explicit search baselines degrade sharply due to search space explosion and scoring hallucinations (DPTS collapses to 0.097, SSDP to 0.057, and ToT uses 41,473 tokens for an accuracy of only 0.469). In contrast, IToT attains 0.812 accuracy using only 3,413 tokens per sample—a 34.3-point jump over ToT with a 12-fold token reduction.
  • Token Efficiency Matching Single-Chain CoT: While explicit search baselines consume tens of thousands of tokens per problem, IToT averages 1,544–3,413 tokens across all six benchmarks, matching CoT (1,517–4,057) and running up to \(20\times\) cheaper than DPTS and ToT.
  • Exceptional Resilience on Smaller Backbones: On smaller models such as Gemma-3-1B, text-based verifier pruning collapses (DPTS drops to 0.135 on GSM8K and 0.081 on MMLU). IToT retains 0.813 on GSM8K and 0.464 on MMLU, demonstrating that latent geometric steering provides far more reliable guidance than noisy text-based self-evaluation.

Highlights & Insights

  • Seamlessly Absorbing Search into Latent Dynamics: Unlike prior efficiency methods that simply prune textual branches, IToT completely eliminates candidate text decoding, mapping tree depth to token windows, branch exploration to representation perturbations, and pruning to cosine alignment.
  • Cheap Yet Generalizable Contrastive Calibrator: Extracting layer-wise difference vectors from merely 10 pairs of correct versus confused traces successfully isolates a task-invariant reasoning vector, avoiding expensive verifier training or reinforcement learning.
  • Surgical, Non-Destructive Activation Control: Scaling steering perturbations by activation norms and applying soft convex blending ensures that high-level conceptual guidance does not distort local syntax or cause autoregressive degeneration.

Limitations & Future Work

  • Calibration Domain Sensitivity: Prototypes are currently extracted from domain-specific calibration sets, which may introduce suboptimality under severe out-of-distribution domain shifts. Extending calibration to multi-task collections could yield more universal reasoning directions.
  • Indirect Proxy Nature of Continuous Alignment: Cosine alignment in latent space serves as a continuous surrogate rather than an exact formal logic prover. Hybrid systems combining continuous latent steering with sparse discrete verification at critical milestones represent a promising avenue.
  • Latent Optimization Latency: Although token generation costs are slashed, executing CMA-ES updates at every window introduces non-trivial tensor computation. Parallelized batching of branch candidates and dynamic window skipping for high-confidence tokens could further enhance throughput.
  • vs Tree-of-Thoughts (ToT) & Graph-of-Thoughts (GoT): ToT and GoT rely on explicit natural language thought generation and auxiliary prompting passes, suffering from severe decoding latency and quadratic token costs. IToT shifts the entire tree search into hidden representations, requiring zero auxiliary token emission.
  • vs Coconut & CODI (Implicit CoT): Coconut and CODI require extensive multi-stage training, continuous thought distillation, or parameter fine-tuning to compress reasoning into latent variables. IToT is completely training-free and model-agnostic, functioning as an inference-time plug-and-play controller.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Pioneering reformulation of multi-path tree search as continuous latent space steering without auxiliary text generation.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 5 backbones (1B to 7B scale), 6 benchmark domains, and extensive ablation and sensitivity studies.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clearly articulated motivations, robust mathematical formulation, and well-structured algorithmic descriptions.
  • Value: ⭐⭐⭐⭐⭐ Offers a high-throughput, low-latency paradigm for deploying advanced structured reasoning on resource-constrained platforms.