Latent Thoughts, Certain Gains: Bayesian Optimization for LLM Reasoning¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Latent reasoning, Bayesian optimization, Variational inference, Test-time compute, Output entropy reward
TL;DR¶
BLTO introduces a training-free framework that casts continuous latent thought generation in frozen LLMs as variational Bayesian inference over stochastic cognitive states, employing self-supervised output predictive entropy as an intrinsic reward and reparameterized gradient updates to achieve consistent gains across complex reasoning benchmarks.
Background & Motivation¶
The emergence of Chain-of-Thought (CoT) prompting has dramatically advanced the ability of large language models (LLMs) to tackle multi-step reasoning, mathematical problem solving, and symbolic inference. By verbalizing intermediate rationales into discrete textual sequences, CoT breaks down complicated inquiries into manageable subproblems. Despite its remarkable success, explicit textual reasoning introduces severe computation and latency overheads, accompanied by an escalating key-value (KV) cache memory footprint as reasoning trajectories lengthen. Furthermore, explicit natural language tokens do not always faithfully mirror the internal cognitive transitions of neural networks, frequently introducing spurious, redundant, or misleading rationales that cause cascading errors.
To address these limitations, latent reasoning has emerged as a promising alternative, representing intermediate thought processes as continuous vectors directly within the hidden or embedding representations rather than externalizing them into lexical tokens. Methods such as Coconut explore continuous reasoning in intermediate layers, while approaches like SoftCoT utilize auxiliary small models to generate soft prompts, and CCOT compresses reasoning into condensed meditation tokens. However, existing latent reasoning techniques suffer from acute fragility and instability, especially on out-of-distribution problems or tasks demanding rigorous logical precision. Because continuous latent vectors exist in an unconstrained, high-dimensional manifold lacking the syntactic and semantic inductive biases of discrete vocabularies, unguided search easily leads to activation collapse and severe semantic drifting away from the pretrained representation space.
Prior solutions have struggled to strike an effective balance: fine-tuning-based approaches like iCoT and CODI alter base model weights and fail to adapt dynamically at inference time, whereas auxiliary prompt generators or policy-gradient approaches like LTPO suffer from severe gradient variance and lack principled regularizers to balance exploration against distribution collapse. Core idea: Formulate continuous latent thought optimization as a variational Bayesian inference problem over latent cognitive states, regularizing the posterior toward an isotropic Gaussian prior and driving it via self-supervised output entropy rewards with reparameterized gradient updates and uncertainty-guided adaptive step allocation.
Method¶
Overall Architecture¶
BLTO enhances frozen autoregressive language models without altering any pre-trained model weights \(\theta\). Given an input token prompt \(x = (x_1, \dots, x_N)\) mapped into continuous token embeddings \(E(x) \in \mathbb{R}^{N \times d}\), BLTO appends \(K\) special placeholder tokens (e.g., [THINK]), whose continuous embeddings \(H = (h_1, \dots, h_K) \in \mathbb{R}^{K \times d}\) serve as learnable latent thoughts. Rather than estimating a deterministic point representation, BLTO treats \(H\) as a stochastic variable governed by a posterior distribution \(p(H|x) \propto p(H) \exp(\beta R(x, H))\). It fits a factorized Gaussian variational posterior \(q_\phi(H)\) by maximizing an evidence lower bound (ELBO) combining closed-form KL divergence regularization with an intrinsic predictive entropy reward, dynamically scheduling inference iterations via posterior uncertainty estimates.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input sequence & placeholder tokens<br/>Concat E(x) with K [THINK] embeddings H"] --> B["Variational Gaussian posterior & prior regularization<br/>Factorized Gaussian q_phi(H) & isotropic prior KL constraint"]
B --> C["Output entropy self-supervised reward & reparameterized updates<br/>Negative mean entropy drives low-variance updates for mu and rho"]
C --> D["Uncertainty-guided adaptive annealing & multi-criteria stopping<br/>Scale sampling temperature and allocate steps via convergence state"]
D --> E["Final answer generation<br/>Autoregressive decoding conditioned on optimal posterior mean"]
Key Designs¶
1. Variational Gaussian posterior & prior regularization: High-dimensional unconstrained search induces representation drift and mode collapse
Searching for optimal thought vectors in continuous space \(\mathbb{R}^{K \times d}\) (which easily spans thousands of dimensions) via direct optimization frequently causes the representations to drift far from the pre-trained embedding manifold, yielding degraded generations. BLTO establishes a principled Bayesian formulation by placing a standard isotropic Gaussian prior \(p(H) = \prod_{k=1}^K \mathcal{N}(h_k; 0, I)\) over latent thoughts, which encodes a preference for small norms and acts as an inherent \(L_2\) regularizer. To make inference computationally tractable, BLTO adopts a factorized Gaussian variational family with diagonal covariance \(q_\phi(H) = \prod_{k=1}^K \mathcal{N}(h_k; \mu_k, \text{diag}(\sigma_k^2))\), parameterized by variational mean \(\mu \in \mathbb{R}^{K \times d}\) and unconstrained log-variance \(\rho \in \mathbb{R}^{K \times d}\) where \(\sigma_k^2 = \exp(\rho_k)\). The resulting minimization objective is the negative ELBO:
Because both the variational posterior and the prior are Gaussian, the KL divergence yields an exact analytical closed-form:
This analytical term provides a deterministic, zero-variance gradient that penalizes deviations of \(\mu\) from zero and drives the variance \(\sigma^2\) toward the unit prior variance, stabilizing continuous exploration.
2. Output entropy self-supervised reward & reparameterized updates: Lack of ground-truth supervision and high variance of policy score-functions
In pure test-time scenarios, external supervision or domain-specific verifiers are unavailable. BLTO devises an intrinsic reward derived entirely from the predictive uncertainty of the frozen model's output distribution. When feeding the concatenated prompt and latent thought embeddings \(E(x) \parallel H\) into the frozen model \(M_\theta\), the model produces vocabulary probability distributions \(P_{N+k}(v)\) across the \(K\) latent thought positions. The entropy at token position \(k\) is \(H_k = -\sum_{v \in \mathcal{V}} P_{N+k}(v) \log P_{N+k}(v)\). BLTO takes the negative average entropy across all latent positions as the self-supervised reward:
To avoid the high variance inherent in standard REINFORCE score-function estimators, BLTO applies the reparameterization trick \(H = \mu + \sigma \odot \epsilon\) with standard normal noise \(\epsilon \sim \mathcal{N}(0, I)\) and \(\sigma = \exp(\frac{1}{2}\rho)\). Using \(S\) Monte Carlo samples, the exact gradients with respect to \(\mu\) and \(\rho\) are:
where \(\nabla_H R^{(s)}\) is obtained via backpropagation through the frozen language model. The mean gradient balances prior regularization against reward maximization, while the log-variance gradient dynamically regulates exploration based on reward landscape curvature and posterior variance.
3. Uncertainty-guided adaptive annealing & multi-criteria stopping: Fixed iteration budgets cause over-optimization and wasteful computation
Different reasoning questions require drastically varying computational budgets: straightforward arithmetic problems can degrade if over-optimized, whereas competition-level challenges require broader search. Furthermore, fixed sampling noise disrupts convergence in late stages. BLTO addresses this by introducing simulated annealing over the sampling noise via posterior uncertainty:
where \(\bar{\sigma}_t = \frac{1}{Kd}\sum_{k,j} \sigma_{k,j}^{(t)}\) is the average posterior standard deviation at step \(t\). When uncertainty is large early on, \(\lambda_t \to \lambda_{\max}\) encourages wide exploration; as the posterior concentrates, \(\lambda_t \to \lambda_{\min}\) sharpens local exploitation around the mean. To govern inference budgets, BLTO establishes a multi-criteria early stopping rule:
where \(\bar{R}_t\) is an exponential moving average of rewards. This early stopping mechanism dynamically allocates 5 steps for simpler problems while permitting up to 22 steps for intricate questions, preventing over-optimization while conserving test-time compute.
Loss & Training¶
BLTO operates strictly at test time, keeping all base model parameters \(\theta\) frozen with zero offline parameter training. For each test query, variational parameters \(\phi = \{\mu, \rho\}\) are optimized online on a single GPU. The mean parameter \(\mu^{(0)}\) is initialized from the placeholder token embeddings with additive Gaussian noise \(\delta \sim \mathcal{N}(0, \epsilon^2 I)\) to break symmetry, while \(\rho^{(0)}\) is initialized to \(\log(\sigma_0^2)\mathbf{1}\). Parameters are updated using AdamW with learning rate \(\eta\), and \(\rho\) is clipped to \([-14, -2.3]\) to avoid numerical instability. The default configuration uses \(S = 5\) trajectory samples, inverse temperature \(\beta = 0.8\), and KL weight \(0.008\). Adaptive search steps range from \(T_{\min} = 5\) to \(T_{\max} = 22\) with patience \(\tau = 5\), reward threshold \(\epsilon_{\text{rew}} = 0.006\), and uncertainty threshold \(\epsilon_{\text{unc}} = 0.025\) triggered after 3 warmup steps.
Key Experimental Results¶
Main Results¶
Evaluation was conducted on LLaMA-3.1-8B-Instruct and Qwen-2.5-7B-Instruct across five mathematical reasoning benchmarks: GSM8K, MATH-500, ASDiv-Aug, AIME2024, and AIME2025.
| Model & Method | GSM8K | MATH-500 | ASDiv-Aug | AIME2024 | AIME2025 | Avg. Accuracy |
|---|---|---|---|---|---|---|
| LLaMA-3.1-8B-Instruct | ||||||
| Zero-Shot CoT | 76.88 | 48.60 | 79.58 | 6.67 | 0.00 | 42.35 |
| Zero-Shot CoT-Unk | 80.67 | 46.20 | 89.50 | 10.00 | 3.33 | 45.94 |
| SoftCoT | 80.36 | 39.80 | 87.57 | 0.00 | 0.00 | 41.55 |
| LTPO | 80.89 | 46.60 | 88.25 | 10.00 | 6.67 | 46.48 |
| BLTO (Ours) | 82.03 | 51.00 | 90.08 | 16.67 | 6.67 | 49.29 |
| Qwen-2.5-7B-Instruct | ||||||
| Zero-Shot CoT | 88.55 | 72.40 | 91.71 | 10.00 | 10.00 | 54.53 |
| Zero-Shot CoT-Unk | 86.20 | 70.80 | 91.52 | 13.33 | 10.00 | 54.37 |
| SoftCoT | 85.29 | 65.20 | 88.92 | 0.00 | 0.00 | 47.88 |
| LTPO | 87.87 | 71.80 | 91.71 | 10.00 | 3.33 | 52.94 |
| BLTO (Ours) | 88.55 | 74.80 | 92.58 | 20.00 | 16.67 | 58.52 |
When compared against training-based methods updating model parameters on LLaMA-3.1-8B-Instruct, BLTO achieves superior performance without updating any model weights:
| Method | Train Params | Supervision Type | GSM8K | MATH-500 | AIME2024 |
|---|---|---|---|---|---|
| Genius | Yes | Self-supervised | 78.09 | 47.60 | 3.33 |
| SimpleRL-Zooโ | Yes | External Data | 79.20 | 23.00 | 0.00 |
| iCoT | Yes | External Data | 50.05 | - | - |
| SoftCoT | Yes | External Data | 80.36 | 39.80 | 0.00 |
| LTPO | No | Self-supervised | 80.89 | 46.60 | 10.00 |
| BLTO (Ours) | No | Self-supervised | 82.03 | 51.00 | 16.67 |
Ablation Study¶
Ablation on the number of latent thought tokens \(K\) using LLaMA-3.1-8B-Instruct highlights task-dependent capacity requirements:
| Tokens \(K\) | ASDiv-Aug (%) | GSM8K (%) | MATH-500 (%) |
|---|---|---|---|
| \(K = 2\) | 90.08 | 80.14 | 47.20 |
| \(K = 4\) | 87.86 | 80.06 | 48.40 |
| \(K = 6\) | 88.63 | 79.98 | 49.00 |
| \(K = 8\) | 89.40 | 81.90 | 49.40 |
| \(K = 10\) | 88.82 | 82.03 | 51.00 |
Across model sizes in the Qwen2.5 family, BLTO shows dramatic gains for lightweight architectures on GSM8K and ASDiv-Aug:
| Dataset | Method | Qwen2.5-0.5B | Qwen2.5-1.5B | Qwen2.5-7B |
|---|---|---|---|---|
| GSM8K | LTPO | 13.57 | 40.80 | 87.87 |
| GSM8K | BLTO (Ours) | 37.76 (+24.19) | 57.86 (+17.06) | 88.55 (+0.68) |
| ASDiv-Aug | LTPO | 50.87 | 56.07 | 91.71 |
| ASDiv-Aug | BLTO (Ours) | 75.53 (+24.66) | 78.90 (+22.83) | 92.58 (+0.87) |
Key Findings¶
- Gains scale with task difficulty: On elementary math benchmarks like GSM8K and ASDiv-Aug, BLTO delivers modest gains of 1% to 2% over LTPO. In contrast, on challenging competition-style problems (MATH-500, AIME2024, and AIME2025), accuracy improvements jump by 3% to 10%, indicating that Bayesian uncertainty exploration is critical for navigating non-trivial reasoning spaces.
- Surpassing parameter fine-tuning without updating weights: BLTO outperforms all parameter-updated baselines (Genius, SimpleRL-Zoo, SoftCoT) on GSM8K (82.03%), MATH-500 (51.00%), and AIME2024 (16.67%), proving that frozen models contain latent reasoning abilities that can be elicited through test-time Bayesian optimization.
- Smaller models experience dramatic improvements: For resource-constrained models like Qwen2.5-0.5B and 1.5B, BLTO delivers improvements exceeding 24% on GSM8K and ASDiv-Aug, compensating for limited capacity by injecting high-quality continuous reasoning representations.
- Latent token capacity demands vary by problem domain: For straightforward word problems (ASDiv-Aug), \(K=2\) tokens suffice and larger counts induce slight degradation; for complex multi-step theorem problems (MATH-500), accuracy monotonically climbs from 47.20% at \(K=2\) to 51.00% at \(K=10\).
Highlights & Insights¶
- From heuristic hill-climbing to grounded variational inference: Previous latent prompting methods treated continuous vector refinement as unregularized policy search or deterministic gradient ascent, which suffer from severe variance and divergence. BLTO derives a closed-form ELBO objective with isotropic Gaussian prior regularization, establishing a principled probabilistic foundation.
- Self-supervised output entropy as intrinsic confidence: By measuring the negative entropy of token predictions at placeholder positions, BLTO derives a smooth, label-free reward that quantifies internal predictive certainty, guiding the model from diffuse uncertainty toward decisive cognitive states.
- Uncertainty-guided test-time compute allocation: Dynamic temperature annealing and multi-criteria convergence stopping allow the system to allocate computation elastically, spending minimal steps on routine queries while dedicating extended search budgets to hard problems.
Limitations & Future Work¶
- Inference latency from test-time gradient backpropagation: Requiring 5 to 22 backward passes through the frozen model per instance adds noticeable latency and memory overhead, limiting deployment in ultra-low-latency real-time applications.
- Scalability to 70B+ frontier models unverified: Experiments cover models from 0.5B up to 8B parameters; evaluating gradient memory overhead and empirical scaling behavior on very large models (70B+) remains future work.
- Static isotropic prior limitations: The standard normal prior is context-agnostic; future work could investigate conditional priors conditioned on prompt representations or low-rank variational subspaces.
Related Work & Insights¶
- vs Coconut / SoftCoT: Coconut relies on multi-stage curriculum fine-tuning to internalize reasoning tokens into hidden states, while SoftCoT requires training a separate small auxiliary model. BLTO requires zero fine-tuning and zero auxiliary models, performing self-supervised optimization entirely at test time.
- vs LTPO (Latent Thought Policy Optimization): LTPO uses standard policy gradients on continuous prompt tokens, which suffers from high gradient variance and lacks regularization against representation drift. BLTO introduces an isotropic Gaussian prior, closed-form KL regularization, reparameterized low-variance gradients, and adaptive stopping.
- vs Discrete Test-time Search (MCTS / Best-of-N): Discrete search methods sample multiple full textual responses and employ external verifiers, incurring massive token generation overhead. BLTO conducts efficient continuous optimization in the compact embedding space before decoding a single final response.
Rating¶
- Novelty: โญโญโญโญโญ Formulates continuous latent thought optimization as variational Bayesian inference with closed-form KL regularization and self-supervised entropy rewards.
- Experimental Thoroughness: โญโญโญโญโ Comprehensive evaluation across two model families, 5 mathematical datasets, commonsense/symbolic benchmarks, and model scaling from 0.5B to 8B.
- Writing Quality: โญโญโญโญโญ Rigorous mathematical formulations, clear motivation, and complete analytical gradient derivations.
- Value: โญโญโญโญโญ Opens an effective, training-free frontier for eliciting complex reasoning from frozen and edge-sized language models at test time.