Latent Thoughts, Certain Gains: Bayesian Optimization for LLM Reasoning¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Latent Thought Optimization, Bayesian Inference, Test-Time Optimization, Variational Inference, Large Language Model Reasoning
TL;DR¶
BLTO is a training-free test-time reasoning framework for frozen LLMs that treats continuous latent thoughts as stochastic variables under a variational Bayesian inference formulation, optimizing latent vectors via an entropy-based intrinsic reward, reparameterized low-variance updates, and an uncertainty-guided adaptive controller.
Background & Motivation¶
Large Language Models (LLMs) have achieved unprecedented success across mathematical problem solving, symbolic manipulation, and formal reasoning, propelled predominantly by the paradigm of Chain-of-Thought (CoT) prompting. By prompting the model to generate intermediate rationales prior to emitting final answers, CoT allows complex problem statements to be systematically decomposed into manageable natural language substeps. Nevertheless, explicit textual CoT introduces severe computational and cognitive overheads: generating long sequences of intermediate tokens substantially drives up inference latency, memory footprint, and communication bottlenecks during deployment, while surface-level textual traces often fail to faithfully reflect internal attention dynamics and can introduce distracting, spurious, or hallucinated reasoning steps.
To overcome these structural limitations, researchers have turned toward latent reasoning, representing intermediate derivations directly as continuous hidden state representations within the model's continuous embedding space. Existing methodologies generally fall into two categories: fine-tuning approaches (such as iCoT, Coconut, and CODI) that adapt base model weights through multi-pass distillation or specialized continuous thinking objectives, and auxiliary module-based architectures (such as SoftCoT) that deploy a separately trained lightweight helper model to inject continuous soft thought prompts. However, unconstrained exploration in a continuous high-dimensional latent space remains notoriously fragile. Without explicit syntactic or semantic guardrails, continuous search easily degrades into representation collapse or out-of-distribution drift, and heuristic point-estimate search policies (such as policy gradient updates in LTPO) lack principled posterior exploration and convergence guarantees when faced with difficult problem instances.
The fundamental challenge is thus formulating a mathematically principled, test-time exploration framework that operates entirely on frozen base models without needing external ground-truth verifiers or expensive task-specific fine-tuning. Because the evolution of intermediate representations is intrinsically probabilistic and subject to epistemic uncertainty, modeling latent cognitive trajectories through the lens of Bayesian inference provides a natural and robust mathematical foundation. Core idea: formulate test-time latent thought optimization in frozen LLMs as Bayesian variational inference over continuous cognitive vectors, balancing an entropy-based self-supervised intrinsic reward against an analytical Gaussian prior KL regularization, while dynamically steering search depth via uncertainty-guided adaptive noise annealing.
Method¶
Overall Architecture¶
BLTO (Bayesian Latent Thought Optimization) intervenes directly at test time on a frozen auto-regressive language model \(M_\theta\). Given an input prompt \(x = (x_1, \dots, x_N)\), the embedding layer yields continuous representations \(E(x) \in \mathbb{R}^{N \times d}\). The model input is concatenated with \(K\) special continuous placeholder tokens \(H = (h_1, \dots, h_K) \in \mathbb{R}^{K \times d}\) representing latent reasoning states, forming \(E(x) \parallel H\). Rather than seeking an isolated point estimate for \(H\), BLTO infers the posterior distribution \(p(H|x) \propto p(H) \cdot \exp(\beta \cdot R(x, H))\). To render inference tractable over high dimensions, BLTO employs a diagonal-covariance Gaussian variational family \(q_\phi(H)\) to approximate the true posterior, turning the maximization of the evidence lower bound (ELBO) into a closed-form trade-off between prior regularization and expected intrinsic reward. Reparameterized gradients drive low-variance parameter updates via AdamW, while a dynamic controller monitors posterior variance to modulate noise schedules and terminate inference adaptively.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Prompt & Placeholders<br/>Concatenate E(x) and latent vectors H"] --> B["Variational Latent Modeling & Prior Regularization<br/>Diagonal Gaussian parameterization and analytical KL"]
B --> C["Entropy-Based Intrinsic Reward<br/>Frozen LLM predictive entropy over latent tokens"]
C --> D["Low-Variance Reparameterized Updates<br/>AdamW updates on variational mean and log-variance"]
D --> E["Uncertainty-Guided Adaptive Controller<br/>Dynamic noise annealing and multi-criteria early stopping"]
E -->|Unconverged / Steps Remaining| B
E -->|Stopping Criteria Met| F["Optimized Latent Representations<br/>Frozen LLM autoregressively generates final answer"]
Key Designs¶
1. Variational Latent Modeling & Prior Regularization: Preventing Latent Space Representation Collapse and Drift
Directly optimizing unconstrained continuous vectors in a high-dimensional space (\(K \times d\), typically thousands of dimensions) easily drifts outside the pretrained embedding manifold, degrading generation quality. BLTO treats each of the \(K\) continuous thought tokens as a stochastic variable parameterized by a factorized Gaussian variational posterior \(q_\phi(H) = \prod_{k=1}^K \mathcal{N}(h_k; \mu_k, \text{diag}(\sigma_k^2))\), governed by learnable variational parameters \(\phi = \{\mu, \rho\}\) with \(\mu \in \mathbb{R}^{K \times d}\) and \(\rho = \log(\sigma^2) \in \mathbb{R}^{K \times d}\). To anchor these vectors within valid semantic domains, a standard isotropic Gaussian prior \(p(H) = \prod_{k=1}^K \mathcal{N}(h_k; 0, I)\) is imposed. Because both the posterior and prior are diagonal Gaussians, the Kullback-Leibler divergence admits a closed-form analytical expression:
This analytical term delivers exact, low-variance gradients: the component \((\sigma_{k,j}^2 - \log \sigma_{k,j}^2 - 1)\) penalizes variance deviations away from the unit prior variance, preventing premature collapse while preserving exploratory capacity, whereas \(\mu_{k,j}^2\) penalizes extreme vector norms. Variational means \(\mu^{(0)}\) are initialized by perturbing baseline token embeddings with Gaussian noise \(\delta \sim \mathcal{N}(0, \epsilon^2 I)\) to break symmetry, while \(\rho^{(0)}\) is initialized to ensure stable initial exploration.
2. Entropy-Based Intrinsic Reward: Driving Reasoning Without External Verification Signals
In pure inference-time adaptation, ground-truth verifiers or task labels are unavailable. BLTO exploits the internal predictive consistency of the frozen LLM \(M_\theta\). Feeding the concatenated representation \(E(x) \parallel H\) through the frozen backbone produces token probability vectors \((P_1, \dots, P_{N+K}) = \text{softmax}(M_\theta(E(x) \parallel H))\). The predictive entropy across vocabulary \(V\) at the \(k\)-th latent position is defined as \(H_k = -\sum_{v \in V} P_{N+k}(v) \log(P_{N+k}(v) + \varepsilon)\). BLTO defines the intrinsic quality reward \(R(x, H)\) as the negative mean entropy across all \(K\) thought tokens:
The insight is that high-quality latent thoughts channel model cognition along coherent, deterministic causal paths, concentrating probability mass onto confident token subsets and thereby minimizing predictive entropy. To guarantee numerical stability during backpropagation, log-probabilities are clamped to \([-20, 0]\) and smoothed with \(\varepsilon = 10^{-10}\), yielding a differentiable intrinsic reward landscape that guides the variational parameters through end-to-end backpropagation.
3. Low-Variance Reparameterized Updates: Circumventing Policy Gradient Variance in High Dimensions
Conventional reinforcement learning or score-function estimators exhibit crippling variance when applied to continuous high-dimensional latent vectors. BLTO adopts the reparameterization trick \(H = \mu + \sigma \odot \epsilon\), where \(\epsilon \sim \mathcal{N}(0, I)\) and \(\sigma = \exp(\frac{1}{2}\rho)\), transforming the ELBO objective \(\mathcal{L}(\phi) = \text{KL}(q_\phi(H) \parallel p(H)) - \beta \mathbb{E}_{q_\phi}[R(x, H)]\) into an expectation over standard noise. Approximating the expectation over \(S\) Monte Carlo samples yields exact analytical gradients:
The gradient with respect to \(\mu\) balances prior shrinkage against reward ascent, while the gradient with respect to \(\rho\) dynamically scales exploration according to local curvature and uncertainty. Parameters are updated using AdamW, and log-variances \(\rho\) are clamped within \([-14, -2.3]\) to eliminate both numerical underflow and explosive dispersion.
4. Uncertainty-Guided Adaptive Controller: Dynamic Annealing and Instance-Aware Early Stopping
Reasoning queries present disparate levels of cognitive difficulty; fixing optimization steps across all instances leads to wasted computation on trivial problems and insufficient exploration on complex puzzles. BLTO introduces an adaptive controller steered by posterior uncertainty. First, sampling temperature is annealed in proportion to the mean posterior standard deviation \(\bar{\sigma}_t = \frac{1}{Kd}\sum_{k,j}\sigma_{k,j}^{(t)}\):
Early in optimization when uncertainty is high (\(\bar{\sigma}_t \approx \bar{\sigma}_0\)), \(\lambda_t \approx \lambda_{\max}\) induces broad stochastic exploration; as optimization converges and variance shrinks, \(\lambda_t \to \lambda_{\min}\), smoothly shifting from exploration to exploitation around the optimal mean. Second, a multi-criteria stopping condition evaluates step bounds \([T_{\min}, T_{\max}]\), stagnation of moving-average reward \(|\bar{R}_t - \bar{R}_{t-\tau}| < \epsilon_{\text{rew}}\), and uncertainty collapse \(\bar{\sigma}_t < \epsilon_{\text{unc}}\), gracefully terminating simple tasks early while dedicating maximum compute to out-of-distribution reasoning.
Loss & Training¶
Because BLTO is an entirely training-free test-time inference method, model parameters \(\theta\) remain permanently frozen. Test-time instance adaptation solves:
Implementation parameters are calibrated for efficiency and stability on a single NVIDIA A100 GPU: inverse temperature \(\beta = 0.8\), KL weighting regularizer \(0.008\), and \(S = 5\) sampled trajectories per step. Adaptive optimization runs between \(T_{\min} = 5\) and \(T_{\max} = 22\) steps with patience \(\tau = 5\), reward improvement threshold \(\epsilon_{\text{rew}} = 0.006\), and uncertainty threshold \(\epsilon_{\text{unc}} = 0.025\) triggered after 3 warmup steps. Final output generation applies top-\(k = 10\) with a maximum length of 4096 tokens.
Key Experimental Results¶
Main Results¶
BLTO is evaluated against major discrete and latent reasoning baselines across two primary open-source model families (LLaMA-3.1-8B-Instruct and Qwen-2.5-7B-Instruct) on five mathematical benchmarks: GSM8K, MATH-500, ASDiv-Aug, AIME 2024, and AIME 2025.
| Model & Method | GSM8K | MATH-500 | ASDiv-Aug | AIME 2024 | AIME 2025 | Avg. Acc (%) |
|---|---|---|---|---|---|---|
| LLaMA-3.1-8B-Instruct | ||||||
| Zero-Shot CoT (Kojima et al., 2022) | 76.88 | 48.60 | 79.58 | 6.67 | 0.00 | 42.35 |
| Zero-Shot CoT-Unk (Kojima et al., 2022) | 80.67 | 46.20 | 89.50 | 10.00 | 3.33 | 45.94 |
| SoftCoT (Xu et al., 2025b) | 80.36 | 39.80 | 87.57 | 0.00 | 0.00 | 41.55 |
| LTPO (Ye et al., 2025) | 80.89 | 46.60 | 88.25 | 10.00 | 6.67 | 46.48 |
| BLTO (Ours) | 82.03 | 51.00 | 90.08 | 16.67 | 6.67 | 49.29 |
| Qwen-2.5-7B-Instruct | ||||||
| Zero-Shot CoT (Kojima et al., 2022) | 88.55 | 72.40 | 91.71 | 10.00 | 10.00 | 54.53 |
| Zero-Shot CoT-Unk (Kojima et al., 2022) | 86.20 | 70.80 | 91.52 | 13.33 | 10.00 | 54.37 |
| SoftCoT (Xu et al., 2025b) | 85.29 | 65.20 | 88.92 | 0.00 | 0.00 | 47.88 |
| LTPO (Ye et al., 2025) | 87.87 | 71.80 | 91.71 | 10.00 | 3.33 | 52.94 |
| BLTO (Ours) | 88.55 | 74.80 | 92.58 | 20.00 | 16.67 | 58.52 |
When compared against methods that update model parameters during training (using LLaMA-3.1-8B-Instruct), BLTO outperforms all supervised and self-supervised alternatives: Genius (self-supervised fine-tuning) achieves 78.09% on GSM8K, 47.60% on MATH-500, and 3.33% on AIME 2024; SimpleRL-Zoo (external data RL) obtains 79.20%, 23.00%, and 0.00%; while training-free BLTO records 82.03%, 51.00%, and 16.67%, respectively.
Ablation Study & Scaling Analysis¶
Investigating the sensitivity to latent thought token length \(K\) on LLaMA-3.1-8B-Instruct reveals task-dependent representational capacity requirements:
| Thought Tokens \(K\) | ASDiv-Aug Acc (%) | GSM8K Acc (%) | MATH-500 Acc (%) | Empirical Observation & Analysis |
|---|---|---|---|---|
| \(K = 2\) | 90.08 | 80.14 | 47.20 | Optimal for simple elementary word math; prevents over-parameterization |
| \(K = 4\) | 87.86 | 80.06 | 48.40 | Slight capacity degradation on simple problems |
| \(K = 6\) | 88.63 | 79.98 | 49.00 | Plateau region with steady progression |
| \(K = 8\) | 89.40 | 81.90 | 49.40 | Benefits multi-step problems via expanded state space |
| \(K = 10\) | 88.82 | 82.03 | 51.00 | Peak performance on advanced high-school and competition math |
Evaluating model scaling across the Qwen2.5 series (0.5B, 1.5B, and 7B) confirms that BLTO yields dramatic gains on small models where parameter capacity restricts explicit CoT generation. On GSM8K, BLTO elevates Qwen2.5-0.5B from 13.57% (LTPO) to 37.76% (+24.19%) and Qwen2.5-1.5B from 40.80% to 57.86% (+17.06%). On ASDiv-Aug, 0.5B accuracy leaps from 50.87% to 75.53% (+24.66%).
Key Findings¶
- Difficulty-Dependent Scaling Gains: Performance gains scale monotonically with problem difficulty. While accuracy improvements on GSM8K and ASDiv-Aug range from 1% to 2% over LTPO, gains on MATH-500 reach 3% to 4%, and peak on the elite AIME competition benchmarks at 6.67% to 10.00%. The adaptive inference controller allocates proportionally deeper compute budgets to challenging reasoning problems.
- Outsized Amplification on Small Models: The performance leap is most profound on 0.5B and 1.5B models, where Bayesian latent optimization compensates for constrained parameter capacity by uncovering compact continuous inference trajectories inaccessible via discrete greedy decoding.
- Robust Cross-Domain Generalization: Beyond mathematics, BLTO demonstrates superior generalization on StrategyQA (achieving 73.69% on Qwen2.5-7B) and Symbolic Date Understanding (reaching 65.2% on LLaMA-3.1-8B, outperforming Zero-Shot CoT by 9.6%), whereas auxiliary-model approaches like SoftCoT suffer substantial domain transfer degradation.
Highlights & Insights¶
- From Heuristic Search to Principled Bayesian Variational Inference: BLTO replaces unstable point-estimate gradient ascent with an elegant probabilistic framework, leveraging Gaussian prior regularization and closed-form KL terms to stabilize continuous optimization against latent drift.
- Predictive Negative Entropy as an Unsupervised Reasoning Reward: Demonstrates that internal token entropy at continuous placeholder positions functions as a high-fidelity proxy for reasoning coherence, eliminating any dependency on external verifiers or ground-truth reward models.
- Uncertainty-Guided Compute Allocation: Using posterior variance to dynamically schedule noise temperatures and enforce early stopping implements an elegant test-time compute allocation mechanism that prevents over-optimization on trivial samples.
Limitations & Future Work¶
- Inference Latency Overhead: Maintaining a variational distribution and conducting iterative forward-backward passes across multiple Monte Carlo trajectories substantially increases Time-to-First-Token (TTFT), limiting real-time interactive deployment.
- Scalability Beyond 8B Parameters: While thoroughly verified on 0.5B to 8B models, scaling gradient-based test-time latent optimization to massive 70B+ or MoE architectures demands specialized memory-efficient backpropagation or low-rank parameterizations.
- Static Prior Assumption: The isotropic standard Gaussian prior is context-agnostic; developing input-conditioned dynamic priors could further accelerate variational convergence.
Related Work & Insights¶
- vs SoftCoT (Xu et al., 2025b): SoftCoT relies on an auxiliary small LLM trained offline to emit soft prompt vectors, introducing external training overhead and catastrophic OOD transfer drops; BLTO is completely training-free and optimizes directly on the frozen target model's intrinsic manifold.
- vs LTPO (Ye et al., 2025): LTPO applies policy gradient optimization on deterministic latent prompts, suffering from acute gradient variance and susceptibility to sharp local minima; BLTO formulates a variational posterior with analytical KL regularization and reparameterized updates, delivering smooth, bounded convergence.
- vs Coconut (Hao et al., 2024) & iCoT (Deng et al., 2024): These fine-tuning methods fundamentally alter base model parameters to force continuous internal reasoning, risking general conversational degradation; BLTO preserves base weights intact, operating as a modular, plug-and-play inference engine.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates test-time continuous thought search as variational Bayesian inference with predictive entropy rewards, establishing a sound probabilistic paradigm.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 5 mathematical benchmarks, 2 symbolic/commonsense datasets, 2 model architectures, and 3 model scales with rigorous ablations.
- Writing Quality: ⭐⭐⭐⭐⭐ Mathematically rigorous, clear prose narrative, and tightly constructed experimental arguments.
- Value: ⭐⭐⭐⭐⭐ Offers a transformative framework for training-free test-time reasoning enhancement and compute scaling in frozen language models.