Skip to content

One-Token Verification for Reasoning Correctness Estimation

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Multimedia-Analytics-Laboratory/OTV
Area: LLM Reasoning
Keywords: Reasoning Correctness Estimation, Test-Time Scaling, Parallel Thinking, One-Token Verification, KV Cache Probing

TL;DR

OTV augments a reasoning LLM with a LoRA-gated verifier that probes the ongoing key-value (KV) cache via a dedicated [ToT] token, estimating token-level reasoning correctness in a single forward pass without degrading primary reasoning and saving up to 90% of generation tokens via confidence-guided pruning on AIME.

Background & Motivation

Large language models have achieved remarkable milestones in multi-step mathematical problem solving and complex logical reasoning. In parallel, test-time scaling—specifically parallel thinking strategies such as majority voting, Best-of-N decoding, and tree-structured exploration—has emerged as a foundational paradigm to push generation accuracy toward theoretical upper bounds like Pass@k. Nevertheless, generating dozens or hundreds of candidate reasoning trajectories inevitably incurs staggering inference latency and token overhead. This computational burden is heavily exacerbated by System-2-style "overthinking," where models engage in repetitive, long-winded rationalizations that provide diminishing returns or even degrade final correctness.

To effectively constrain the inference budget without sacrificing task performance, decoding systems require dependable mechanisms to assess the correctness of individual reasoning traces early during generation. Existing approaches fall into two polarized paradigms, both suffering from acute bottlenecks. Internal methods rely on the model's intrinsic token probabilities, entropy, or self-queried verbalized confidence; while essentially free of auxiliary model overhead, they suffer from severe miscalibration and struggle to distinguish correct trajectories from flawed ones across extended contexts. Conversely, external methods deploy dedicated outcome or process reward models (ORMs/PRMs); however, treating the base LLM as an opaque black box forces full-sequence re-encoding, injects massive latency overhead, and suffers from domain mismatch while failing to support anytime, on-the-fly step verification before a full trajectory is produced.

This paper's angle of attack is that the key-value (KV) cache accumulated throughout autoregressive decoding constitutes a sufficient statistic of the reasoning prefix, encoding rich, multi-layer internal representations that vastly surpass compressed single-vector hidden states. If this native context can be tapped without modifying the base generator's on-policy reasoning path, one can achieve external-grade verification accuracy at internal-level execution speeds. Core idea: integrate a gated LoRA verifier into the base LLM and activate it via a dedicated truth token [ToT] that probes the native KV cache through a linear-ramp pseudo-confidence objective, delivering single-pass, token-level correctness estimates that guide aggressive parallel pruning.

Method

Overall Architecture

The OTV (One-Token Verification) architecture establishes a unified framework that seamlessly toggles an autoregressive LLM between generation and verification modes. The input consists of the user query and an ongoing partial reasoning trace, and the output is a continuous scalar confidence score \(\hat{c}_t \in [0, 1]\) estimating the likelihood that the current trajectory leads to a correct terminal solution. During standard text generation, the model executes normal token decoding and accumulates multi-layer KV states. Whenever verification is requested at position \(t\), a binary gate activates the LoRA pathway, and a specialized learnable token [ToT] (Token of Truth) is inserted at position \(t+1\). This token attends directly to the accumulated KV cache across all layers, and a three-layer MLP regression head maps its final hidden state to the scalar confidence score.

During training, OTV preserves token-level parallelism through a triangular-masked probe sequence, allowing an entire sequence of \(T\) token positions to be supervised simultaneously in a single forward pass against dense, linearly interpolated pseudo-confidence targets. At test time, OTV scores candidate prefixes to power weighted majority voting and flexible dynamic pruning protocols (including Drop@10, Stop@600, and Halve@300), actively penalizing protracted redundant traces while retaining compact, correct mathematical derivations.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Query and Partial Reasoning Trace<br/>Autoregressive token generation"] --> B["Gated LoRA Verification Pathway<br/>Switch between reasoner and verifier via m_t"]
    B --> C["KV Cache-Based Probing via [ToT]<br/>Single-token attention reusing cached states"]
    C --> D["Token-Level Pseudo-Confidence Labeling<br/>Linear ramp inducing inverse-length preference"]
    D --> E["Single-Pass Parallel Trace Probing<br/>Triangular mask enforcing causal prefix visibility"]
    E --> F["Confidence-Guided Pruning and Aggregation<br/>Weighted voting and early exit cutting 90% tokens"]

Key Designs

1. Gated LoRA Verification Pathway: Decoupling Reasoner and Verifier Modes Directly fine-tuning an LLM to perform verification risks causing catastrophic forgetting of its foundational mathematical reasoning skills. OTV circumvents this tension by attaching a parallel low-rank adaptation pathway \(W_{\text{LoRA}} = W + m_t R S\) specifically to the linear projections (Query, Key, and Value weights) of selected Transformer layers. The binary gate \(m_t \in \{0, 1\}\) dictates the operational role: when \(m_t = 0\), the LoRA update is completely bypassed, ensuring the model functions identically to the frozen baseline reasoner without any memory or computational overhead. When \(m_t = 1\), the low-rank matrices \(R \in \mathbb{R}^{d_{\text{out}} \times r}\) and \(S \in \mathbb{R}^{r \times d_{\text{in}}}\) are engaged, transforming the network into a verification probe. This architectural decoupling preserves the generative distribution while embedding high-capacity discriminative estimation directly into the native model.

2. KV Cache-Based Probing via [ToT]: Sufficient Statistic Reuse and Instant Scalar Regression Conventional probing approaches discard token-wise and layer-wise context by training linear heads solely on top-layer hidden states, while external PRMs require costly recomputation of the entire token sequence. OTV recognizes that the multi-layer KV cache \(C_t = \{K_t^{(l)}, V_t^{(l)}\}_{l=1}^L\) accumulated up to token \(t\) serves as a lossless sufficient statistic of the ongoing prefix. In verification mode, a dedicated learnable token [ToT] is appended at position \(t+1\). At each layer \(l\), its LoRA-augmented query, key, and value vectors \(\tilde{q}_{t+1}^{(l)}, \tilde{k}_{t+1}^{(l)}, \tilde{v}_{t+1}^{(l)}\) attend to the cached prefix and itself: $$ \tilde{h}{t+1}^{(l)} = \left[ V_t^{(l)}, \tilde{v}}^{(l)} \right] \text{softmax}\left( \frac{\left[ K_t^{(l)}, \tilde{k{t+1}^{(l)} \right]^\top \tilde{q} \right) $$ Upon propagating through all }^{(l)}}{\sqrt{D}\(L\) layers, the final hidden state \(\tilde{h}_{t+1}^{(L)} \in \mathbb{R}^{D \times 1}\) is fed into a lightweight three-layer MLP head \(g(\cdot)\) to yield the continuous scalar prediction \(\hat{c}_t = g(\tilde{h}_{t+1}^{(L)}) \in [0, 1]\). Because the entire prefix history is retrieved directly from memory, the query requires only a single forward token step, eliminating vocabulary-level token decoding biases (e.g., lexical preferences over "correct" vs "wrong") and providing well-calibrated scalar routing signals.

3. Token-Level Pseudo-Confidence Labeling: Outcome-Driven Dense Supervision with Inverse-Length Bias Collecting step-level human annotations or conducting extensive Monte Carlo Tree Search rollouts to supervise process verifiers is prohibitively expensive. OTV addresses this bottleneck by deriving dense token-level targets \(c_t\) exclusively from terminal outcome correctness \(y \in \{0, 1\}\). Beginning from a neutral prior of maximal uncertainty \(c_0 = 0.5\), confidence is linearly interpolated toward the terminal outcome over trace length \(T\): $$ c_t = \text{linear}(t) = 0.5 + (y - 0.5) \frac{t}{T} $$ Under mean squared error (MSE) optimization, the verifier learns to approximate the conditional expectation \(\mathbb{E}[c_t \mid s_t]\). Rigorous theoretical analysis reveals that the linear ramp mathematically induces an inverse-length preference: for two prefixes with equal conditional success probabilities, the shorter trajectory accumulates confidence at a higher rate and reaches decision thresholds earlier. This intrinsic inductive bias naturally encourages the model to prioritize concise, robust reasoning paths over redundant, protracted derivations.

4. Single-Pass Parallel Trace Probing: Preserving Full Transformer Training Throughput Although test-time verification operates sequentially as a single-token probe, sequential token-by-token evaluation during training would dismantle Transformer batch parallelism. OTV introduces a parallel training scheme that evaluates all \(T+1\) prefix positions within a single forward pass. A probe sequence \([\text{ToT}]_{1:T+1}\) is constructed, and a structured lower-triangular causal attention mask \(M \in \mathbb{R}^{T \times (T+1)}\) (with \(M_{i,j} = 0\) for \(i < j\) and \(-\infty\) for \(i \ge j\)) is applied to ensure that probe \(t+1\) attends solely to the cached prefix \(1:t\). Inter-probe attention is strictly restricted to diagonal self-logits. This formulation is mathematically equivalent to \(T+1\) independent single-token forward evaluations while sharing the common cached prefix computation, allowing 1,000 training traces to be calibrated in just 28.5 minutes on a single A100 GPU.

5. Confidence-Guided Pruning and Aggregation: High-Efficiency Test-Time Scaling OTV empowers both post-hoc aggregation and online trajectory pruning. For offline aggregation, the verifier averages token scores over the final 100 tokens of each trace, discards the bottom 50% of candidates, and conducts confidence-weighted majority voting. In online decoding, OTV drives three compute-efficient Best-of-N variants: Drop@10 (pruning the single lowest-scoring candidate every 10 tokens until one remains), Stop@600 (terminating all but the highest-scoring candidate at 600 tokens), and Halve@300 (halving the candidate pool every 300 tokens). These strategies drastically prune unpromising reasoning branches, delivering state-of-the-art competition math accuracy while slashing decoding token budgets by up to 90%.

Loss & Training

OTV freezes the base LLM weights and trains only the LoRA adapter matrices and the three-layer regression head. Optimization is conducted using AdamW via LlamaFactory for 3 epochs with a learning rate of \(10^{-4}\) and a batch size of 128. The MSE objective is applied exclusively across response tokens, masking out prompt and question tokens: $$ \ell = \frac{1}{T} \sum_{t=1}^T \left( c_t - \hat{c}_t \right)^2 $$

Key Experimental Results

Main Results

The primary evaluation focuses on demanding competition-grade benchmarks—AIME 2024 and AIME 2025—across multiple model scales (Qwen3-4B-Instruct, Qwen3-8B, and DAPO-Qwen-32B). Results report weighted majority-voting accuracy (%) over 128 sampled candidate traces, averaged across 64 runs, with the lowest-scoring 50% of traces pruned prior to voting.

Model Category Method AIME 2024 Accuracy (%) AIME 2025 Accuracy (%)
Qwen3-4B Oracle Bound Pass@128 91.46 83.13
Qwen3-4B Unweighted Baseline Pass@1 / Maj@128 60.29 / 75.42±1.61 46.67 / 66.46±1.16
Qwen3-4B Internal Verifier DeepConf 77.76±2.43 66.77±1.18
Qwen3-4B Internal Verifier GenRM 79.11±2.06 66.72±1.82
Qwen3-4B External PRM VersaPRM-8B 75.52±1.69 66.72±1.10
Qwen3-4B External PRM Math-Shepherd-7B 74.27±1.50 66.56±1.02
Qwen3-4B External PRM Qwen2.5-PRM-7B 78.18±1.85 63.49±2.39
Qwen3-4B Proposed Method OTV (Ours) 83.33±1.57 69.32±1.46
DAPO-Qwen-32B Oracle Bound Pass@128 83.75 68.75
DAPO-Qwen-32B Unweighted Baseline Pass@1 / Maj@128 51.77 / 66.72±0.93 36.42 / 41.77±1.76
DAPO-Qwen-32B Internal Verifier DeepConf 66.77±1.17 44.79±1.65
DAPO-Qwen-32B External PRM Qwen2.5-PRM-7B 67.45±2.26 46.77±3.06
DAPO-Qwen-32B Proposed Method OTV (Ours) 70.83±1.86 49.58±1.10

Under online compute-constrained decoding (Best-of-N and efficient pruning variants at \(N=128\)), OTV consistently matches or outperforms competing methods while generating considerably shorter reasoning outputs:

Model Verifier Best-of-128 Drop@10 Stop@600 Halve@300
Qwen3-4B DeepConf 64.95 (9664) 62.86 (7044) 59.90 (7150) 61.98 (6742)
Qwen3-4B Qwen2.5-PRM-7B 71.77 (3720) 63.80 (3040) 65.73 (4211) 66.46 (3173)
Qwen3-4B OTV (Ours) 73.44 (5447) 63.39 (4427) 63.75 (4431) 67.03 (4132)
DAPO-32B DeepConf 53.92 (5101) 50.52 (6382) 51.82 (7176) 50.94 (3772)
DAPO-32B Qwen2.5-PRM-7B 57.03 (4888) 55.52 (5525) 55.57 (4939) 51.98 (5660)
DAPO-32B OTV (Ours) 63.18 (4623) 55.95 (3397) 53.54 (3211) 55.05 (3436)

(Values in parentheses indicate average generated trace length in tokens)

Ablation Study

The ablation analysis systematically verifies the effectiveness of pseudo-confidence labeling strategies and model capacity scaling.

Labeling Heuristic Maj@128 (AIME24) Maj@128 (AIME25) Halve@300 (AIME24) Halve@300 (AIME25) Description
Linear Ramp (Default) 83.33±1.57 69.32±1.46 67.03 (4132) 49.02 (3170) Linear progression from 0.5 to outcome \(y\)
Constant Label 81.41±1.65 68.49±2.04 66.41 (3951) 49.27 (4503) Uniform hindsight label \(c_t = y\)
Sigmoid Ramp (\(\alpha=4\)) 80.05±1.91 66.51±0.92 68.44 (4210) 46.30 (4294) S-curve smooth transition
Noise-Perturbed Ramp 81.87±1.65 67.92±1.61 71.30 (4134) 48.33 (5736) Linear base plus sinusoidal & Gaussian noise
Stepwise Ramp 82.40±1.50 68.80±1.60 70.63 (4231) 37.60 (7095) Step-level plateau progression via delimiters

In architectural ablations on GSM8K under matched parameter counts (3.88M to 10.85M parameters), OTV's KV cache probing achieved 94.14% to 96.04% accuracy, substantially outperforming a top-layer hidden-state probe baseline (92.82% to 93.51%), demonstrating the crucial role of multi-layer KV representations.

Key Findings

  • Superior Trace Stratification: Across qualitative problem analyses, OTV demonstrates sharp separation between valid and invalid solutions: scores steadily climb along correct paths while remaining deeply suppressed on incorrect ones. Competing PRMs and DeepConf produce heavily overlapping bands.
  • Natural Mitigation of Overthinking: The linear ramp objective mathematically creates an inverse-length penalty. Shorter valid reasoning chains reach high confidence thresholds earlier, protecting concise derivations from being pruned and mitigating pathological looping.
  • Halve@300 Yields Optimal Trade-Offs: Among dynamic pruning strategies, Halve@300 requires only 254 verification queries per problem (a geometric series upper-bounded by \(2N\)) while beating full-trace majority voting and saving up to 90% of token generation costs.

Highlights & Insights

  • Gated In-Model Verification: By leveraging a binary gate with LoRA adapters and reusing the native KV cache, OTV bridges internal and external verification, offering external PRM discriminative power at near-zero incremental inference latency.
  • Single-Pass Parallel Probing Scheme: Employing lower-triangular causal masks and diagonal probe-to-probe attention enables token-level verification across thousands of tokens in a single forward pass, maintaining standard Transformer training throughput.
  • Theoretically Sound Length Prior: Deriving dense supervision via an outcome-based linear ramp provably induces an inverse-length preference, providing an elegant, RL-free mechanism to curb verbose overthinking.

Limitations & Future Work

  • Approximation of Monotonic Heuristics: While effective in practice, the linear ramp heuristic cannot fully capture non-monotonic dynamics in human reasoning, such as productive dead ends, backtracking, or late-stage self-corrections.
  • Model-Specific Architecture Dependency: Because OTV directly couples with the internal hidden dimensions and KV cache layout of the base model, it cannot be transferred across distinct model families without retraining.
  • Future Directions: The authors highlight joint co-adaptation between generator and verifier, iterative bootstrapping of higher-fidelity pseudo-labels, ternary confidence outputs (correct / incorrect / uncertain) for risk-controlled abstention, and expansion to code generation.
  • vs DeepConf: DeepConf computes token confidence using heuristic unsupervised log-likelihood, requiring no training but suffering from miscalibration; OTV trains an explicit regression verifier directly targeting outcome correctness, yielding far superior calibration.
  • vs Process Reward Models (e.g., Math-Shepherd, Qwen2.5-PRM): PRMs depend on expensive step annotations or extensive MCTS rollouts and require complete re-encoding of generated text; OTV requires zero human step labels and executes via single-token KV cache probing.
  • vs Hidden-State Probing: Prior probing works attach heads to the final layer's hidden states, discarding multi-layer token interactions; OTV probes the full KV cache through cross-layer attention via [ToT].

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Elegant integration of KV cache probing, LoRA gating, and single-token scalar regression.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across 4B to 32B models on difficult competition benchmarks with extensive ablations and trajectory visualizations.
  • Writing Quality: ⭐⭐⭐⭐⭐ Rigorous methodology, comprehensive mathematical proofs in the appendix, and well-structured empirical analyses.
  • Value: ⭐⭐⭐⭐⭐ Provides an ultra-efficient, highly practical blueprint for test-time scaling in reasoning LLMs.