Skip to content

Bag-of-Tokens Tuning: Calibrating LLM Logits with Contextual Probabilities

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/qiuchenmeng/BoTTuning
Area: LLM Reasoning
Keywords: Uncertainty Calibration, Context-Logit Gap, Bag-of-Tokens Tuning, Soft-label Supervision, LLM Sampling

TL;DR

Addressing the severe Context-Logit Probability Gap where large language models fail to align their output logits with probability distributions explicitly specified in context, this paper proposes Bag-of-Tokens (BoT) Tuning—a lightweight, synthetic soft-label alignment method that directly supervises output distributions via KL-divergence minimization, delivering dramatic sampling fidelity gains alongside consistent improvements in downstream QA accuracy and uncertainty calibration.

Background & Motivation

In modern decision-making pipelines and probabilistic reasoning frameworks, the normalized output logits of large language models are routinely interpreted as confidence scores or utilized to simulate stochastic sampling processes. However, state-of-the-art LLMs consistently suffer from profound calibration errors, manifesting as severe overconfidence and substantial distributional divergence. Their raw output probabilities frequently fail to reflect either empirical answer correctness or numeric probability distributions explicitly presented in prompt contexts. More critically, when prompted to act as random samplers—such as performing coin flips, rolling dice, or drawing marbles from categorical collections—LLMs exhibit strong intrinsic logit biases (e.g., favoring the first presented choice or high-frequency vocabulary items), failing dramatically to replicate even elementary specified target distributions.

This systemic breakdown stems directly from what the authors identify as the Context-Logit Probability Gap: a fundamental disconnect between contextual probabilities conveyed in natural language and the model's physical next-token logits. Conventional post-hoc calibration techniques, most notably Temperature Scaling (TS) and its feature-conditioned adaptive variants, attempt to rectify miscalibration by rescaling output distribution sharpness at inference time. Nevertheless, because such monotonic transformations preserve the relative rankings of logits, they cannot correct underlying ranking errors; furthermore, real-world models simultaneously exhibit mixed patterns of over-estimation and under-estimation across diverse tokens, rendering global rescaling fundamentally inadequate. Conversely, existing fine-tuning paradigms for uncertainty calibration rely almost universally on hard-label supervision. Hard-label cross-entropy enforces discrete, winner-take-all updates on single target tokens rather than communicating full-spectrum probability densities, entirely bypassing the core challenge of closing the continuous context-logit discrepancy.

Recognizing that neither inference-time heuristics nor discrete hard labels address the root cause, the authors re-examine how language models internalize and express probabilistic specifications. They observe that synthetic contexts can pair arbitrary bags of candidate tokens with explicit numerical distributions, providing a direct mechanism to supply dense, soft supervision over candidate token spaces without requiring costly human annotations. Core idea: construct synthetic Bag-of-Tokens (BoT) data that explicitly encode candidate token bags and target probability distributions, aligning predicted next-token logits with in-context distributions via soft-label KL-divergence minimization to eliminate the Context-Logit Probability Gap.

Method

Overall Architecture

The Bag-of-Tokens (BoT) Tuning framework comprises three coordinated stages: constructing a robust candidate vocabulary pool covering both single-token and multi-token vocabulary units; generating diverse continuous probability distributions embedded in structured natural language context templates; and enforcing token-level distributional alignment at a dedicated prediction prefix position via KL-divergence minimization jointly optimized with general question-answering supervision.

The end-to-end conceptual flow and optimization pipeline are illustrated below:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Candidate Pool Construction & Bag Sampling<br/>High-frequency single tokens & HellaSwag multi-token words"] --> B["Diverse Contextual Probability Distribution Generation<br/>Bounded uniform maximum sampling & Dirichlet distributions"]
    B --> C["Target Prediction Position via Prefix<br/>Harmonized single-token and multi-token evaluation anchor"]
    C --> D["Dual-Task Soft-Label KL-Divergence Tuning<br/>Joint optimization of synthetic distributions and general QA"]
    D --> E["Calibrated Model Output<br/>Low-KL sampling fidelity + high-accuracy low-ECE reasoning"]

Key Designs

1. Candidate Pool Construction & Bag Sampling: Mitigating vocabulary fragmentation and multi-token probability distortion
Sampling candidate tokens uniformly from the entire model vocabulary inevitably introduces fragmented subwords, punctuation artifacts, and obscure lexical units that confound prompt interpretation and waste training capacity. To construct an interpretable and representative candidate pool, the authors first filter the tokenizer vocabulary to retain single tokens with Zipf frequency scores exceeding 6. To extend applicability beyond single subwords to natural multi-token words, common words appearing more than 30 times in the diverse HellaSwag corpus are incorporated into the pool. For any candidate word \(w_i\) composed of multiple subword tokens \((t_1, \dots, t_m)\), its composite probability is autoregressively evaluated through the exact chain rule: $\(p_\theta(w_i \mid x) = \prod_{\ell=1}^m p_\theta(t_\ell \mid x, t_{<\ell})\)$ This formulation ensures that the model learns continuous calibration not only across monolithic vocabulary embeddings but also across sequential multi-step autoregressive decoding paths. The resulting candidate vocabulary of approximately 2,000 words is partitioned into training and held-out test splits (7:3 ratio) to facilitate rigorous out-of-distribution evaluation.

2. Diverse Contextual Probability Distribution Generation: Preventing distributional memorization and mode collapse
To prevent the model from overfitting to idiosyncratic numerical patterns, the framework employs flexible, parameterized distribution generators. In the default construction, given \(K\) candidate tokens, a maximum probability mass \(m\) is sampled uniformly from \([1/K, 1-\epsilon)\) and assigned to a randomly designated candidate \(w_j\) (\(p_j = m\)), while the remaining probability mass \(1-m\) is distributed across the remaining \(K-1\) candidates subject to the constraint that no individual candidate exceeds probability \(m\). By sweeping \(m\) across its dynamic range, this mechanism synthesizes target distributions spanning from near-uniform configurations (\(m \approx 1/K\)) to highly peaked distributions (\(m \to 1\)). As an alternative, symmetric Dirichlet sampling \(p \sim \text{Dirichlet}(\alpha)\) is supported to systematically vary distributional entropy. These numerical distributions and candidate sets are then populated into seven distinct families of prompt templates (mathematical variables, marble bags, stochastic containers, and paraphrased task descriptions) that explicitly instruct the model to ignore lexical semantics and strictly adhere to numeric probabilities.

3. Target Prediction Position via Prefix: Eliminating contextual position shift and decoding divergence
In open-ended autoregressive decoding, unconstrained generation allows candidate tokens to appear at variable offsets, subjecting each candidate's logit evaluation to divergent prefix histories and distorting probability comparisons. To enforce a uniform evaluation anchor, BoT introduces a prefix conditioning mechanism. A standardized prompt prefix (e.g., chosen:, predict word:, or Randomly selected word:) is appended to shift the model's receptive field directly to the exact decision boundary where candidate generation occurs. The candidate logits are evaluated in parallel immediately following this prefix token. This design guarantees that every candidate's score is computed under identical attention context and KV-cache representations, completely neutralizing sequence-length disparities and ungrounded generation artifacts.

4. Dual-Task Soft-Label KL-Divergence Tuning: Regularizing distribution alignment against task-solving degradation
Tuning exclusively on synthetic probability puzzles severely compromises pre-trained commonsense reasoning and instruction compliance. To prevent representational degradation, BoT combines synthetic BoT examples with general question-answering datasets (BoolQ for realistic search queries and RACE for deep multi-step passage comprehension). Crucially, both data streams are mathematically unified under a single distributional alignment objective. For BoT samples, the target distribution \(q(w \mid x)\) represents the continuous contextual probabilities; for QA samples, multiple-choice options define the candidate space and the ground-truth answer is encoded as a one-hot distribution. The unified training loss minimizes the Kullback-Leibler (KL) divergence: $\(\mathcal{L}_{\text{KL}} = D_{\text{KL}}(q(w \mid x) \parallel p_\theta(w \mid x)) = \sum_{i=1}^K q(w_i \mid x) \log \frac{q(w_i \mid x)}{p_\theta(w_i \mid x)}\)$ The dense soft labels provide informative gradient signals that actively realign logit rankings and relative intervals, while the simultaneous QA objective serves as an empirical anchor that preserves general reasoning accuracy.

Loss & Training

The fine-tuning protocol updates all model parameters with an input sequence length of 4096. Training is executed using the AdamW optimizer with a learning rate of \(1 \times 10^{-5}\) for a single epoch. Per-device batch size is set to 1 with 8 gradient accumulation steps, yielding an effective batch size of 8. Training utilizes bfloat16 mixed precision accelerated with TensorFloat-32 (TF32). Fine-tuning on a hybrid dataset comprising 100K synthetic BoT examples alongside QA supervision completes in approximately 10 hours on a single 96GB NVIDIA RTX PRO 6000 GPU, demonstrating outstanding computational efficiency.

Key Experimental Results

Main Results

The authors conduct comprehensive empirical evaluations across five distinct sampling benchmarks (ID Token, OOD Token, Marble Color, Binomial Distribution, Uniform Distribution) and six established multiple-choice QA benchmarks (ARC-Easy, ARC-Challenge, CSQA, HellaSwag, MMLU, CEval), benchmarking Qwen3 (1.7B, 8B, 14B), LLaMA-3.1-8B, and Mistral-7B.

Table 1: Distributional alignment on sampling benchmarks measured by KL Divergence (↓ lower is better)

Model Family Method ID Token OOD Token Marble Color Binomial Dist. Uniform Dist.
Qwen3-1.7B Baseline 4.3148 4.3972 7.4190 15.3000 9.1580
Temperature Scaling (TS) 0.5015 0.4999 0.2019 0.5748 0.0061
BoT (Ours) 0.0831 0.0856 0.0129 0.2019 0.0060
Qwen3-8B Baseline 3.6550 3.7828 4.7568 2.8541 2.6322
Temperature Scaling (TS) 0.3193 0.3292 0.1335 0.2111 0.0127
Verbalized Rejection (VRS) 0.3688 0.3784 0.1592 0.4178 0.2067
BoT (Ours) 0.0405 0.0555 0.0362 0.0470 0.0071
Qwen3-14B Baseline 3.1105 3.2096 0.7248 4.5420 2.1391
Temperature Scaling (TS) 0.2663 0.2879 0.0479 0.3211 0.0151
BoT (Ours) 0.0932 0.0996 0.0039 0.0012 0.0108
LLaMA-3.1-8B Baseline 2.2192 2.1967 0.1084 2.0049 1.6495
Temperature Scaling (TS) 0.4937 0.4894 0.1023 0.4289 0.0073
BoT (Ours) 0.0103 0.0150 0.0028 0.0331 0.0070
Mistral-7B Baseline 2.3977 2.2302 2.1657 0.4477 2.9984
Temperature Scaling (TS) 0.5176 0.5352 0.1145 0.4289 0.0134
BoT (Ours) 0.0364 0.0394 0.0068 0.0122 0.0064

Table 2: Downstream QA accuracy (ACC % ↑) and calibration error (ECE % ↓) across benchmarks

Model Family Method ARC-E (Acc/ECE) ARC-C (Acc/ECE) CSQA (Acc/ECE) HellaSwag (Acc/ECE) MMLU (Acc/ECE) CEval (Acc/ECE)
Qwen3-1.7B Baseline 87.58 / 11.45 73.29 / 24.42 62.74 / 34.69 47.47 / 41.21 54.62 / 39.77 53.68 / 36.38
BoT 90.19 / 9.20 78.67 / 19.42 69.62 / 27.72 53.89 / 38.64 59.21 / 34.77 56.71 / 36.34
BoT w/ TS 90.19 / 1.84 78.67 / 3.34 69.62 / 3.17 53.89 / 3.57 59.21 / 4.47 56.71 / 3.86
Qwen3-8B Baseline 96.46 / 2.87 90.19 / 8.37 81.00 / 16.73 74.48 / 15.83 70.10 / 23.85 68.47 / 21.60
BoT 97.05 / 1.65 91.98 / 5.18 82.06 / 9.29 75.46 / 6.23 72.05 / 18.40 74.10 / 15.36
BoT w/ TS 97.05 / 0.37 91.98 / 2.18 82.06 / 1.95 75.46 / 0.88 72.05 / 5.50 74.10 / 4.15
LLaMA-3.1-8B Baseline 90.61 / 4.12 75.34 / 13.95 73.71 / 15.03 47.20 / 23.96 57.85 / 26.80 45.75 / 27.84
BoT 90.86 / 7.21 79.86 / 15.66 76.09 / 14.26 58.13 / 21.80 60.01 / 24.79 46.07 / 14.72
BoT w/ TS 90.86 / 3.29 79.86 / 3.25 76.09 / 4.50 58.13 / 2.22 60.01 / 7.76 46.07 / 2.16
Mistral-7B Baseline 89.10 / 8.90 75.43 / 20.86 70.51 / 26.08 54.50 / 30.70 58.56 / 31.44 38.79 / 41.36
BoT 88.17 / 2.80 78.58 / 3.63 67.81 / 5.31 54.00 / 3.42 58.61 / 11.53 40.62 / 15.64
BoT w/ TS 88.17 / 2.38 78.58 / 2.62 67.81 / 3.31 54.00 / 2.79 58.61 / 4.82 40.62 / 3.10

Ablation Study

The authors construct four targeted ablation variants to isolate the causal drivers of performance gains: (1) W/O BOT (excludes synthetic BoT data, training exclusively on QA data); (2) W/O QA (excludes QA data, training exclusively on synthetic BoT soft labels); (3) HARD LABEL (converts continuous BoT distributions into discrete, one-hot argmax target tokens); and (4) RANDOM (substitutes grounded soft labels with ungrounded random Dirichlet vectors to test whether gains stem merely from output entropy increase).

Table 3: Ablation performance and training dynamics (evaluated on Qwen3-8B sampling and QA tasks)

Ablation Configuration Sampling KL Divergence (Marble Color) ↓ Downstream QA Accuracy (CSQA) ↑ Probabilistic Calibration (Brier Score) ↓ Core Mechanism & Behavioral Manifestation
Full BoT Model 0.0362 82.06% 0.3526 Soft labels align distribution; QA data preserves general accuracy
w/o BoT Supervision 4.2180 81.12% 0.3820 Discards distribution alignment; sampling KL degrades to baseline levels
w/o QA Supervision 0.0351 74.30% 0.4415 Preserves distribution fitting but causes severe catastrophic forgetting on QA
Hard Label Variant 3.1540 76.50% 0.4120 Single-token discrete updates fail to teach distributions and harm QA reasoning
Random Soft-Labels 1.8420 78.20% 0.3950 Arbitrary probability smoothing fails to instill structured logit alignment

Key Findings

  • Continuous Soft Labels are Essential: The failure of the Hard Label variant (high KL divergence and depressed QA accuracy) conclusively confirms that discrete single-token supervision cannot resolve distribution-level logit miscalibration; continuous probability masses are required to reshape internal logit intervals.
  • Data Scaling Shows Rapid Saturation: Sensitivity analyses indicate that as few as 5,000 synthetic BoT examples suffice to realize the vast majority of calibration gains and downstream accuracy boosts; expanding beyond 30,000 examples yields diminishing returns on QA benchmarks while marginally refining strict distributional KL.
  • Selective Prediction Gains without Coverage Loss: In selective prediction regimes where models abstain when confidence falls below calibrated thresholds, BoT substantially boosts F1 scores while expanding empirical coverage, proving that confidence improvements stem from genuine uncertainty awareness rather than aggressive abstention.

Highlights & Insights

  • Repurposing In-Context Numerical Semantics as Continuous Supervision: Rather than treating prompt-specified probabilities merely as passive conditioning text, the method converts them into active, backpropagatable soft-label supervision targets, closing the context-logit gap at negligible cost.
  • Restoring Post-Hoc Calibration Efficacy: By restructuring internal logit rankings that post-hoc scaling cannot alter, BoT provides a properly calibrated foundation, allowing downstream temperature scaling (BoT w/ TS) to achieve unprecedented ECE reductions (often below 1% to 2%).
  • Cross-Task Generalization to Open-Ended Mathematical Reasoning: On free-form mathematical reasoning benchmarks (SVAMP, GSM8K, ASDiv), combining BoT with Chain-of-Thought (CoT) simultaneously improves problem-solving accuracy and reduces calibration error, proving that logit calibration generalizes beyond categorical MCQA.

Limitations & Future Work

  • Opaque Parametric Mechanisms: While empirical convergence is established across benchmarks, the mechanistic changes occurring within internal attention layers and feedforward weight spaces during soft-label logit alignment remain unexplored.
  • Reliance on Fixed Decoding Prefixes: The pipeline utilizes structured prefix phrases to establish evaluation positions; extending dynamic anchor detection to unstructured conversational dialogs and unconstrained streaming decoding presents an open research challenge.
  • Exploration of Reinforcement Learning Alternatives: Because the alignment objective is formulated directly over KL divergence, future research could formulate distributional alignment as an intrinsic reinforcement learning reward signal, enabling self-calibrating capabilities without supervised fine-tuning.
  • vs Temperature Scaling & Post-Hoc Variants (Guo et al., 2017; Xie et al., 2024): Post-hoc methods only adjust global distribution temperature and cannot alter corrupted logit rankings; BoT fundamentally recalibrates logit ranks via fine-tuning, achieving up to an order-of-magnitude lower KL divergence.
  • vs Verbalized Rejection Sampling (Xiao et al., 2025): VRS operates exclusively on binomial distributions and incurs heavy inference latency via repeated stochastic re-sampling; BoT natively handles arbitrary \(K\)-ary distributions in a single forward pass.
  • vs Output Prefilling (Cappelletti et al., 2026): Output prefilling is an inference-time heuristic that provides inconsistent gains and fails when base prompts already impose rigid formatting; BoT modifies underlying model parameters, delivering robust, model-wide calibration gains.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Insightful formulation of the Context-Logit Probability Gap, elegantly resolved using synthetic soft-label alignment.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Extensive evaluation covering 5 sampling tasks, 6 MCQA benchmarks, and 3 open-ended mathematical reasoning datasets across multiple open-weight LLM families.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear pedagogical structure, rigorous problem formulation, and exceptionally thorough empirical analysis.
  • Value: ⭐⭐⭐⭐⭐ Offers a practical, low-cost training paradigm that simultaneously enhances LLM sampling fidelity, reasoning precision, and uncertainty calibration.