BORA-BUDGET: Learning When to Think Under a Token Budget¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://anonymous.4open.science/r/BORA-BUDGET-E777
Area: LLM Reasoning
Keywords: Hybrid-thinking LLMs, Budgeted inference, Test-time compute allocation, Adoption gating, Harmful override mitigation
TL;DR¶
Addressing the excessive token cost and harmful override risks of uniformly invoking expensive reasoning in hybrid-thinking models, BORA-BUDGET introduces a seed-preserving control framework that predicts marginal gains and risks from cheap seed outputs, allocates thinking via AutoThreshold under explicit token budgets, and filters updates through an adoption gate.
Background & Motivation¶
Recent advances in large language models demonstrate that deliberate test-time reasoning—via extended chain-of-thought, self-consistency sampling, and tree search—substantially enhances mathematical and logical problem solving. The emergence of hybrid-thinking architectures, such as Qwen3, makes deliberate reasoning directly configurable within a single model by providing both a fast, inexpensive non-thinking mode (/no_think) and an expensive, expressive thinking mode (/think). Consequently, the practical deployment dilemma shifts from whether a model can reason to determining precisely when expensive test-time reasoning is worth its token overhead and potential override risks.
Uniformly activating deliberate thinking yields strong accuracy but squanders immense token budgets on simple queries where cheap answers are already correct. Conversely, uniformly disabling thinking suppresses computational costs but leaves complex queries unsolved. Crucially, conventional cascade and routing frameworks evaluate success purely through aggregate accuracy, neglecting the pervasive risk of harmful overrides: cases where the cheap non-thinking seed was already correct, but invoking the expensive mode produces an erroneous answer. Existing test-time budget controllers primarily govern reasoning trace lengths or search depths rather than identifying which problem instances actually possess positive marginal utility relative to the seed.
This paper tackles the challenge by formulating hybrid-thinking inference as a seed-preserved budgeted opportunity allocation problem. A controller first generates a cheap seed answer as a zero-cost default baseline, triggers expensive thinking only when the predicted marginal improvement exceeds token costs and override penalties, and enforces an explicit adoption gate before overwriting the seed. Core idea: recast hybrid-thinking deployment into budgeted opportunity allocation with seed preservation, predicting marginal helpfulness, harmful override risk, and token costs from lightweight seed-state features, adaptively triggering thinking under total token budgets via an AutoThreshold policy, and decoupling triggering from adoption to safeguard correct baseline answers.
Method¶
Overall Architecture¶
BORA-BUDGET structures inference around four synchronized steps: obtaining a zero-cost cheap seed, estimating marginal value and cost from seed-state features, adaptively allocating deliberate thinking under a total extra-token budget, and verifying candidates via an adoption gate. Given an input query, the frozen model produces a fast seed prediction alongside trace statistics. The controller estimates the probabilities of helpful rescue and harmful override, together with predicted token expenditures. Using AutoThreshold, the controller triggers deliberate thinking only for high-value candidates. Non-triggered queries retain their seed answers at zero additional cost, while triggered candidates are evaluated by an adoption gate that verifies output integrity and consistency, falling back to the seed if checks fail.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Problem x"] --> B["Stage 1: Seed-State Feature Extraction<br/>Obtain cheap seed trace and extract parser/statistical/consistency cues"]
B --> C["Stage 2: Opportunity and Cost Modeling<br/>Estimate helpfulness, harmfulness risk, and expected token cost"]
C --> D["Stage 3: AutoThreshold Budget Allocation<br/>Determine trigger threshold and candidate set under total token budget"]
D -->|Below Trigger Threshold| E["Default Accept Path<br/>Retain seed answer with zero extra cost"]
D -->|Meets Trigger Threshold| F["Stage 4: Adoption Gate Verification<br/>Generate thinking candidate and verify parsing/sign consistency"]
F -->|Gate Passed| G["Adopt Thinking Answer"]
F -->|Gate Rejected| H["Fallback to Retain Seed Answer"]
Key Designs¶
1. Seed-State Feature Extraction: Query-only difficulty is obscure -> Extract multi-view lightweight trace indicators -> Contextualize reliability without gold labels
Assessing problem difficulty and model uncertainty directly from raw problem text is unreliable, whereas traces generated by the cheap non-thinking mode provide immediate behavioral signals. Under the frozen model's /no_think policy, the system generates seed answer \(\hat{y}_i^0\) and trace \(\tau_i^0\), extracting five feature groups: answer parsing features (parse success, numerical value, sign, integer/decimal type, candidate diversity); trace statistics (token count, word/character lengths, near-cap flags, numeric span counts); seed self-consistency features (boxed answer counts, agreement between first and last answer, numerical spread across candidates, and correction cue words like "wait", "mistake", "check", "confirm"); problem-shape lexical indicators (algebra, geometry, number theory, extrema, and proof keywords); and optional independent sanity checkers. All features are extracted without accessing ground truth labels or invoking expensive thinking calls.
2. Opportunity and Cost Modeling: Unchecked thinking incurs harmful flips -> Estimate marginal helpfulness and harmfulness -> Opportunity score with explicit risk penalties
Invoking deliberate thinking is not universally beneficial: it can either correct an erroneous seed (Helpful: \(\hat{y}^0 \neq y \land \hat{y}^T = y\)) or corrupt an already correct seed (Harmful: \(\hat{y}^0 = y \land \hat{y}^T \neq y\)). To calibrate this trade-off, BORA-BUDGET trains gradient-boosted decision trees (HistGradientBoostingClassifier) to predict the conditional probabilities \(P_{\text{help}}(s)\) and \(P_{\text{harm}}(s)\) from the seed-state context \(s\), defining the opportunity score:
$\(v(s) = P_{\text{help}}(s) - \alpha P_{\text{harm}}(s)\)$
where \(\alpha\) is a harmful-flip penalty parameter (set to 2 in main experiments). In parallel, a regressor of identical architecture estimates the expected additional token cost \(\hat{C}(s) = \mathbb{E}[c^T - c^0 \mid s]\). This formulation penalizes override risks directly within the allocation objective rather than treating inference as an unconstrained accuracy pursuit.
3. AutoThreshold Budget Allocation: Fixed trigger ratios cannot meet token limits -> Solve Lagrangian shadow prices dynamically -> Streamlined allocation under total budget In production environments, computational resources are governed by aggregate token budgets \(B_{\text{extra}}\) rather than arbitrary static percentiles (e.g., Top-30%). BORA-BUDGET operationalizes a multiple-choice knapsack formulation using an AutoThreshold rule. The allocator identifies an optimal token price \(\lambda_B\) such that total predicted expenditures match the budget constraint, triggering deliberate thinking for all queries satisfying: $\(P_{\text{help}}(s) - \alpha P_{\text{harm}}(s) - \lambda_B \hat{C}(s) > 0\)$ In streaming settings, this corresponds to ranking instances by predicted marginal return per token \(r_i = \frac{v(s_i)}{\hat{C}(s_i)}\) until the budget is exhausted. The theoretical analysis establishes that for estimation errors bounded by \(\epsilon_G, \epsilon_H, \epsilon_C\), the regret of AutoThreshold against the oracle policy is bounded by \(2(\epsilon_G + \mu \epsilon_H + \lambda \epsilon_C)\), guaranteeing near-optimal budgeted utility.
4. Adoption Gate Verification: Extended reasoning introduces hallucinations -> Decouple triggering from answer adoption -> Strict gating to prevent harmful overwrites Deciding to pay for deliberate reasoning does not mandate blindly accepting the outcome. Deep thinking traces are prone to overthinking, subtle algebraic slips, or context-length exhaustion that corrupt correct initial seeds. BORA-BUDGET decouples paid generation from candidate adoption via a modular gate \(g(\hat{y}^0, \tilde{y}^T, \tau^T)\). The main gate requires that the thinking candidate is parseable and that the generation does not terminate near the maximum token limit. The strict gate adds a sign-consistency check between seed and thinking answers for scalar numerical predictions. When the gate rejects a candidate, the system falls back to the seed answer \(\hat{y}^0\), suppressing harmful overrides to near-zero levels.
Loss & Training¶
The controller relies on an offline two-stage calibration pipeline:
1. Model Calibration: Using rollout data from offline calibration folds (such as the leave-one-seed-out protocol across matched stochastic seeds), the system extracts seed-state vectors \(s\). For binary targets \(Y_{\text{help}} = \mathbb{I}(\hat{y}^0 \neq y \land \hat{y}^T = y)\) and \(Y_{\text{harm}} = \mathbb{I}(\hat{y}^0 = y \land \hat{y}^T \neq y)\), two HistGradientBoostingClassifier models are trained with max_iter=200, learning rate \(\eta=0.05\), and \(L_2\) regularization of 0.01. The cost model \(\hat{C}(s)\) is fitted using HistGradientBoostingRegressor with matching hyper-parameters.
2. AutoThreshold Calibration: Given the target token budget \(B_{\text{extra}}\), the controller performs binary search over the shadow price \(\lambda_B\) on the training rollouts to ensure the anticipated token spend satisfies \(\sum_i \hat{C}(s_i) \cdot \mathbb{I}(v(s_i) - \lambda_B \hat{C}(s_i) > 0) \approx B_{\text{extra}}\).
Key Experimental Results¶
Main Results¶
The primary evaluation is conducted on MATH500 across Qwen3-4B, Qwen3-8B, and Qwen3-14B using three matched stochastic seeds under the fixed-completion evaluator. The strict adoption gate is applied.
| Model | Method | Acc. (%) | Std | Avg. Tokens | Helpful Rescues | Harmful Flips |
|---|---|---|---|---|---|---|
| Qwen3-4B | /no_think | 72.93 | 0.46 | 605.6 | – | – |
| Qwen3-4B | Always /think@8k | 91.00 | 0.80 | 3754.9 | – | – |
| Qwen3-4B | Always /think@12k | 93.93 | 0.12 | 4080.9 | – | – |
| Qwen3-4B | Random top50 strict | 82.50 | 0.35 | 2644.4 | 145 | 1 |
| Qwen3-4B | GBDT top30 strict (Ours) | 90.60 | 0.53 | 2664.6 | 266 | 1 |
| Qwen3-4B | GBDT top50 strict (Ours) | 91.87 | 0.70 | 3418.1 | 286 | 2 |
| Qwen3-8B | /no_think | 73.40 | 0.53 | 610.2 | – | – |
| Qwen3-8B | Always /think@8k | 90.73 | 0.58 | 3867.8 | – | – |
| Qwen3-8B | Always /think@12k | 93.33 | 1.21 | 4293.1 | – | – |
| Qwen3-8B | Random top50 strict | 82.71 | 0.59 | 2755.1 | 140 | 0 |
| Qwen3-8B | GBDT top30 strict (Ours) | 90.53 | 1.29 | 2781.0 | 258 | 1 |
| Qwen3-8B | GBDT top50 strict (Ours) | 91.40 | 0.87 | 3533.5 | 272 | 2 |
| Qwen3-14B | /no_think | 75.60 | 0.72 | 607.0 | – | – |
| Qwen3-14B | Always /think@12k | 95.07 | 0.46 | 3902.7 | – | – |
| Qwen3-14B | Random top50 strict | 84.45 | 0.50 | 2557.5 | 134 | 3 |
| Qwen3-14B | Trace top50 strict | 92.33 | 0.70 | 3439.0 | 254 | 3 |
| Qwen3-14B | GBDT top30 strict (Ours) | 92.73 | 0.46 | 2685.3 | 257 | 0 |
| Qwen3-14B | GBDT top50 strict (Ours) | 93.13 | 0.61 | 3360.1 | 265 | 2 |
Ablation Study¶
The table below details the trigger selectivity diagnostic on Qwen3-8B MATH500 (Table 7) alongside the explicit token-budget sweep using AutoThreshold (Figure 3), demonstrating high precision at low budgets and smooth budget compliance.
| Configuration / Budget | Metric / Coverage Profile | Note |
|---|---|---|
| Trigger Diagnostic Top10 | Seed-wrong coverage: 34.84% / Seed-correct exposure: 1.00% | Ultra-low budget focuses almost entirely on errors |
| Trigger Diagnostic Top20 | Seed-wrong coverage: 68.92% / Seed-correct exposure: 2.27% | High return-per-token operating region |
| Trigger Diagnostic Top30 | Seed-wrong coverage: 88.72% / Seed-correct exposure: 8.72% | Captures ~89% of erroneous seeds with <9% correct exposure |
| Trigger Diagnostic Top50 | Seed-wrong coverage: 94.49% / Seed-correct exposure: 33.88% | High recall, with increased exposure of correct answers |
| Explicit Budget 400k (AutoThreshold) | Acc: 82.20% / Induced trigger rate: 10.40% / Avg. 1434.8 tokens | +8.80% over /no_think with minimal extra compute |
| Explicit Budget 1.0M (AutoThreshold) | Acc: 90.00% / Induced trigger rate: 28.53% / Avg. 2636.6 tokens | Reaches 90% accuracy; harmful overrides remain <1 |
| Explicit Budget 1.5M (AutoThreshold) | Acc: 91.47% / Induced trigger rate: 55.27% / Avg. 3619.0 tokens | Surpasses Always /think@8k (90.73%) at reduced token spend |
| Explicit Budget 1.63M (AutoThreshold) | Acc: 91.60% / Induced trigger rate: 62.40% / Avg. 3866.8 tokens | Approaches marginal return saturation point |
Key Findings¶
- Substantial Gains Over Random Triggering: Across all three parameter scales, GBDT top50 strict outperforms random top50 by 8.6 to 9.3 percentage points, confirming that marginal thinking value is predictable.
- High Selectivity at Low Budgets: The trigger diagnostic indicates that at Top30, the controller captures 88.72% of seed-wrong queries while touching only 8.72% of seed-correct instances, rescuing over 250 problems with minimal risk.
- Trace Length Competitiveness: In problem-disjoint evaluations (qid-disjoint splits), seed trace length serves as a surprisingly robust baseline (e.g., Qwen3-14B Trace top50 reaches 92.33%). The learned GBDT allocator consistently matches or slightly exceeds trace length, demonstrating robust non-random allocation across seeds.
Highlights & Insights¶
- Decoupled Paid Thinking and Answer Adoption: While standard routers commit irrevocably to whichever branch they invoke, BORA-BUDGET recognises that thinking models can hallucinate plausible yet incorrect alternatives, preventing regression through post-generation gating.
- Lagrangian AutoThreshold for Direct Budget Control: Replacing heuristic Top-k tuning with a shadow-price mechanism allows seamless compliance with hard token budgets across heterogeneous problem sets.
- Non-Invasive and Compute-Light Control Layer: The framework requires no internal weight modification or heavy auxiliary verifiers, relying entirely on lightweight decision trees and surface trace statistics.
Limitations & Future Work¶
- Task-Dependent Calibration: Opportunities and costs must currently be calibrated per task distribution; cross-task transfer to out-of-domain benchmarks like GPQA Diamond shows uneven efficacy.
- Wrong-to-Wrong Drift Remains Unchecked: The adoption gate effectively stops seed-correct to final-wrong degradation, but cannot detect when an incorrect seed is replaced by an equally erroneous thinking candidate.
- Underperforming Multi-Budget Arm Selection: While K-armed budget allocation exhibits large oracle headroom, learned multi-budget selectors currently trail the reliable binary single-budget switch.
Related Work & Insights¶
- vs Cross-Model Routers (FrugalGPT, RouteLLM): Model routers dispatch across disparate external endpoints, whereas BORA-BUDGET manages intra-model test-time compute allocations between fast and deliberate modes of a single frozen hybrid system.
- vs In-Model Adaptive Thinking (Thinkless, AdaCoT): Rather than modifying model parameters or reward objectives to teach when to output thinking tokens, BORA-BUDGET acts as an external risk-mitigating orchestrator.
Rating¶
- Novelty: ⭐⭐⭐⭐ [Formulates hybrid reasoning as budgeted opportunity allocation with explicit override mitigation]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive cross-scale analysis, budget sweeps, ablation diagnostics, and theoretical bounds]
- Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical formulation, honest analysis of failure cases, and well-structured empirical exposition]
- Value: ⭐⭐⭐⭐⭐ [Highly practical for lowering inference costs and stabilizing reasoning model deployments]