Skip to content

JuDi: Revisiting Judge Decoding from First Principles via Training-Free Distributional Divergence

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/sunshy-1/JuDi
Area: LLM Evaluation
Keywords: speculative decoding, judge decoding, KL divergence, training-free verification, logic-pivoting tokens

TL;DR

Uncovering that token criticality learned by costly supervised judges is fundamentally encoded in draft-target distributional divergence, JuDi establishes a training-free verification mechanism via token-level KL divergence with high-confidence fallback, matching or outperforming learned judges while eliminating supervision bottlenecks.

Background & Motivation

Large language models deliver outstanding performance across multi-step mathematical reasoning and algorithmic code generation, yet their real-world serving remains severely bottlenecked by the high latency and memory bandwidth demands of sequential autoregressive decoding. Speculative Decoding (SD) has emerged as the standard acceleration paradigm by utilizing a lightweight draft model to propose blocks of candidate tokens that are subsequently verified in parallel by the target model. However, standard speculative decoding strictly enforces a lossless criterion to guarantee exact distributional sampling, rejecting any draft token that deviates from the target model's sampling space. This verification is overly conservative, rejecting harmless semantic equivalents (such as "6+1=7" versus "6 plus 1 equals 7") and sharply capping the achievable wall-clock speedup.

To break past this conservative bound, recent research has pivoted towards Judge Decoding, a lossy speculative paradigm that deploys a learned classifier (a "judge") to inspect the validity of drafted tokens beyond raw lexical matching. Nonetheless, existing judge decoding frameworks rely on expensive and noisy supervision signals. Frameworks based on human annotation suffer from external alignment mismatch and coarse span labeling, where marking an entire erroneous text span obscures the specific logic-pivoting tokens that trigger the reasoning divergence. Automated heuristic mining methods like AutoJudge address human labor by checking whether token substitutions flip final answers, but at prohibitive computational costsโ€”requiring over 2,700 GPU hours to mine supervision on GSM8K with a 70B modelโ€”while suffering from severe stochastic instability where only 25.4% of mined tokens maintain consistent criticality across rollouts.

This dilemma between prohibitive labor costs and noisy heuristic mining calls for rethinking judge decoding from first principles. By observing that an LLM's internal representations naturally signal mistakes through latent error-detection dynamics, we ask whether the criticality score learned by complex classifiers is fundamentally an external artifact or an internal property of the model pair. Core idea: The token criticality captured by costly supervision is intrinsically encoded in the draft-target distributional divergence, with linear judges and Kullback-Leibler (KL) divergence sharing identical pairwise logit-gap primitives, thereby enabling a simple, training-free, and robust verification mechanism that entirely bypasses dataset mining and classifier training.

Method

Overall Architecture

JuDi introduces an annotation-free, training-free judge decoding pipeline that dynamically relaxes token verification based on the distributional divergence between the draft and target models. In each decoding cycle, the draft model proposes a candidate block of \(\gamma\) tokens while caching draft logits; the target model verifies all candidate positions in a single parallel forward pass; JuDi then evaluates each position using a confidence fallback safeguard and full-vocabulary KL divergence to decide whether to accept the draft token, accept the target greedy prediction, or truncate and re-draft.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Prompt and Context"] --> B["Draft Candidate Block<br/>MD generates ฮณ tokens and stores draft logits"]
    B --> C["Parallel Target Verification<br/>MT parallel forward pass yielding logits and greedy tokens"]
    C --> D["Token-Level Verification & Confidence Fallback<br/>If max target probability exceeds 0.9, enforce exact match"]
    D --> E["Training-Free KL Divergence Check<br/>Compute full-vocabulary divergence D_KL(P_T || P_D)"]
    E -->|KL below threshold or greedy match| F["Accept Candidate & Append Bonus Token"]
    E -->|KL exceeds threshold & mismatch| G["Reject Candidate & Truncate for Re-drafting"]

Key Designs

1. Structural Correspondence Theorem: Grounding Learned Classifiers in Logit Primitives

Prior works view judge decoding as an opaque classification problem over hidden embeddings. Let \(z_t = W_t h_t + b_t\) and \(z_d = W_d h_d + b_d\) represent the target and draft logits, respectively. For any vocabulary index pair \((i, j)\), define the pairwise logit-gap difference: $\(\Delta_{ij}(x) := \left(z_t(i) - z_t(j)\right) - \left(z_d(i) - z_d(j)\right)\)$ This affine primitive captures how the target model shifts relative token preferences compared to the draft model. By applying a second-order Taylor expansion around the draft logits via the Fisher information metric \(F(P_d) = \text{diag}(P_d) - P_d P_d^\top\), the Kullback-Leibler divergence decomposes into a Fisher-weighted quadratic aggregation of these primitives: $\(D_{\text{KL}}(P_t \parallel P_d) \approx \frac{1}{4} \sum_{i=1}^V \sum_{j=1}^V P_d(i) P_d(j) \Delta_{ij}(x)^2\)$ Furthermore, because vocabulary size \(V\) vastly exceeds the hidden state dimension \(d\) (\(V \gg d\)), the primitive direction vectors form an overcomplete frame spanning \(\mathbb{R}^{2d}\). Consequently, any linear classifier trained on concatenated representations \(x = [h_t; h_d]\) equivalently acts as a linear superposition over the exact same primitives: \(\text{Cls}(x) \equiv \sigma\left(\sum_{i,j} \beta_{ij} \Delta_{ij}(x) + b'\right)\). This proves that supervised judges and KL divergence operate on identical underlying signals, differing only in whether they apply linear or quadratic decision boundaries.

2. Training-Free Full-Vocabulary Divergence Verification: Replacing Heuristic Mining with Distributional Disagreement

To eliminate expensive rollout mining, JuDi directly reuses the logits produced during the draft and target forward passes to perform parameter-free verification. At each position \(i \in \{1, \dots, \gamma\}\), the discrete KL divergence between the target distribution \(P_T\) and draft distribution \(P_D\) is computed over the entire vocabulary \(\mathcal{V}\): $\(D_{\text{KL}}(P_T \parallel P_D) = \sum_{v \in \mathcal{V}} P_T(v) \log \frac{P_T(v)}{P_D(v)}\)$ When draft token \(d_i\) disagrees with target greedy prediction \(t_i = \arg\max z_T^{(i)}\), JuDi accepts \(d_i\) if \(D_{\text{KL}}(P_T \parallel P_D) < \tau\) for a pre-calibrated threshold \(\tau\). If the divergence is small, the target model's overall probability landscape across competing tokens remains concordant with the draft model, signaling a benign, non-semantic lexical shift (such as capitalization or synonym choice). Conversely, an abrupt divergence spike accurately flags a logic-pivoting error, triggering immediate rejection and correction.

3. Top-1 Confidence Fallback: Zero-Cost Safeguard for High-Certainty Regimes

To ensure absolute fidelity when the target model is unambiguous, JuDi implements a lightweight confidence fallback. When the target model's top probability exceeds a strict confidence threshold \(\tau_c = 0.9\): $\(\max_{v \in \mathcal{V}} P_T(v) > \tau_c\)$ the system bypasses the KL divergence relaxation and defaults strictly to exact verification (accepting \(d_i\) if and only if \(d_i = t_i\)). This safeguard requires no learnable parameters and introduces negligible overhead (a single scalar comparison). Empirical profiling shows that this fallback triggers on fewer than 5% of token-level decisions (0.1โ€“0.4 times per query on average), acting as a reliable safety valve that protects sharp, definitive factual commitments without restricting decoding speed elsewhere.

A Worked Example

Consider an arithmetic problem from GSM8K: "Marissa hikes 12 miles. She walks 4 miles in 1 hour, then 2 miles in 1 hour. What speed must she walk for the rest of the hike to average 4 mph overall?" When calculating the initial duration, the target model expects Time = 4 / 1, whereas the draft model introduces a calculation blunder, outputting Time = 4 / 4. This semantic pivot causes the target model's distribution to sharply diverge from the draft distribution, resulting in a large token-level KL divergence of 3.5781 (well above threshold \(\tau = 0.5\)). JuDi correctly identifies this as a critical error, rejecting 4 and replacing it with 1. Later in the solution, the target model outputs Average Speed (capitalized) while the draft proposes Average speed (lowercase). Because this variation is purely surface-level, both models assign near-identical relative probabilities across subsequent continuation tokens, yielding a tiny KL divergence of 0.0469. JuDi safely accepts Average speed, preventing an unnecessary pipeline stall and redrafting step.

Key Experimental Results

Main Results

Evaluated on GSM8K and LiveCodeBench across multiple model pairs (Llama-3.2-1B / Llama-3.1-8B, Llama-3.1-8B / Llama-3.1-70B, and Qwen3-0.6B / Qwen3-8B), JuDi matches or exceeds the performance of the supervised AutoJudge while outperforming Top-K and entropy-based baselines. Real-world deployment inside vLLM confirms tangible end-to-end wall-clock speedups.

Model Pair & Benchmark Method Accuracy Mean Accepted Tokens (MAT) Real Speedup (vLLM)
Llama-1B / 8B (GSM8K) Vanilla SP (Exact baseline) 82.50% 10.61 1.00ร— (38.40 tok/s)
Llama-1B / 8B (GSM8K) AutoJudge (Supervised classifier) 80.29% 16.18 1.17ร— (45.06 tok/s)
Llama-1B / 8B (GSM8K) JuDi (KL Divergence, ฯ„=0.3) 80.96% 17.35 1.18ร— (45.35 tok/s)
Llama-1B / 8B (GSM8K) JuDi (KL Divergence, ฯ„=0.9) 77.13% 19.65 1.60ร— (61.45 tok/s)
Llama-8B / 70B (GSM8K) Vanilla SP (Exact baseline) 93.00% 13.35 1.00ร— (24.19 tok/s)
Llama-8B / 70B (GSM8K) AutoJudge (Supervised classifier) 92.54% 29.55 1.22ร— (29.55 tok/s)
Llama-8B / 70B (GSM8K) JuDi (KL Divergence, ฯ„=0.4) 91.94% 30.55 1.24ร— (30.10 tok/s)
Llama-8B / 70B (GSM8K) JuDi (KL Divergence, ฯ„=1.4) 90.27% 35.10 1.45ร— (35.10 tok/s)
Qwen3-0.6B / 8B (GSM8K) Vanilla SP (Long reasoning) ~88.0% 6.91 1.00ร—
Qwen3-0.6B / 8B (GSM8K) JuDi (Long reasoning, ฯ„=0.6) ~86.0% 19.15 (+2.77ร—) โ€”

Under domain shifts, when evaluating on MATH-500-Hard and MMLU-Pro using classifiers trained solely on GSM8K, AutoJudge exhibits significant performance degradation due to task overfitting. In contrast, JuDi maintains superior robustness: on MATH-500-Hard, JuDi increases MAT from 13.93 to 32.02 (1B/8B) with under 1% accuracy drop. On LiveCodeBench, where strict syntactic precision is paramount, Top-K and entropy baselines frequently generate non-executable code, whereas JuDi achieves a 104.0% MAT increase (8.83 \(\to\) 18.01) with preserved execution validity.

Ablation Study

The ablation analysis examines vocabulary truncation strategies, decision granularity, and the operational behavior of the confidence fallback.

Configuration (GSM8K, 1B/8B) Accuracy MAT Latency / Architectural Mechanism
Full-vocabulary KL (Full-vocab, ฯ„=0.3) 87.50% 13.33 Continuous streaming memory access; lowest latency (0.068 ms)
Truncated Top-20 KL (ฯ„=0.3) 86.00% (โ†“1.50%) 13.45 Branch-intensive sorting & gather breaks coalescing (0.127 ms)
Truncated Top-200 KL (ฯ„=0.3) 86.38% (โ†“1.12%) 13.35 Density threshold boundary error causes false acceptances
Step-level KL Aggregation (MATโ‰ˆ13) 85.62% (โ†“1.88%) 13.20 Mean pooling dilutes isolated logic-pivoting token signals
Step-level KL Aggregation (MATโ‰ˆ20) 78.62% (โ†“4.50%) 19.80 Coarse block rejections cause collateral discarding of valid tokens
Without Confidence Fallback (ฯ„=0.6) 86.62% 17.21 Trigger rate is 1.1%; minimal impact on aggregate performance
Full JuDi with Fallback (ฯ„=0.6) 85.50% 17.35 Zero-cost safety valve guarding rare high-certainty divergences

Key Findings

  • Counter-Intuitive Latency of Vocabulary Truncation: Restricting KL calculation to Top-K tokens theoretically cuts operations from \(\mathcal{O}(V)\) to \(\mathcal{O}(K)\). However, in practice on modern GPU architectures, Top-K selection and index gathering involve branch-heavy, uncoalesced memory access. As a result, full-vocab streaming KL executes 1.8ร— faster in wall-clock time (0.068 ms vs. 0.126 ms) while avoiding estimation error.
  • Granularity Matters: Moving from token-level to step-level (newline-delimited) KL aggregation degrades accuracy by 2%โ€“5% at comparable MAT. Aggregation dilutes the sharp discrepancy of logic-pivoting tokens while collaterally rejecting valid prefix tokens when a threshold is breached.
  • Superior Scaling on Long-Chain Reasoning: In reasoning-focused models like Qwen3, where reasoning paths are lengthy and verbose, JuDi achieves an outstanding 2.77ร— MAT improvement (6.91 \(\to\) 19.15), proving especially potent for chain-of-thought generation.

Highlights & Insights

  • First-Principles Demystification: Proves that the "criticality" signal learned through labor-intensive annotation or thousands of GPU hours of rollout mining is mathematically equivalent to the intrinsic divergence of logit primitives, turning a supervised engineering problem into a clean statistical evaluation.
  • Zero Cold-Start & Universal Deployment: Requires zero training samples, no specialized classifier weights, and no domain-specific adaptation, enabling immediate integration into serving runtimes like vLLM.
  • Systems-Aware Algorithmic Choice: Highlights the importance of hardware-aware algorithm design in deep learning, demonstrating how memory bandwidth patterns (streaming element-wise vs. irregular indexing) dictate actual throughput over theoretical operation counts.

Limitations & Future Work

  • Author-Acknowledged Limitations: Due to compute constraints, the method has not yet been validated on ultra-large 405B-scale models. Furthermore, the current vLLM integration relies on standard PyTorch routines without custom fused Triton kernels, leaving further latency gains on the table.
  • Intrinsic Consensus Constraint: If both the draft model and target model share the same systematic bias or hallucination, their low KL divergence will accept the erroneous token. This consensus limitation is an inherent boundary of the speculative decoding framework itself, which guarantees distributional fidelity rather than objective factuality.
  • Future Directions: Developing dedicated fused C++/Triton kernels combining Log-Softmax and KL reduction directly within the target verification pass; extending distributional divergence verification to tree-structured draft verification and diffusion language models.
  • vs Vanilla Speculative Decoding (Chen et al., 2023; Leviathan et al., 2023): Standard SD rejects any token not matching the exact target distribution. JuDi relaxes this constraint to accept semantically consistent tokens, boosting MAT by 60%โ€“130% and delivering 1.3ร—โ€“1.6ร— wall-clock speedups.
  • vs Supervised Judge Decoding (Bachmann et al., 2025; Garipov et al., 2025): Bachmann et al. rely on costly human annotations with overextended span errors, while AutoJudge consumes 2,700+ GPU hours mining rollout labels. JuDi proves these classifiers merely approximate distributional divergence, matching their speed-quality tradeoff without any training or vulnerability to domain shifts.
  • vs Entropy-based Relaxation (Wang et al., 2025): Relying solely on target or draft entropy causes steep accuracy degradation on precision-sensitive tasks like code generation. JuDi measures joint draft-target alignment divergence, maintaining robust accuracy across coding and math benchmarks.

Rating

  • Novelty: โญโญโญโญโญ Elegant first-principles proof showing that learned linear judges and KL divergence share the exact same logit primitives.
  • Experimental Thoroughness: โญโญโญโญโญ Comprehensive evaluation across GSM8K, MATH-500-Hard, LiveCodeBench, MMLU-Pro, Qwen3, with real vLLM wall-clock profiling and GPU memory kernel analysis.
  • Writing Quality: โญโญโญโญโญ Rigorous theoretical derivation, exceptionally clear exposition, and seamless progression from motivation to empirical proof.
  • Value: โญโญโญโญโญ Delivers immediate, practical engineering value for LLM inference serving by removing the data and training bottlenecks of judge decoding.