Draft-OPD: On-Policy Distillation for Speculative Draft Models¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: speculative decoding, draft model, on-policy distillation, error replay, lossless acceleration
TL;DR¶
Addressing the offline-to-inference mismatch where supervised fine-tuning (SFT) of speculative draft models quickly plateaus, Draft-OPD introduces on-policy distillation via target-assisted rollout, verification error replay, and acceptance-aware asymmetric loss, delivering over 5× lossless inference speedup.
Background & Motivation¶
Large language models have achieved extraordinary milestones across intricate reasoning, mathematics, and code synthesis tasks, yet their deployment is increasingly throttled by surging inference latency and high computational overhead during autoregressive generation. Speculative decoding provides an elegant, mathematically lossless remedy: a lightweight draft model rapidly proposes candidate token blocks, which are subsequently verified and corrected in parallel by the high-capacity target model. The achievable speedup hinges on the average acceptance length \(\tau\) per verification cycle. Contemporary state-of-the-art draft systems such as EAGLE-3 and DFlash rely predominantly on supervised fine-tuning (SFT) over static token trajectories generated beforehand by the frozen target model.
However, continuing offline SFT rapidly confronts a performance plateau: once initial convergence is attained, pouring in additional training FLOPs or dataset tokens yields no improvement in acceptance length, and can even induce regression. The fundamental cause of this ceiling is an offline-to-inference state mismatch. During offline SFT, the drafter is trained exclusively on pristine prefixes produced by the target model; during actual speculative inference, the verifier evaluates multi-token blocks proposed under the drafter's own policy. Any early speculative error casts subsequent drafting into unvisited out-of-distribution states, amplifying compounding prediction drift that triggers widespread rejections.
Directly applying on-policy distillation (OPD) to draft models encounters an acute operational dilemma. Conventional OPD presupposes that a student model can independently roll out coherent complete sequences. Yet training-based draft models (such as single-layer or shallow Transformer blocks sharing the target model's embedding and LM head) are engineered specifically for short-span parallel prediction; forcing them to roll out trajectories autonomously results in severe repetition and collapse. Conversely, if one employs naive target-assisted rollouts, standard speculative verification silently discards all rejected candidate tokens and substitutes them with target-verified continuations, effectively scrubbing the sequence back to the target distribution and completely discarding the most informative error signals. Core idea: use target-assisted rollouts to preserve global generation stability, record verification-exposed error positions to replay draft prefixes, and optimize the drafter using an acceptance-aware distillation objective that reinforces accepted tokens via forward KL while penalizing rejected tokens via position-decayed reverse KL.
Method¶
Overall Architecture¶
The Draft-OPD pipeline operates across three coordinated stages: First, a hybrid rollout process couples the lightweight drafter with the target verifier under standard speculative decoding, securing full-sequence semantic stability while systematically logging the starting index of each drafted block as an anchor. Second, an anchor replay stage re-evaluates the drafted token blocks from their recorded prefix contexts, prompting both the draft model and the frozen target model to score log-probabilities across both accepted and rejected tokens. Third, an acceptance-aware distillation loss partitions candidate tokens according to verification outcomes, applying forward KL to accepted tokens and exponentially decaying reverse KL to rejected tokens to update draft model parameters.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Prompts & Prefix Context"] --> B["Speculative Rollout with Error Collection<br/>Target ensures trajectory stability while logging anchors"]
B --> C["Anchor Replay for Log-Probability Computation<br/>Dual draft-target scoring on complete candidate blocks"]
C --> D["Acceptance-Aware Distillation Objective<br/>Forward KL on accepted tokens + Decayed reverse KL on rejected tokens"]
D --> E["Updated Lightweight Draft Model Policy"]
Key Designs¶
1. Speculative Rollout with Error Collection: Resolving Trajectory Collapse via Target Assistance
To circumvent the inability of shallow draft models to generate coherent long sequences independently, Draft-OPD conducts trajectory generation via active speculative decoding guided by the target verifier. At rollout step \(m\), with the verified prefix terminating at position \(a_m\), the draft model \(q_\phi\) proposes a candidate block \(d_m = (d_{m,1}, \dots, d_{m,K})\) of size \(K\) conditioned on \(c_m = (x, y_{\le a_m})\). The target model \(p_\theta\) evaluates the block in parallel and accepts the longest valid prefix of length \(r_m\). The initial block position \(a_m\) is explicitly bookmarked as an anchor before advancing, with the subsequent anchor advancing to \(a_{m+1} = a_m + r_m\). This design guarantees that the resulting global trajectory maintains authentic target distribution fidelity without degenerate repetition, while precisely cataloging all critical decision boundaries where the draft policy operated during inference. Furthermore, target hidden states generated along verified prefixes are cached and reused, eliminating redundant recomputation during downstream replay.
2. Anchor Replay for Log-Probability Computation: Reconstructing Teacher Supervision on Draft Errors
Standard speculative decoding routinely drops rejected tokens, completely erasing the failure modes required for on-policy learning. Draft-OPD resolves this by replaying candidate blocks from each saved anchor \(a_m\) after rollout completion. Beginning from context prefix \(c_m\), the exact draft block \(d_m\) is re-instantiated. For each token position \(d_{m,k}\), student and teacher token log-probabilities are evaluated synchronously: $\(\log q_{m,k}(d_{m,k}) = \log q_\phi(d_{m,k} \mid c_m, d_{m,<k})\)$ $\(\log p_{m,k}(d_{m,k}) = \log p_\theta(d_{m,k} \mid c_m, d_{m,<k})\)$ This replay mechanism exposes the frozen teacher model directly to the unverified, draft-generated token prefixes—including segments that failed speculative verification. Consequently, the teacher model provides direct token-level supervisory distributions over the draft model's self-induced failure states.
3. Acceptance-Aware Distillation Objective: Asymmetric Divergences for Reinforcement and Error Suppression
Verification outcomes naturally partition the candidate tokens into an accepted set \(\mathcal{I}_{\text{acc}} = \{(m,k) \mid 1 \le k \le r_m\}\) and a rejected set \(\mathcal{I}_{\text{rej}} = \{(m,k) \mid r_m < k \le K\}\). For accepted tokens, the draft model already matches the target well, making forward KL divergence ideal for matching the target distribution: $\(\mathcal{L}_{\text{acc}} = \frac{1}{|\mathcal{I}_{\text{acc}}|} \sum_{(m,k) \in \mathcal{I}_{\text{acc}}} D_{\text{KL}}(p_{m,k} \parallel q_{m,k})\)$ Mathematically, this objective is equivalent (up to an entropy constant) to minimizing target-weighted cross-entropy. For rejected tokens, however, the draft model has placed excessive probability mass on incorrect modes. Reverse KL divergence is deployed here due to its mode-seeking nature, directly penalizing the draft model's confident disagreements with the teacher: $\(\mathcal{L}_{\text{rej}} = \frac{1}{Z} \sum_{(m,k) \in \mathcal{I}_{\text{rej}}} w_k D_{\text{KL}}(q_{m,k} \parallel p_{m,k})\)$ Here, \(w_k = \gamma^{k-1}\) denotes an exponential position decay factor (\(\gamma \in (0, 1]\), defaulted to 0.8), normalized by \(Z = \sum_{(m,k) \in \mathcal{I}_{\text{rej}}} w_k\). This decay is motivated by the sequential nature of speculative verification: the earliest failed token is the single bottleneck terminating acceptance, whereas subsequent rejected tokens are conditioned on already-invalidated draft prefixes and carry diminishing supervisory utility. The final objective balances both components: $\(\mathcal{L}_{\text{Draft-OPD}} = \frac{\lambda_{\text{acc}} \mathcal{L}_{\text{acc}} + \lambda_{\text{rej}} \mathcal{L}_{\text{rej}}}{\lambda_{\text{acc}} + \lambda_{\text{rej}}}\)$
Loss & Training¶
In all main experiments, the loss weights are set to \(\lambda_{\text{acc}} = \lambda_{\text{rej}} = 1\), and positional decay is set to \(\gamma = 0.8\). The draft model is initialized from a checkpoint pre-trained with 6 SFT epochs, followed by 8 OPD epochs over a 16K prompt pool (composed of 2K GSM8K, 5K MATH, 4K AoPS, and 5K CodeAlpaca queries, using prompts exclusively while target model outputs are generated online). The maximum generation sequence length is capped at 4096 tokens for thinking mode and 2048 tokens for non-thinking mode. Training uses the AdamW optimizer with a learning rate of \(3 \times 10^{-4}\), a cosine learning rate schedule with a 5% warmup ratio, and a fixed block size of \(K=16\).
Key Experimental Results¶
Main Results¶
Draft-OPD was evaluated on the Qwen3 model family (Qwen3-4B, Qwen3-8B, and Qwen3-30B-A3B-Thinking) across mathematical reasoning, code generation, and multi-turn chat. All comparisons enforce a matched total training FLOPs budget against EAGLE-3 and DFlash baselines. The following table highlights performance under thinking mode (maximum 8192 generation tokens):
| Model | Method | GSM8K (0.0/0.6) | MATH-500 (0.0/0.6) | AIME25 (0.0/0.6) | MBPP (0.0/0.6) | HumanEval (0.0/0.6) | SWE-Lite (0.0/0.6) | MT-Bench (0.0/0.6) | Mean Speedup (0.0/0.6) | Mean \(\tau\) (0.0/0.6) |
|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3-4B | EAGLE-3 | 4.41× / 3.90× | 4.15× / 3.77× | 3.30× / 3.07× | 4.29× / 3.75× | 4.38× / 3.77× | 3.47× / 2.75× | 3.07× / 2.77× | 3.87× / 3.40× | 5.33 / 5.03 |
| Qwen3-4B | DFlash | 4.51× / 4.21× | 4.88× / 4.46× | 4.69× / 4.25× | 4.39× / 3.92× | 4.77× / 4.17× | 4.12× / 3.17× | 2.96× / 2.73× | 4.33× / 3.84× | 5.51 / 4.77 |
| Qwen3-4B | Draft-OPD | 5.31× / 4.73× | 5.55× / 4.91× | 5.28× / 4.56× | 4.85× / 4.21× | 5.17× / 4.40× | 4.66× / 3.38× | 3.18× / 2.87× | 4.86× / 4.15× | 5.96 / 5.13 |
| Qwen3-8B | EAGLE-3 | 4.58× / 4.13× | 4.45× / 4.09× | 4.01× / 3.57× | 4.55× / 3.93× | 4.46× / 3.95× | 3.38× / 2.83× | 3.02× / 2.83× | 4.06× / 3.62× | 5.64 / 5.33 |
| Qwen3-8B | DFlash | 4.67× / 4.23× | 5.01× / 4.57× | 4.77× / 4.34× | 4.50× / 3.85× | 4.71× / 4.12× | 3.72× / 2.88× | 3.01× / 2.76× | 4.34× / 3.82× | 5.19 / 4.66 |
| Qwen3-8B | Draft-OPD | 5.36× / 4.77× | 5.80× / 5.07× | 5.51× / 4.80× | 4.86× / 4.17× | 5.19× / 4.47× | 4.31× / 3.13× | 3.18× / 2.91× | 4.89× / 4.19× | 5.73 / 5.05 |
Under non-thinking mode (maximum 2048 generation tokens), Draft-OPD achieves a mean greedy speedup of 5.31× (\(\tau=6.60\)) on Qwen3-4B and 5.60× (\(\tau=6.57\)) on Qwen3-8B. Real-world serving benchmarks on SGLang under varying concurrency (1 to 32) demonstrate that Draft-OPD scales effectively: on Qwen3-30B-A3B-Thinking at concurrency 32, throughput reaches 5007 tok/s compared to 4462 tok/s for DFlash (a 12% improvement), increasing \(\tau\) from 5.44 to 5.95.
Ablation Study¶
A component ablation on Qwen3-4B (thinking mode enabled, greedy decoding) demonstrates that each algorithmic component is critical for achieving optimal speedups:
| Configuration / Variant | MATH-500 (Speedup / \(\tau\)) | HumanEval (Speedup / \(\tau\)) | MT-Bench (Speedup / \(\tau\)) | Note |
|---|---|---|---|---|
| Draft-OPD (Full) | 5.55× / 6.57 | 5.17× / 6.18 | 3.18× / 4.44 | Full error replay + asymmetric KL |
| w/o Weight Decay | 5.13× / 6.18 | 4.96× / 6.01 | 3.07× / 4.29 | Uniform weight across all rejected tokens |
| All-reverse KL | 5.11× / 6.14 | 4.94× / 5.98 | 3.08× / 4.33 | Penalizes modes even on valid accepted tokens |
| All-forward KL | 5.34× / 6.35 | 5.01× / 6.09 | 3.09× / 4.33 | Insufficient penalization of confident draft errors |
| Random Anchors | 5.04× / 6.08 | 4.99× / 6.07 | 2.96× / 4.24 | Replays from random states rather than verification failures |
| Naive Target Rollout | 5.06× / 6.07 | 4.80× / 5.93 | 3.02× / 4.25 | Discards rejected tokens; reduces to target SFT |
Key Findings¶
- Crucial Asymmetry in Objective Divergence: Relying exclusively on all-reverse KL drops MATH-500 speedup by 0.44×, whereas all-forward KL fails to suppress hallucinated draft modes. The partition into forward KL for accepted tokens and reverse KL for rejected tokens matches the underlying verification dynamics.
- Verification Failure as Primary Training Signal: Replacing error-located anchors with random sequence positions causes marked performance drops across all domains, verifying that post-training must focus specifically on verification-time failure states.
- Primacy of the First Rejected Token: Analysis of the First-rejected-only variant (Table 5) reveals an average \(\tau\) of 5.94 across seven benchmarks, virtually identical to the 5.95 achieved with complete exponential decay. This demonstrates that the first point of divergence dictates overall block acceptance, providing a clear path toward truncated replay for computational savings.
Highlights & Insights¶
- Turning Verification Rejections into On-Policy Signals: Rather than discarding rejected draft tokens as computational waste, Draft-OPD recycles them as high-leverage negative samples for on-policy alignment.
- Asymmetric KL Formulations Aligned with Verification: Applying forward KL to preserve distribution coverage on verified states while deploying reverse KL to penalize overconfident incorrect modes establishes a coherent optimization framework for speculative models.
- Plug-and-Play Lossless Inference Acceleration: Operates strictly as a post-training scheme for the draft model without altering target model weights or verification guarantees, offering seamless integration with serving frameworks like SGLang.
Limitations & Future Work¶
- Training Rollout Sequence Truncation: Due to GPU memory and compute constraints, training rollouts are capped at 4096 tokens, leaving the late-stage reasoning dynamics of ultra-long chains (e.g., >8192 tokens) partially uncovered during distillation.
- Requirement of White-Box Teacher Access: Generating teacher log-probabilities on draft-induced prefixes requires direct access to the target model's logits and internal states, precluding application to closed-source API-only models.
- Architectural Scope: The core empirical investigation centers on the DFlash non-autoregressive drafting architecture and Qwen3 target models; broader validation on tree-based speculative decoding (e.g., Medusa, EAGLE-3) across diverse model families remains an open direction.
Related Work & Insights¶
- vs. EAGLE-3 / DFlash: While prior methods achieve substantial speedups via specialized draft architectures trained with offline SFT, they plateau due to exposure bias. Draft-OPD unlocks continued scaling beyond this plateau, yielding 13%–23% speedup gains under matched compute budgets.
- vs. Standard On-Policy Distillation (OPD): Conventional OPD targets standalone autoregressive models capable of full sequence rollouts. Draft-OPD bridges this concept to speculative decoding by introducing target-assisted rollouts paired with error-state anchor replay.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates an effective on-policy distillation methodology tailored specifically to speculative draft models.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous comparisons across 3 model scales, 7 diverse benchmarks, concurrency stress testing, and granular ablations.
- Writing Quality: ⭐⭐⭐⭐⭐ Lucid problem formulation, clear mathematical exposition, and transparent empirical analyses.
- Value: ⭐⭐⭐⭐⭐ Unlocks substantial efficiency gains for long-form reasoning models in production serving infrastructure.