Skip to content

Draft-OPD: On-Policy Distillation for Speculative Draft Models

Conference: EMNLP 2026
Paper: ACL Anthology
Area: Model Compression
Keywords: Speculative Decoding, On-Policy Distillation, Draft Model, Exposure Bias, Lossless Acceleration

TL;DR

Addressing the rapid performance plateau of speculative draft models trained via offline supervised fine-tuning (SFT) on static target trajectories, Draft-OPD introduces on-policy distillation using target-assisted rollouts with error-position replay and an acceptance-aware bidirectional KL objective, delivering up to 5.60× lossless end-to-end inference speedup on Qwen3 models.

Background & Motivation

Speculative decoding (SD) has emerged as an indispensable cornerstone for accelerating large language model inference by pairing a heavy target model with a lightweight draft model that proposes multiple candidate future tokens verified in parallel. By strictly matching the target model's output distribution, speculative decoding provides mathematically lossless acceleration. Modern training-based draft models, epitomized by EAGLE-3 and DFlash, achieve remarkable decoding speeds by training custom draft modules on target-generated trajectories via offline supervised fine-tuning (SFT), minimizing multi-token prediction error against teacher-forced targets.

Despite this progress, offline SFT for draft models quickly encounters an intractable performance plateau. Empirical observations reveal that after a brief initial warm-up, the mean accepted token length \(\tau\) on evaluation benchmarks ceases to improve; dedicating additional training compute or scaling offline trajectory volume merely fluctuates around a static ceiling, and continuing SFT on out-of-domain data can even degrade acceptance rates. The root cause lies in a severe offline-to-inference distribution mismatch: in offline SFT, the drafter is conditioned exclusively on flawless prefixes generated by the target model, whereas during speculative decoding, it must propose token blocks conditioned on its own previously proposed states. Once the draft model deviates slightly from the target distribution, it enters compounded error states that it was never exposed to during offline training.

Adapting on-policy distillation (OPD) to draft models presents a fundamental dilemma. Mainstream training-based draft modules (such as lightweight 5-8 layer Transformers) depend heavily on the target model's hidden states and are designed to predict short multi-token blocks rather than function as standalone autoregressive generators. Forcing draft modules to roll out entire sequences autonomously leads to rapid degeneration, repetitive loops, and meaningless sequences that render teacher distillation futile. Conversely, naive target-assisted rollouts maintain grammatical coherence but restore the target distribution, discarding rejected candidate tokens where the draft policy's most informative error signals reside. This paper resolves this dilemma by pairing target-assisted rollouts with anchor-guided error replay. Core idea: stabilize sequence rollout using target verification while recording the exact starting anchors of speculative failures, then replay draft-generated blocks to extract token-level teacher-student log-probabilities under an acceptance-aware distillation objective that reinforces accepted tokens and penalizes draft-induced failure modes.

Method

Overall Architecture

The Draft-OPD pipeline is structured to circumvent the instability of student-only generation while preserving the policy-specific negative signals generated during speculative drafting. The workflow operates through three coordinated stages: 1. Rollout with Error-Position Collection: Speculative drafting interacts with the target model to generate stable text completions. The starting position of every proposed candidate block is indexed as an anchor. The target verifier accepts the longest valid prefix and rejects the remaining suffix, naturally partitioning proposed tokens into accepted and rejected sets. 2. Replay for Log-Probability Computation: From each recorded anchor context, the system replays the exact candidate token block originally proposed by the draft model. Both the student (drafter) and teacher (target model) compute conditional token-level log-probabilities across both accepted and rejected positions, reusing cached target hidden representations computed during rollout. 3. Acceptance-Aware Distillation Objective: Replayed tokens are routed to asymmetric loss formulations: forward KL divergence aligns the drafter with the target on accepted states, while reverse KL divergence penalizes the drafter's false-positive modes on rejected states, modulated by an exponential positional decay.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Prompt x"] --> B["Rollout with Error-Position Collection<br/>Target-assisted speculative decode, record block anchors"]
    B --> C["Replay for Log-Probability Computation<br/>Re-evaluate candidate blocks, fetch teacher/student log-probs"]
    C --> D["Acceptance-Aware Distillation Objective<br/>Forward KL on accepted tokens, reverse KL on rejected suffix"]
    D --> E["Update Draft Model Parameters<br/>Break SFT plateau, scale accepted length tau"]

Key Designs

1. Rollout with Error-Position Collection: Overcoming Student Degeneration while Anchoring Failure States

Lightweight training-based draft heads lack standalone long-horizon autoregressive capability; attempting autonomous generation leads to degenerate repetitive text, destroying distillation signal quality. Draft-OPD leverages the frozen target model \(p_\theta\) and the draft model \(q_\phi\) in a speculative decoding loop to generate rollouts \(y = (y_0, \dots, y_{T-1})\). At rollout step \(m\), with verified prefix index \(a_m\) (starting from \(a_0 = -1\)), the drafter proposes a block of \(K\) tokens: $\(d_m = (d_{m,1}, \dots, d_{m,K}) \sim q_\phi(\cdot \mid x, y_{\le a_m})\)$ The target model verifies the block in parallel, accepting \(r_m\) tokens (\(0 \le r_m \le K\)), advancing the next anchor to \(a_{m+1} = a_m + r_m\). This interaction accomplishes two critical functions: first, the final rollout trajectory follows the verified target distribution, ensuring syntactic and semantic integrity across long contexts; second, the anchors \(\{a_m\}\) mark the precise decision boundaries where the draft policy was invoked. Furthermore, because target hidden states for \(y_{\le a_m}\) are already computed during verification, they are cached to eliminate redundant feature extraction during replay.

2. Replay for Log-Probability Computation: Reconstructing Draft-Conditioned Teacher-Student States

Standard target-assisted rollouts evaluate loss only along the verified trajectory, which completely discards rejected draft proposals and degrades into standard offline SFT. Draft-OPD introduces a replay mechanism that revisits each anchor \(a_m\) with prefix context \(c_m = (x, y_{\le a_m})\). The system rolls out the original proposed candidate block \(d_m\) without early stopping, evaluating both student and teacher log-probabilities conditioned on identical draft-generated prefixes: $\(\log q_{m,k}(d_{m,k}) = \log q_\phi(d_{m,k} \mid c_m, d_{m,<k})\)$ $\(\log p_{m,k}(d_{m,k}) = \log p_\theta(d_{m,k} \mid c_m, d_{m,<k})\)$ Verification outcomes divide the replayed tokens into accepted positions \(I_{\text{acc}} = \{(m,k) : 1 \le k \le r_m\}\) and rejected positions \(I_{\text{rej}} = \{(m,k) : r_m < k \le K\}\). By forcing the teacher to score the rejected prefix states, Draft-OPD directly exposes where the drafter assigned excessive probability to trajectories divergent from the target model.

3. Acceptance-Aware Distillation Objective: Asymmetric Bidirectional Divergence with Positional Decay

Accepted and rejected tokens carry fundamentally opposing optimization objectives. Applying forward KL globally forces the drafter to place probability mass across teacher states even when conditioned on invalid draft prefixes, causing mode diffusion. Conversely, applying reverse KL globally impairs mode-covering on valid prefixes where the student should mirror the teacher. Draft-OPD deploys an asymmetric objective: - For accepted tokens (\(I_{\text{acc}}\)), where the drafter already aligns with the target, Forward KL (equivalent to cross-entropy under the target distribution) reinforces mode-covering and stability: $\(L_{\text{acc}} = \frac{1}{|I_{\text{acc}}|} \sum_{(m,k) \in I_{\text{acc}}} D_{\text{KL}}(p_{m,k} \parallel q_{m,k})\)$ - For rejected tokens (\(I_{\text{rej}}\)), where the drafter made mistakes, Reverse KL penalizes the student's high-probability modes that disagree with the target distribution: $\(L_{\text{rej}} = \frac{1}{Z} \sum_{(m,k) \in I_{\text{rej}}} w_k D_{\text{KL}}(q_{m,k} \parallel p_{m,k})\)$ where \(Z = \sum_{(m,k) \in I_{\text{rej}}} w_k\). Within a rejected block, the first rejected token (\(k = r_m + 1\)) marks the primary point of failure, whereas subsequent tokens are conditioned on an already invalidated prefix, introducing compounded noise. Draft-OPD applies an exponential positional decay weight \(w_k = \gamma^{k-1}\) (\(\gamma = 0.8\) by default) to prioritize the earliest error states during gradient backpropagation.

Loss & Training

The combined Draft-OPD distillation loss averages the accepted and rejected objectives: $\(\mathcal{L}_{\text{Draft-OPD}} = \frac{\lambda_{\text{acc}} L_{\text{acc}} + \lambda_{\text{rej}} L_{\text{rej}}}{\lambda_{\text{acc}} + \lambda_{\text{rej}}}\)$ All primary experiments set \(\lambda_{\text{acc}} = \lambda_{\text{rej}} = 1\). The draft model is initialized from a checkpoint trained for 6 SFT epochs using SpecForge on the standard DFlash mixture. The subsequent OPD phase samples 16K prompts from GSM8K (2K), MATH (5K), AoPS (4K), and CodeAlpaca (5K), discarding static answers to generate on-policy target completions online. Training is executed across 8 epochs (~5.3K steps) using verl on 8× NVIDIA H200 GPUs with global batch size 24, AdamW optimizer (learning rate \(3 \times 10^{-4}\), cosine decay, warmup ratio 0.05). Maximum rollout length is set to 4096 for thinking models and 2048 for non-thinking models, with draft block size \(K=16\). For Qwen3-4B, the OPD phase takes approximately 15 hours (~120 H200 GPU-hours).

Key Experimental Results

Main Results

Evaluation spans the Qwen3 model family (Qwen3-4B, Qwen3-8B, and Qwen3-30B-A3B-Thinking) across mathematical reasoning, code generation, and multi-turn chat benchmarks. Baselines are matched on total training FLOPs budget.

Model Decoding Setting Benchmark DFlash \(\tau\) Draft-OPD \(\tau\) DFlash Speedup Draft-OPD Speedup Gain
Qwen3-4B (Thinking) \(T=0\) (Greedy) MATH-500 5.51 6.68 4.88× 5.55× +13.7%
Qwen3-4B (Thinking) \(T=0\) (Greedy) AIME25 5.51 6.59 4.69× 5.28× +12.6%
Qwen3-4B (Thinking) \(T=0\) (Greedy) SWE-Lite 5.51 6.12 4.12× 4.66× +13.1%
Qwen3-4B (Thinking) \(T=0\) (Greedy) 7-Task Mean 5.51 5.96 4.33× 4.86× +12.2%
Qwen3-8B (Thinking) \(T=0\) (Greedy) MATH-500 5.19 6.64 5.01× 5.80× +15.8%
Qwen3-8B (Thinking) \(T=0\) (Greedy) 7-Task Mean 5.19 5.73 4.34× 4.89× +12.7%
Qwen3-4B (Thinking) \(T=0.6\) (Sampling) 7-Task Mean 4.77 5.13 3.84× 4.15× +8.1%
Qwen3-4B (Non-Thinking) \(T=0\) (Greedy) MATH-500 6.04 6.60 6.35× 7.22× +13.7%
Qwen3-4B (Non-Thinking) \(T=0\) (Greedy) 7-Task Mean 6.04 6.60 4.84× 5.31× +9.7%
Qwen3-8B (Non-Thinking) \(T=0\) (Greedy) 7-Task Mean 6.04 6.57 5.11× 5.60× +9.6%

In production serving tests with SGLang (FlashAttention-3 backend), Draft-OPD boosts throughput on Qwen3-30B-A3B-Thinking from 4014 tok/s to 4718 tok/s at concurrency 32 on AIME25 (a 17% gain), demonstrating that higher acceptance lengths translate directly into real-world serving improvements.

Ablation Study

Ablations on Qwen3-4B (Thinking, \(T=0\)) dissect the impact of each architectural component.

Configuration MATH-500 Speedup MATH-500 \(\tau\) HumanEval Speedup HumanEval \(\tau\) MT-Bench Speedup MT-Bench \(\tau\) Note
Draft-OPD (Full) 5.55× 6.57 5.17× 6.18 3.18× 4.44 Full asymmetric KL + position decay
w/o Weight Decay 5.13× 6.18 4.96× 6.01 3.07× 4.29 \(w_k=1\); late-stage noise degrades learning
All-Forward KL 5.34× 6.35 5.01× 6.09 3.09× 4.33 Student error modes improperly fitted
All-Reverse KL 5.11× 6.14 4.94× 5.98 3.08× 4.33 Mode collapse on verified tokens
Random Anchors 5.04× 6.08 4.99× 6.07 2.96× 4.24 Replaying random states dilutes error signals
Naive Rollout 5.06× 6.07 4.80× 5.93 3.02× 4.25 Discarding rejected tokens reverts to SFT

Key Findings

  • Data Scaling vs. On-Policy Exposure: Continuing offline SFT with target responses on the 16K OPD prompt pool yields only 4.28× (EAGLE-3) and 5.14× (DFlash) speedup on MATH-500, substantially lower than Draft-OPD's 5.55×. This confirms that gains stem from exposing the draft model to its own inference-induced errors rather than dataset expansion.
  • Dominance of the First Rejected Token: Restricting the rejected loss strictly to the first failed token (First-rejected-only) achieves an average \(\tau\) of 5.94 (vs. 5.95 for full Draft-OPD, and an identical 6.57 on MATH-500). This indicates that the initial divergence point carries the decisive negative gradient signal, enabling efficient truncated replay implementations.
  • Asymmetry of Positional Decay: Applying decay (\(\gamma = 0.8\)) to accepted tokens (Accept-decay) harms performance, decreasing average \(\tau\) from 5.95 to 5.86. Because accepted tokens follow the true target trajectory, later accepted positions remain fully reliable and should not be downweighted.
  • Rightward Shift of Failure Positions: Profiling the first-rejection distribution reveals that rejections at early positions (1–3) decrease from 39.61% to 38.23% between the first and second halves of training, while rejections at late positions (7–15) increase from 28.48% to 29.50%, and full-block acceptances rise from 5.68% to 6.02%.

Highlights & Insights

  • Decoupling Rollout Stability from Error Mining: By exploiting speculative decoding as an interactive trajectory generator, Draft-OPD bypasses the classical student collapse problem while preserving exact failure states via lightweight replay anchors.
  • Principled Asymmetric Distillation: Formulating accepted tokens under forward KL and rejected tokens under reverse KL accurately mirrors the information-theoretic role of positive coverage versus negative mode suppression.
  • Replay Feature Reuse: Directly reusing the target model's forward hidden states from the rollout verification phase avoids redundant computations, keeping the OPD training cost to an accessible 120 GPU-hours.

Limitations & Future Work

  • Training Rollout Length Bottlenecks: Due to memory constraints during distributed training, rollout trajectories are capped at 4096 tokens, whereas reasoning models generate up to 8192 tokens during inference, leaving late-sequence error states underexplored.
  • Requirement of White-Box Model Access: The method requires full access to token probability distributions and hidden states from the target model, limiting its direct applicability to black-box proprietary APIs.
  • Extension to Lossy Speculative Regimes: The framework is tailored for exact lossless decoding; investigating on-policy error replay under approximate or lossy speculative verification represents a promising research avenue.
  • vs EAGLE-3 / DFlash: Existing draft architectures rely on static SFT on teacher-generated data. Draft-OPD serves as an architecture-agnostic post-training framework that sits on top of DFlash to break through the offline plateau without altering inference-time runtime code.
  • vs DistillSpec / Online Speculative Decoding: Prior distillation methods either adjust divergence metrics on static datasets or fine-tune smaller autoregressive models (e.g., LLaMA-68M) during serving. Draft-OPD is the first to solve on-policy distillation for non-autoregressive block draft modules that cannot independently generate coherent sequences.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Elegant reconciliation of on-policy distillation with non-autonomous speculative draft heads.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive benchmarks spanning 4B/8B/30B models, thinking and non-thinking regimes, greedy and sampling configurations, and SGLang serving.
  • Writing Quality: ⭐⭐⭐⭐⭐ Lucid problem formulation, clear mathematical justifications, and insightful error distribution analyses.
  • Value: ⭐⭐⭐⭐⭐ Establishes a foundational on-policy post-training paradigm for speculative draft models.