Skip to content

DFlare: Scaling Up Draft Capacity for Block Diffusion Speculative Decoding

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Tencent/AngelSlim
Area: LLM Reasoning
Keywords: Speculative Decoding, Block Diffusion Drafting, Adaptive Layer Fusion, Heterogeneous KV Projection, Inference Acceleration

TL;DR

Addressing the capacity bottlenecks and rigid target feature conditioning in block diffusion speculative decoding, DFlare introduces lightweight layer-wise adaptive fusion, heterogeneous KV projections, and a progressive position-weighted loss, achieving up to 5.52× wall-clock acceleration across math, code, and chat benchmarks.

Background & Motivation

Large language models have demonstrated extraordinary capabilities across mathematical reasoning, code generation, and multi-turn conversation. However, standard autoregressive generation suffers from memory bandwidth constraints due to sequential token-by-token decoding, resulting in severe inference latency. Speculative decoding has emerged as a dominant, mathematically lossless acceleration paradigm that employs a lightweight draft model to predict a sequence of candidate tokens, followed by a single parallel verification pass by the primary target model. Conventional draft models rely on autoregressive architectures (e.g., the EAGLE series), where draft tokens are produced sequentially; consequently, adding more draft layers directly multiplies drafting latency, strictly bounding the draft model's depth and parameter capacity. Recent breakthroughs in block diffusion drafting (such as DFlash) redefine this trade-off by predicting all tokens within a block simultaneously via discrete diffusion in a single forward pass, decoupling drafting latency from block length and opening the door to deeper, more expressive draft models.

Despite this architectural promise, existing state-of-the-art block diffusion methods like DFlash fail to reap the full benefits of scaling draft capacity. A rigorous anatomical inspection reveals two tightly coupled bottlenecks: first, in terms of target knowledge injection, DFlash uniformly samples hidden states from only a few target layers (e.g., 5 layers), concatenates them, and compresses them through a single fully-connected (FC) layer into a solitary fused representation that is identically broadcast across all draft layers. Feeding the exact same fused vector to every layer deprives individual draft layers of the ability to specialize across different semantic granularities. Second, in terms of depth scaling, this homogenized input causes deeper draft networks to hit a sharp saturation ceiling, where marginal acceptance length gains from extra layers are easily erased by the added computation.

To dismantle this vicious cycle of conditioning bottlenecks and depth saturation, the information highway connecting the target model to the draft network must be fundamentally redesigned. This work sets out to flare out the narrow conditioning bottleneck by granting each draft layer its own dedicated view of multi-level target representations, while resolving internal representation interference and optimizing training schedules. Core idea: flaring out the narrow conditioning bottleneck through lightweight layer-wise adaptive fusion and heterogeneous KV projections, coupled with a progressive position-weighted loss and 2.4M-scale data, unlocking depth scalability and delivering massive wall-clock speedups for block diffusion speculative decoding.

Method

Overall Architecture

DFlare operates on the block diffusion speculative decoding paradigm, structured into three primary phases: multi-level target knowledge extraction, single-pass block-parallel candidate drafting, and parallel verification by the target model. During inference, the target model produces the current decode token \(x_t\) along with historical context hidden states across multiple uniformly sampled layers. Rather than collapsing these states via a blunt linear projection, DFlare employs an Adaptive Layer Fusion module that computes a customized, normalized weighted combination of target hidden states for each draft layer. Inside each Transformer layer of the draft network, Heterogeneous KV Projections map the fused target context and the evolving draft block states into independent representational subspaces before computing bidirectional attention. The draft network then produces \(B-1\) candidate tokens in a single forward step, which are verified in parallel by the target model to maximize the accepted sequence length.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Target Forward Pass & Multi-Layer Context Extraction"] --> B["Adaptive Layer-Wise Target Feature Fusion"]
    B --> C["Heterogeneous KV Projection Subspace Decoupling"]
    C --> D["Progressive Weighted Block Diffusion Parallel Denoising"]
    D --> E["Target Single-Pass Parallel Verification & Acceptance"]

Key Designs

1. Adaptive Layer-Wise Target Feature Fusion: Eliminating Draft Layer Homogenization from Shared Compression Prior approaches concatenate hidden states from a small subset of target layers and pass them through a single linear layer, broadcasting identical context to all draft layers. Because shallow and deep target layers encode fundamentally different abstractions—from local syntax to global semantics—forcing all draft layers to consume the same compressed vector inhibits per-layer specialization. DFlare introduces an extremely lightweight, layer-specific adaptive fusion mechanism. Given hidden states \(h_t^{(j)} \in \mathbb{R}^d\) from \(T\) selected target layers (\(j = 1, \dots, T\), e.g., \(T=9\)), a learnable parameter matrix \(W_{\text{fuse}} \in \mathbb{R}^{D \times T}\) is defined for a \(D\)-layer draft model. For draft layer \(i\), its normalized attention weights and layer-specific fused target context are computed as: $\(f_t^{(i)} = \text{RMSNorm}\left(\sum_{j=1}^T \alpha_j^{(i)} h_t^{(j)}\right), \quad \alpha^{(i)} = \text{softmax}\left(W_{\text{fuse}[i, :]}\right)\)$ This design adds only \(D \times T\) scalar parameters (e.g., 63 scalars for \(D=7, T=9\)). Crucially, the softmax weights \(\alpha^{(i)}\) can be precomputed and cached offline after training; at runtime, fusion collapses into a static scalar-weighted linear combination, adding negligible inference latency. Each draft layer thus receives a dedicated target perspective tailored to its depth in the hierarchy, breaking the expressiveness ceiling.

2. Heterogeneous KV Projections: Resolving Subspace Interference Between Target Knowledge and Draft Noise Inside the draft model's attention sublayers, queries attend simultaneously to the injected target context and internal block positions. DFlash forces the target context features and draft hidden states to share the same Key and Value projection matrices. However, these two information streams originate from vastly different domains: target hidden states carry clean, high-fidelity semantic representations from the verified prefix, whereas draft hidden states represent noisy, intermediate predictions undergoing iterative discrete diffusion. Forcing them into an identical projection space causes mutual representational interference. DFlare resolves this by decoupling the Key-Value projection pathways: $\(K = \left[ F^{(i)} W_K^t \,;\, x_t W_K^d \,;\, \hat{X} W_K^d \right], \quad V = \left[ F^{(i)} W_V^t \,;\, x_t W_V^d \,;\, \hat{X} W_V^d \right]\)$ where \(W_K^t, W_V^t \in \mathbb{R}^{d \times d_{kv}}\) are dedicated projections reserved exclusively for the fused target context \(F^{(i)}\), while \(W_K^d, W_V^d \in \mathbb{R}^{d \times d_{kv}}\) independently map the target decode token \(x_t\) and masked draft block states \(\hat{X}\). This separation grants the attention mechanism independent representational degrees of freedom, significantly boosting the draft model's ability to extract and attend to target conditioning signals.

3. Progressive Position-Weighted Loss: Balancing Early-Prefix Convergence and Late-Token Optimization Speculative decoding enforces strict longest-common-prefix verification: an error at an early position in a draft block immediately invalidates all subsequent tokens. While early positions require high precision, employing a fixed, aggressively decaying exponential loss weight throughout training leaves deeper positions severely under-optimized, capping the maximum attainable acceptance length. DFlare introduces a dynamic curriculum schedule where the decay parameter \(\gamma\) warms up linearly with training steps: $\(w_k(s) = \exp\left(-\frac{k - 1}{\gamma(s)}\right), \quad \gamma(s) = \gamma_0 + \frac{s}{S}(\gamma_{\max} - \gamma_0)\)$ where \(s\) denotes the current training step and \(S\) is the total step budget. In the early training phase (\(\gamma \approx \gamma_0 = 4.5\)), loss is concentrated on the earliest block positions to ensure fast and stable prefix convergence. As training progresses and \(\gamma\) increases towards \(\gamma_{\max}\), the weight distribution flattens, gradually shifting gradient pressure toward harder tail positions. This curriculum ensures thorough optimization across the entire block without degrading early-position accuracy.

Loss & Training

The draft models are trained using masked discrete diffusion aligned with the inference procedure. For each training sequence (up to 3072 tokens), 512 anchor positions are sampled as target decode tokens \(x_t\), each followed by \(B-1\) masked tokens \(\hat{X}\) (with block size \(B=16\)). All sampled blocks within a sequence are batched using a block-diagonal bidirectional attention mask, enabling full-sequence parallel training in a single forward/backward pass. To prevent distribution shift, training targets are regenerated using the frozen base model under sampling temperature 0.6. The draft model shares and freezes the target model's embedding layer and LM head, updating only its 7 draft Transformer layers and fusion parameters. Models are optimized using AdamW (learning rate 6e-4, cosine schedule with 0.04 warmup ratio, gradient clipping 1.0) over 6 epochs. Training data is scaled from the standard 800K mixture to 2.4M high-quality samples drawn from NVIDIA Nemotron Post-Training V2, CodeAlpaca, and Step-3.5-Flash-SFT.

Key Experimental Results

Main Results

Evaluated on Qwen3-4B, Qwen3-8B, and GPT-OSS-20B across six benchmarks spanning mathematical reasoning (GSM8K, MATH500, AIME25), code generation (HumanEval, MBPP), and conversation (MT-Bench) under greedy decoding (\(T=0\)):

Base Model Benchmark DFlash Speedup DFlash Acc. Len \(\tau\) DFlare Speedup DFlare Acc. Len \(\tau\) Relative Gain
Qwen3-4B GSM8K 5.31× 6.51 6.00× 7.93 +13.0%
Qwen3-4B MATH500 6.20× 7.80 6.76× 8.97 +9.0%
Qwen3-4B AIME25 5.78× 7.31 6.31× 8.35 +9.2%
Qwen3-4B HumanEval 4.62× 6.65 5.41× 7.17 +17.1%
Qwen3-4B MBPP 4.89× 6.13 5.30× 7.08 +8.4%
Qwen3-4B MT-Bench 3.13× 4.40 3.33× 5.30 +6.4%
Qwen3-4B Average 4.99× 6.47 5.52× 7.47 +10.6%
Qwen3-8B GSM8K 5.28× 6.53 6.02× 7.88 +14.0%
Qwen3-8B MATH500 6.29× 7.88 6.72× 8.95 +6.8%
Qwen3-8B AIME25 5.77× 7.11 6.21× 8.12 +7.6%
Qwen3-8B HumanEval 5.32× 6.54 5.42× 7.08 +1.9%
Qwen3-8B MBPP 4.81× 5.97 5.16× 6.86 +7.3%
Qwen3-8B MT-Bench 2.85× 4.30 3.23× 5.09 +13.3%
Qwen3-8B Average 5.05× 6.39 5.46× 7.33 +8.1%
GPT-OSS-20B GSM8K 3.75× 4.54 4.06× 4.98 +8.3%
GPT-OSS-20B MATH500 3.96× 4.86 4.20× 5.19 +6.1%
GPT-OSS-20B AIME25 3.85× 4.93 3.88× 4.98 +0.8%
GPT-OSS-20B HumanEval 3.49× 4.25 3.71× 4.55 +6.3%
GPT-OSS-20B MBPP 3.44× 4.20 3.72× 4.58 +8.1%
GPT-OSS-20B MT-Bench 3.74× 5.20 3.87× 5.32 +3.5%
GPT-OSS-20B Average 3.71× 4.66 3.91× 4.93 +5.4%

Ablation Study

Component ablations on Qwen3-4B under a controlled 270K training data setting:

Architecture Variant GSM8K Speedup / \(\tau\) HumanEval Speedup / \(\tau\) MT-Bench Speedup / \(\tau\) Overall Avg Speedup Overall Avg Acc. Len \(\tau\) Key Analysis & Impact
DFlare (Full Model) 3.80× / 4.94 3.55× / 4.62 2.76× / 4.00 3.37× 4.52 Optimal synergy across all components
- Softmax (Unconstrained Weights) 3.70× / 4.85 3.45× / 4.53 2.77× / 4.02 3.31× 4.47 Unbounded layer weights destabilize multi-layer fusion
- KVProj (Shared KV Projections) 3.77× / 4.90 3.50× / 4.54 2.78× / 3.99 3.35× 4.48 Subspace collision between clean target and noisy draft
- Loss (Static Fixed Loss) 3.72× / 4.84 3.56× / 4.63 2.73× / 3.93 3.34× 4.47 Tail positions under-optimized without curriculum
+ FC (Extra Linear Layer after Fusion) 3.74× / 4.87 3.53× / 4.59 2.78× / 4.06 3.35× 4.51 Additional parameter overhead without accuracy gain

Key Findings

  • Unlocking Positive Depth Scaling: Expanding draft depth from 5 to 7 layers causes DFlash to saturate quickly (speedup plateaus from 4.98× to 5.07× and regresses to 5.01×). In contrast, DFlare shows steady, monotonic scaling from 5.12× to 5.16× speedup, and acceptance length expands consistently from 6.55 to 6.90.
  • Pronounced Layer-Wise Specialization: Empirical entropy and weight analysis on Qwen3-4B confirm that draft layers develop sharp, distinct target layer preferences. Draft layers 2 and 3 assign over 60% of their attention to deep semantic layer 21, draft layer 6 concentrates 82.24% on lexical layer 1, and draft layer 5 focuses heavily on top layers 29 and 33. This heterogeneity validates the necessity of abandoning uniform FC compression.
  • Capacity-Data Symbiosis: Scaling training data from 270K to 800K and 2.4M drives DFlare on Qwen3-8B from 3.74× (acc. len 4.90) to 4.97× (6.59) and finally 5.46× (7.33), demonstrating that deeper draft models possess high capacity that demands larger corpora to fully exploit.
  • High Concurrency Serving Robustness: Benchmarked on SGLang using an H20 GPU, DFlare scales exceptionally well under heavy server loads: at concurrency 32 on GSM8K, DFlare delivers 2603.7 tokens/s compared to DFlash's 2264.3 tokens/s and baseline's 2570.5 tokens/s.

Highlights & Insights

  • Zero-Inference-Overhead Adaptive Routing: DFlare injects multi-granularity target knowledge across all draft layers using only \(D \times T\) learnable scalar weights that are pre-softmaxed and cached at inference, yielding rich per-layer conditioning at virtually zero FLOP overhead.
  • Rethinking the "Acceptance Length vs. Latency" Trade-off: Under identical 270K training conditions, autoregressive EAGLE-3 achieves higher acceptance length than DFlare (4.78 vs 4.48), yet its wall-clock speedup is strictly lower (3.06× vs 3.18×) due to sequential token generation latency. Block diffusion fundamentally alters the latency profile in favor of parallel drafting.
  • Modular Integration: The proposed enhancements cleanly integrate into open-source inference frameworks (packaged in AngelSlim), preserving standard speculative decoding verification without altering target model distributions.

Limitations & Future Work

  • Significant Upfront Training Cost: Scaling draft depth and dataset volume to 2.4M examples incurs substantial pre-computation costs (e.g., ~160 GPU hours on 32 H20 GPUs for Qwen3-8B). However, this one-time cost is rapidly amortized during high-throughput deployment.
  • Sub-linear Scaling on Ultra-Large Base Models: On GPT-OSS-20B, the speedup gain is 5.4% (vs. 10.6% on 4B), constrained by an 8-token block size and wider representational gaps between draft and target models.
  • Future Directions: Exploring heterogeneous draft architectures (e.g., Mamba or linear attention draft backbones paired with adaptive layer fusion) and scaling up to 10M+ training samples with reinforcement learning alignment.
  • vs DFlash: DFlash compresses a small set of target layers through a single shared FC layer and forces shared KV projections across draft states, causing premature depth saturation. DFlare establishes layer-wise adaptive conditioning, heterogeneous KV projections, and curriculum loss, unlocking deep draft scaling.
  • vs EAGLE-3 / Medusa: Autoregressive tree drafting (EAGLE-3) suffers from sequential multi-step draft latency, while multi-head models (Medusa) struggle with token dependency modeling across long blocks. DFlare leverages bidirectional diffusion to model intra-block dependencies in a single forward pass, providing superior wall-clock acceleration under high concurrency.

Rating

  • Novelty: ⭐⭐⭐⭐☆ Pinpoints the core bottleneck of feature homogenization in block diffusion drafting and introduces an elegant, lightweight adaptive solution.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive validation across three base models, six diverse benchmarks, rigorous ablations, data scaling laws, and real-world SGLang serving benchmarks.
  • Writing Quality: ⭐⭐⭐⭐⭐ Highly coherent technical narrative, precise mathematical formulations, and thorough qualitative and quantitative analyses.
  • Value: ⭐⭐⭐⭐⭐ Establishes a new state of the art in speculative decoding acceleration and provides immediately deployable tools via AngelSlim.