Domino: Decoupling Causal Modeling from Autoregressive Drafting in Speculative Decoding¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM (Other)
Keywords: Speculative Decoding, LLM Inference Acceleration, Non-autoregressive Drafting, Causal Modeling Decoupling, Low-rank Residual Correction
TL;DR¶
Addressing the quality-cost trade-off between sequential autoregressive drafting and parallel block drafting in speculative decoding, Domino decouples intra-block causal modeling from draft execution by combining a single-pass parallel backbone with a lightweight low-rank logit correction head, yielding superior acceptance lengths with minimal drafting overhead.
Background & Motivation¶
Standard autoregressive decoding in large language models is fundamentally memory-bound, as sequentially generating one token at a time leaves the massive compute parallelism of modern GPUs vastly underutilized. Speculative decoding has emerged as a prevailing acceleration paradigm, proposing multiple candidate tokens via a lightweight draft model and verifying them concurrently within a single forward pass of the target model. While mathematically preserving the exact output distribution of the target model, the practical end-to-end speedup is jointly governed by two competing factors: the acceptance length per cycle and the wall-clock latency incurred during drafting.
This trade-off exposes a critical dilemma across existing drafting paradigms. Autoregressive drafters, prominently represented by the EAGLE family, construct candidate sequences token by token, thereby explicitly capturing intra-block causal dependencies and closely mimicking the target model's causal distribution. However, drafting \(B\) tokens mandates \(B\) sequential forward passes and repeated full-vocabulary language model head (LM Head) projections, accumulating sequential latency that scales linearly with the speculation budget and often offsets the benefits of longer accepted prefixes. Conversely, fully parallel drafters like DFlash generate an entire candidate block in a single forward execution, drastically trimming draft-time latency but abandoning intra-block causal dependencies, which severely compromises candidate quality and truncates acceptance length.
This raises an essential question: can we retain the draft-quality advantages of causal modeling while preserving the minimal latency profile of block-parallel drafting? The authors observe that causal dependency does not require executing deep autoregressive neural network layers at every draft step; rather, prefix-dependent causal cues can be decoupled and delivered as a residual refinement in logit space. Core idea: Domino decouples causal dependency modeling from autoregressive draft execution by using a parallel draft backbone to generate preliminary block distributions in one pass, followed by a lightweight Domino head that injects prefix causal context via a GRU state and low-rank residual projection in logit space, stabilized by a base-anchored training curriculum.
Method¶
Overall Architecture¶
Domino consists of two primary modules: a Parallel Draft Backbone and a Domino Head. At each speculation cycle, the target model's verified prefix context features are passed into the parallel draft backbone alongside a masked draft block, outputting block-level hidden states in a single non-autoregressive forward pass. A frozen target LM Head projects these states into parallel base logits across all block positions. Subsequently, the Domino head sequentially steps through the candidate block: a lightweight causal encoder tracks the prefix history from sampled token embeddings, while a low-rank correction head computes residual logit adjustments. The final draft tokens are sampled from the sum of the base logits and correction logits.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Target verified prefix and masked block input"] --> B["Parallel Draft Backbone<br/>Single forward pass computes block hidden states and base logits via frozen LM Head"]
B --> C["Causal Encoder<br/>Lightweight GRU sequentially encodes prefix token embeddings into causal states"]
C --> D["Low-Rank Correction Head<br/>Bottleneck projection produces residual logit corrections"]
D --> E["Base-Anchored Curriculum<br/>Time-varying loss annealing prevents shortcut-induced backbone collapse"]
E --> F["Target model parallel verification and longest prefix acceptance"]
Key Designs¶
1. Parallel Draft Backbone: Eliminating Sequential Drafting Bottlenecks via Single-Pass Block Prediction To remove the linear runtime growth \(\mathcal{O}(B \cdot (t_{\text{net}} + t_{\text{head}}))\) inherent to sequential autoregressive drafters, Domino utilizes a block-parallel drafting architecture. Conditioned on the verified context \(x_{\le t}\), the last verified token \(x_t\) serves as an anchor to formulate a masked candidate block \(\tilde{x}_{t:t+B-1} = [x_t, \text{[MASK]}, \dots, \text{[MASK]}]\). The backbone digests the context representations \(C_t\) and masked block embeddings in a single non-autoregressive pass, producing block-level hidden representations \(H_{t:t+B-1}\). These hidden states are then projected into initial candidate distributions across all future positions simultaneously using the target model's frozen LM Head: \(L^{\text{base}}_i = \text{LMHead}(H_i)\) for \(i = t+1, \dots, t+B-1\). By executing representation extraction and full-vocabulary mapping only once per block, GPU utilization is maximized and drafting latency is reduced to a constant-level overhead.
2. Causal Encoder: Tracking Inter-Token History via Lightweight Recurrent States While the parallel backbone generates base logits efficiently, its non-autoregressive nature treats future positions independently, lacking the conditional dependency on earlier sampled tokens. Domino integrates a lightweight causal encoder to bridge this gap during rollout. Specifically, a gated recurrent unit (GRU) with a compact hidden dimension of 1024 sequentially summarizes the embeddings of previously sampled draft tokens within the block: \(S_{i-1} = \text{GRU}(E_{\le i-1}) \in \mathbb{R}^{d_s}\). This causal state vector \(S_{i-1}\) encapsulates the prefix trajectory without requiring any expensive multi-layer Transformer forward computation, delivering causal conditioning with orders of magnitude lower latency than full-model autoregression.
3. Low-Rank Correction Head: Residual Refinement in Logit Space to Avoid Full-Vocabulary Projections Performing causal updates in hidden space would necessitate reapplying the full LM Head after every step to obtain probability distributions, reintroducing the costly projection bottleneck. Instead, Domino performs residual correction directly in logit space. Combining the backbone hidden feature \(H_i\) and the causal state \(S_{i-1}\), the low-rank correction head generates an additive residual: $$ \Delta L_i = W_2 \, \sigma(W_1 [H_i; S_{i-1}]) $$ Here, \(W_1\) projects the concatenated representation down to a low-rank dimension (\(r = 256\)), \(\sigma\) denotes SiLU non-linearity, and \(W_2\) expands the low-rank representation back to vocabulary size. The final sampling distribution is defined as \(L_i = L^{\text{base}}_i + \Delta L_i\). Because \(r \ll d_{\text{model}}\), computation remains minimal. Implemented with fused Triton kernels and CUDA Graphs, the single-step Domino head latency drops from 2.64ms to 1.20ms.
4. Base-Anchored Curriculum: Annealed Multi-Task Supervision to Prevent Backbone Collapse During training, the causal encoder receives ground-truth token embeddings under a teacher-forcing scheme, which precisely aligns with speculative decoding because a candidate token at step \(i\) contributes to acceptance length if and only if all preceding tokens match the target sequence. However, directly optimizing solely the final corrected distribution encourages the low-rank branch to shortcut the parallel backbone, leaving base representations unoptimized. To prevent this collapse, Domino employs a joint base-anchored loss with time-varying annealing: $$ \mathcal{L} = (1 - \lambda_t) \mathcal{L}{\text{final}} + \lambda_t \mathcal{L} $$ where }\(\mathcal{L}_{\text{base}}\) and \(\mathcal{L}_{\text{final}}\) are position-weighted cross-entropy losses evaluated on \(L^{\text{base}}_i\) and \(L_i\) using exponential decay factors \(w_k = \exp(-k/\gamma)\). Annealing \(\lambda_t\) linearly from 1 to 0 over training forces the parallel backbone to establish high-fidelity base predictions initially before transferring optimization to the residual causal refinement.
Loss & Training¶
The target LLM remains completely frozen throughout training. Draft components are trained on mlabonne/open-perfectblend (1.42M samples) with all responses regenerated by the corresponding target model. Sequences are truncated to 3072 tokens with a fixed draft block size of 16. Training runs for 3 epochs using 8 NVIDIA A100 GPUs with AdamW, a peak learning rate of \(6 \times 10^{-4}\), zero weight decay, gradient clipping norm of 1.0, and a cosine learning rate schedule with 4% warmup. Position-dependent weighting \(w_k = \exp(-k/\gamma)\) prioritizes earlier tokens that gate subsequent block verification.
Key Experimental Results¶
Main Results¶
On Qwen3-4B and Qwen3-8B using the Transformers backend under greedy decoding (\(T=0\)), Domino achieves consistent gains across reasoning, code, and dialogue tasks over prior autoregressive and parallel baselines (data excerpted from Table 1 of the paper):
| Model | Method (Budget) | GSM8K (Speed/\(\tau\)) | HumanEval (Speed/\(\tau\)) | MBPP (Speed/\(\tau\)) | MT-Bench (Speed/\(\tau\)) | Overall Avg. (Speed/\(\tau\)) |
|---|---|---|---|---|---|---|
| Qwen3-4B | EAGLE-3 (16) | 2.24ร / 3.32 | 2.09ร / 3.09 | 2.02ร / 2.99 | 1.95ร / 2.94 | 2.04ร / 3.04 |
| EAGLE-3 (60) | 2.57ร / 3.83 | 2.36ร / 3.53 | 2.30ร / 3.44 | 2.25ร / 3.41 | 2.32ร / 3.49 | |
| DART (60) | 2.16ร / 2.65 | 2.48ร / 2.99 | 2.47ร / 3.01 | 2.19ร / 2.69 | 2.25ร / 2.76 | |
| DFlash (16) | 5.18ร / 6.51 | 5.23ร / 6.63 | 4.77ร / 6.12 | 2.72ร / 4.12 | 4.70ร / 6.11 | |
| DOMINO (16) | 8.02ร / 10.12 | 5.66ร / 7.02 | 5.59ร / 7.11 | 3.28ร / 5.12 | 5.47ร / 7.08 | |
| Qwen3-8B | EAGLE-3 (16) | 2.21ร / 3.27 | 2.17ร / 3.21 | 1.93ร / 2.86 | 1.82ร / 2.75 | 1.97ร / 2.97 |
| EAGLE-3 (60) | 2.56ร / 3.80 | 2.50ร / 3.74 | 2.22ร / 3.31 | 2.07ร / 3.17 | 2.26ร / 3.41 | |
| DART (60) | 2.28ร / 2.71 | 2.52ร / 2.95 | 2.39ร / 2.98 | 2.27ร / 3.03 | 2.29ร / 2.82 | |
| DFlash (16) | 5.21ร / 6.59 | 5.21ร / 6.52 | 4.71ร / 5.98 | 2.73ร / 4.18 | 4.66ร / 6.06 | |
| DOMINO (16) | 7.92ร / 10.03 | 5.89ร / 7.39 | 5.53ร / 7.04 | 3.29ร / 5.18 | 5.49ร / 7.17 |
Ablation Study¶
The impact of the Domino head and the proposed training scheme is ablated on Qwen3-8B using controlled ShareGPT training (data from Tables 4 and 6 of the paper):
| Configuration | GSM8K (\(\tau\) / Speed) | HumanEval (\(\tau\) / Speed) | MBPP (\(\tau\) / Speed) | Avg. Acceptance \(\tau\) | Avg. Speedup | Description |
|---|---|---|---|---|---|---|
| w/o Domino Head | 3.82 / 3.17ร | 3.69 / 3.04ร | 3.30 / 2.70ร | 3.49 | 2.84ร | Parallel backbone only without causal head |
| w/ Domino Head | 4.80 / 3.84ร | 4.35 / 3.51ร | 3.94 / 3.16ร | 4.19 | 3.31ร | Full architecture with low-rank correction |
| Strategy: TTT | - | - | - | 3.80 | - | Multi-step rollouts simulated during training |
| Strategy: TF | - | - | - | 3.96 | - | Teacher forcing without curriculum annealing |
| Strategy: TF + Curr | - | - | - | 4.19 | - | Teacher forcing with base-anchored curriculum |
Key Findings¶
- High Gain at Negligible Overhead: Relative to DFlash, Domino introduces only 56M parameters (+5.3% draft capacity) and a 2.8% latency increase during draft-then-verify execution on an A100 GPU, while boosting average acceptance length by 16.6% (reaching \(\tau > 10\) on GSM8K).
- Serving Concurrency Resilience: Under SGLang serving across concurrency levels 2 to 32, autoregressive drafters degrade rapidly as sequential draft latency chokes execution (dropping below 1.0ร baseline throughput at batch 32). Domino scales efficiently, delivering 5509 TPS (1.9ร speedup) on Qwen3-4B at concurrency 32.
- Backbone Collapse Avoided: Tracking backbone loss across training reveals that without curriculum annealing, the parallel backbone loss stalls above 7.0 due to shortcutting. Anchoring the objective initially on base logits ensures steady loss reduction to ~2.0, preserving a reliable parallel baseline.
Highlights & Insights¶
- Decoupling Causal Modeling from Execution: Conventional autoregressive drafters conflate causal sequence modeling with multi-layer neural network execution and full-vocabulary mapping. Domino establishes that causal dependencies can be distilled via a compact recurrent state and injected as an additive low-rank logit residual.
- Harmonizing Teacher Forcing with Verification: While teacher forcing typically suffers from exposure bias in standard autoregressive tasks, speculative decoding invalidates candidate trajectories as soon as an erroneous token occurs. Hence, learning causal updates strictly over ground-truth prefixes concentrates optimization entirely on the accepted-prefix regime.
- Triton Operator Fusion and Graph Capture: Fusing the GRU update, low-rank matrix multiplications, and logit addition into dedicated Triton kernels captured within CUDA Graphs halves the sequential overhead from 2.64ms to 1.20ms, ensuring that the sequential correction pass never becomes a runtime bottleneck.
Limitations & Future Work¶
- Serving Engine and Hardware Diversity: Evaluations are predominantly conducted on SGLang and NVIDIA Ampere architectures; integrating and validating custom fused kernels within vLLM, TensorRT-LLM, and alternative accelerators (AMD ROCm, Huawei Ascend) remains to be demonstrated.
- Fixed Block Speculation Budget: The current block length is fixed at 16 tokens. Implementing dynamic budget allocation based on runtime entropy or sequence difficulty could further optimize drafting resources.
- Extension to Multimodal and Long-Context Drafting: Extending decoupled logit correction to diffusion-based vision-language generation or ultra-long context reasoning where causal history tracking is particularly challenging.
Related Work & Insights¶
- vs EAGLE-3 / EAGLE-2: EAGLE variants rely on autoregressive feature progression and tree-based expansion, which scale steeply in sequential compute cost; Domino achieves comparable or higher acceptance lengths with single-pass parallel drafting and residual correction, yielding more than 2x higher serving throughput.
- vs DFlash / DART: Parallel diffusion and multi-token drafters eliminate sequential execution but drop intra-block causal structure; Domino recovers intra-block causal alignment with only a 2.8% latency increase, boosting acceptance length by 16.6%.
- vs FR-Spec / SpecVocab: These methods prune vocabulary projections to accelerate autoregressive drafters; Domino circumvents repeated full projections entirely by computing low-rank corrections directly in logit space.
Rating¶
- Novelty: โญโญโญโญโญ Elegant decoupling of causal intra-block dependency modeling from heavy autoregressive execution.
- Experimental Thoroughness: โญโญโญโญโญ Comprehensive evaluations spanning Qwen3 models, 8 major benchmarks, high-concurrency throughput, and exhaustive ablations.
- Writing Quality: โญโญโญโญโญ Clear exposition, insightful latency-breakdown analysis, and well-motivated engineering choices.
- Value: โญโญโญโญโญ Closes the quality gap between parallel and autoregressive drafting, offering immediate production utility for LLM inference serving.