Spexis: Speculative Lookahead Scheduling for LLM Inference¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/mlsys-seo/spexis
Area: LLM Efficiency
Keywords: Speculative Parallelism, Lookahead Scheduling, Pipeline Parallelism, KV Cache Memory, Remaining-Length Prediction
TL;DR¶
Spexis introduces speculative parallelism (SP) as an orthogonal scheduling dimension for multi-GPU LLM inference that overlaps self-speculative drafting with downstream pipeline verification while sharing the same KV cache, augmented with confidence-based selective speculation and lookahead remaining-length prediction to achieve up to 34% serving throughput speedup over optimized TP+PP baselines.
Background & Motivation¶
As frontier large language models grow dramatically in parameter size and long-context KV cache footprints, single-GPU setups fail to sustain high-concurrency serving workloads, making multi-GPU and distributed cluster inference standard across modern production infrastructure. Within distributed serving systems, tensor parallelism (TP) and pipeline parallelism (PP) serve as the primary foundational execution strategies. However, both encounter severe scaling bottlenecks as the GPU count expands: TP mandates frequent cross-device synchronization collectives (All-Reduce) after both attention and feed-forward projections within each transformer layer, leaving compute cores idling even across high-bandwidth intra-node NVLink fabrics; conversely, PP partitions consecutive layers into stages and introduces micro-batching to hide inter-stage bubbles, requiring each micro-batch to maintain its own independent KV cache state and forcing micro-batch sizes to shrink drastically, degrading GPU compute core utilization and serving throughput.
Although production serving engines frequently adopt hybrid TP+PP configurations, this compromise fails to resolve the underlying tension. Raising the pipeline parallelism degree escalates per-stage memory pressure from micro-batches, whereas increasing the tensor parallelism degree compounds interconnect communication latencies. Meanwhile, speculative decoding has proven effective in breaking the sequential token-by-token dependency of autoregressive generation, yet it remains confined to an algorithmic acceleration trick on single-request or single-GPU tiers without being integrated into distributed pipeline and memory scheduling.
To overcome these structural limitations, this paper investigates whether speculative decoding can be elevated into a first-class system scheduling primitive where drafting and verification seamlessly overlap across GPUs without incurring extra micro-batch KV cache allocations. Core idea: introduce speculative parallelism (SP) as a memory-efficient scheduling axis orthogonal to TP and PP that executes self-speculative drafting concurrently with downstream pipeline verification over shared KV cache entries, guided by selective confidence pruning and remaining-length lookahead simulation to maximize serving throughput and eliminate cache evictions.
Method¶
Overall Architecture¶
Spexis is built on top of the vLLM and Ray distributed runtimes. Its architectural foundation replaces the conventional PP execution modelโwherein distinct micro-batches demand isolated KV cache states across pipeline stagesโwith a decoupled self-speculative pipeline. The end-to-end inference execution orchestrates two cooperative layers: in the physical execution plane, the early layers of the base model serve as the drafting stage hosted on the initial pipeline stage, while the remaining deeper layers act as the verification stage on subsequent pipeline stages. Once candidate draft tokens are produced by the initial stage, Spexis immediately launches speculative execution for the next step on the first stage in parallel with the verification of preceding draft tokens on the second stage. Because speculative execution projects future tokens along existing request contexts, it directly shares the underlying KV cache with verification, completely avoiding memory duplication.
In the upper-level scheduling plane, Spexis deploys an analytic cost model alongside a lookahead scheduler. The scheduler continuously monitors running sequences via a lightweight three-layer MLP that predicts remaining decode lengths across ordinal thresholds and projects memory usage over a 64-iteration lookahead horizon. When admitting a new request risks memory exhaustion and subsequent request eviction, the scheduler quantitatively evaluates the tradeoff between delayed admission throughput loss and eviction recomputation penalties. Simultaneously, draft tokens are filtered by confidence scores, pruning low-quality speculation branches to ensure compute cycles and interconnect bandwidth are spent strictly on high-probability tokens.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Request Arrival & Input Batch"] --> B["Lookahead Scheduling Decision<br/>Guided by Remaining-Length & Eviction Cost Model"]
B -->|Scheduled for Admission| C["Early Pipeline Stage: Drafting Adapter<br/>Emit Draft Tokens & Confidence Scores"]
C --> D["Selective Speculation Filtering<br/>Discard Bottom-ฮฑ Low-Confidence Drafts"]
D --> E["Speculative-Parallel Execution Overlap<br/>Stage 1 Speculation โ Stage 2 Verification in Parallel"]
E -->|Draft Tokens Accepted| F["Emit Verified Tokens & Update Shared KV Cache"]
E -->|Draft Tokens Rejected| G["Roll Back & Resume Normal Verification Path"]
Key Designs¶
1. Speculative Parallelism (SP) and Steady-State Cost Modeling: Eliminating Micro-Batch Memory Explosion
In standard pipeline parallelism (PP), mitigating inter-stage pipeline bubbles requires cycling \(N\) micro-batches across \(N\) stages, restricting the maximum concurrent sequence capacity per stage to \(B/N\) (where \(B\) is the maximum non-pipelined batch size). This division rapidly exhausts GPU memory due to multiple independent KV cache allocations. Spexis resolves this bottleneck by adapting a self-speculative model architecture (similar to Kangaroo, attaching a lightweight adapter to early transformer layers) and distributing it across pipeline stages. During iteration \(t\), Stage 1 executes speculative forward passes on drafted tokens while Stage 2 concurrently verifies preceding tokens. Because speculative tokens expand upon existing request sequences, they directly share the identical KV cache space with normal execution, avoiding any additional micro-batch memory overhead.
To characterize system behavior and determine the optimal parallelism partitioning across TP, PP, and SP, Spexis derives a recurrence-based analytical cost model. Let \(B\) be the maximum batch capacity, \(N\) the pipeline stages, \(\Theta\) the speculation acceptance rate, and \(s^a_t, s^r_t, n_t\) the accepted speculative, rejected speculative, and normal executions at iteration \(t\) (satisfying \(s^a_t + s^r_t + n_t = B\)). In the steady-state limit (\(t \to \infty\)), the expected effective batch size \(B^\star_{\text{eff}}\) of accepted tokens follows:
By coupling this effective batch capacity with profiled single-stage decoding latency functions \(T_{\text{sp}}(B)\), the system accurately models throughput and dynamically selects the best execution topology for any given hardware setup.
2. Confidence-Ranked Selective Speculation: Pruning Low-Yield Computation
The throughput gains of speculative parallelism depend directly on the token acceptance rate \(\Theta\). Under saturated serving loads, blindly applying speculation across all running requests risks squandering compute resources: low-confidence draft tokens at the tail of the batch exhibit high rejection rates, triggering pipeline rollbacks and wasteful re-executions. To address this, Spexis incorporates confidence-ranked selective speculation.
Specifically, Spexis ranks all drafted tokens within a batch based on the softmax confidence emitted by the drafting adapter, dropping a fixed fraction \(\alpha \in [0.15, 0.3]\) of the lowest-ranked tokens. These discarded tokens fall back to standard non-speculative execution, allowing the retained subset to achieve a substantially higher acceptance rate \(\Theta_\alpha\). Under selective speculation, the steady-state total batch size evolves as \(B^\star_\alpha = B / [1 + \alpha(N-1)]\), yielding an effective accepted batch capacity of:
For models with lower raw drafting accuracy such as Qwen3-32B, this selective dropping improves effective speculation accuracy by 8% to 14%, significantly amplifying overall decoding throughput under heavy request pressure.
3. Remaining-Length-Aware Lookahead Scheduling: Balancing Admission Gains Against Recomputation Overhead
Because KV cache memory footprint grows monotonically with decoded sequence length, conventional schedulers face an inefficient dilemma under high load: either schedule conservatively and underutilize compute, or schedule aggressively and trigger memory exhaustion, forcing active requests to be evicted and later recomputed via costly prefill passes. Spexis introduces an ordinal-threshold remaining-length prediction mechanism coupled with lookahead memory simulation (Algorithm 1).
A compact three-layer MLP processes the hidden representations from the drafting layer to predict whether a request's remaining output length falls below 4, 8, 16, 32, or 64 tokens. With prediction precision exceeding 83%, the scheduler simulates the upcoming 64 decoding iterations using binomial distributions to compute request completion probabilities \(P^{\text{fin}}(r)\) and future KV cache release patterns. Using this lookahead trajectory, the scheduler calculates the analytic throughput gain \(G\) of delaying request admission versus aggressive immediate admission:
where \(p_{\text{dr}} = t_p / t_d\) represents the prefill-to-decode latency ratio, \(n\) is the projected number of evictions under aggressive admission, and \(i^k_{\text{wait}}, i_{\text{delay}}\) denote waiting iterations before recomputation and scheduled admission delay, respectively. The scheduler postpones request entry if and only if \(G > 1\), effectively shielding active requests from cache evictions and maximizing sustained service throughput.
Loss & Training¶
Spexis requires only lightweight auxiliary components without retraining base model weights: 1. Drafting Adapter: Designed with a linear projection layer and an attention layer followed by the frozen pre-trained LM head, inserted at the \(1/2\) mark (two-stage pipeline) or \(1/4\) mark (four-stage pipeline) of the base architecture. It is trained on the ShareGPT dataset using the AdamW optimizer for 5 epochs with standard cross-entropy loss. For Llama-3.3-70B, adapter training requires approximately 12 hours on a single H200 GPU. 2. Remaining-Length Predictor: Formulated as a three-layer MLP with two GELU activations, trained via binary cross-entropy on ShareGPT to output confidence scores across five length thresholds (4 to 64 tokens). Training completes in approximately 15 hours on a single H100 GPU (20 epochs). During deployment, to balance memory across pipeline stages when adding an LM head to the first stage, Spexis offloads input embedding weights (which match the LM head size) to host RAM and performs CPU table lookups, introducing negligible latency overhead (~several hundred microseconds) while maintaining pipeline symmetry.
Key Experimental Results¶
Main Results¶
Evaluations span private clustersโboth inter-node (2 nodes, each with 1 NVIDIA RTX PRO 6000 GPU over 1GbE) and intra-node (8 NVIDIA A40 GPUs over PCIe 4.0)โas well as public cloud Runpod instances (8-GPU nodes spanning H100 SXM, A100 SXM, RTX PRO 6000, and L40S) across Llama-3.3-70B and Qwen3-32B on SpecBench, LMSYS-Chat, and UltraChat.
The fixed-batch decoding throughput (tokens/s at batch size 64) across 8-GPU Runpod configurations is summarized below (from Table 2 of the original paper):
| Hardware Config | Interconnect | Evaluated Model | Spexis (Ours) | TP-only Baseline | Speedup |
|---|---|---|---|---|---|
| H100 SXM | NVLink via NVSwitch | Llama-3.3-70B | 3587 | 3652 | 0.98ร |
| H100 SXM | NVLink via NVSwitch | Qwen3-32B | 2402 | 5068 | 0.47ร |
| A100 SXM | NVLink via NVSwitch | Llama-3.3-70B | 1228 | 966 | 1.27ร |
| A100 SXM | NVLink via NVSwitch | Qwen3-32B | 1454 | 2058 | 0.71ร |
| RTX PRO 6000 | PCIe 5.0 ร16 | Llama-3.3-70B | 1492 | 1236 | 1.21ร |
| RTX PRO 6000 | PCIe 5.0 ร16 | Qwen3-32B | 2201 | 2062 | 1.07ร |
| L40S | PCIe 4.0 ร16 | Llama-3.3-70B | 993 | 930 | 1.07ร |
| L40S | PCIe 4.0 ร16 | Qwen3-32B | 1509 | 1473 | 1.02ร |
In inter-node network-constrained deployments (2 RTX PRO 6000 nodes connected via 1Gbps Ethernet), Spexis demonstrates substantial advantages over PP: achieving up to 34% higher request serving rates (Qwen3-32B on LMSYS-Chat) under end-to-end Poisson arrival loads, and up to 54% higher decoding throughput at large batch sizes (Llama 64โ100, Qwen 128โ256).
Ablation Study¶
Incremental gains from adding Speculative Parallelism (SP), Selective Speculation (Selective), and Remaining-Length-Aware Scheduling (Length-Aware) on SpecBench are reported below (from Figure 8 of the original paper):
| Model Config | Component Added | Cumulative Gain vs. Baseline | Note |
|---|---|---|---|
| Llama-3.3-70B | Baseline (PP baseline) | 0.0% | Standard two-stage pipeline parallelism |
| + SP (Speculative Parallelism) | +10.0% | Overlaps drafting & verification without extra KV cache | |
| + Selective (\(\alpha=0.15\)) | +13.0% (+3 pp) | Discards lowest 15% draft tokens, boosting acceptance | |
| + Length-Aware (Full model) | +16.0% (+3 pp) | Lookahead simulation postpones eviction recomputation | |
| Qwen3-32B | Baseline (PP baseline) | 0.0% | Standard two-stage pipeline parallelism |
| + SP (Speculative Parallelism) | +15.0% | Basic speculative overlap gains | |
| + Selective (\(\alpha=0.30\)) | +27.0% (+12 pp) | Dramatically trims wasted draft cycles on weaker base model | |
| + Length-Aware (Full model) | +30.0% (+3 pp) | Prevents secondary prefill penalties during load spikes |
Key Findings¶
- Selective speculation is critical for models with moderate drafting confidence: On Qwen3-32B where baseline self-speculation accuracy is near 60%, standalone SP provides a 15% gain, whereas pruning the lowest 30% confidence tokens raises the effective acceptance rate from 62.6% to 76.7%, yielding an additional 12 percentage points in throughput.
- Interconnect bandwidth defines the Pareto frontier between SP and TP: On commodity PCIe buses and Ethernet-linked distributed nodes, TP suffers heavy communication latency from All-Reduce barriers, allowing Spexis to decisively outperform TP (speedups of 1.02รโ1.21ร on PCIe and 1.34ร on Ethernet). Conversely, inside pure NVSwitch fabrics (e.g., H100 SXM) where TP synchronization overhead is minimal, low draft acceptance can lead to net compute overheads, favoring pure TP.
- Ordinal threshold predictions reliably avert memory thrashing: The three-layer MLP achieves 70%โ90.7% precision across sequence length thresholds, empowering the lookahead scheduler to preemptively halt false-alarm evictions and eliminate redundant prefill passes.
Highlights & Insights¶
- Reframing speculative decoding as a system-level parallelism axis: Replaces the conventional view of speculative decoding as a purely algorithmic trick by elevating it into a core scheduling primitive that pairs with TP and PP. Sharing KV cache allocations across pipeline stages permanently overcomes the micro-batch memory bottleneck in pipeline parallelism.
- Analytic throughput modeling of hybrid parallelism tradeoffs: Introduces closed-form steady-state formulation linking drafting acceptance rates \(\Theta\), pruning ratios \(\alpha\), pipeline stages \(N\), and hardware latency constants, giving distributed inference engines a principled tool for automatic topology selection.
- Asymmetric memory rebalancing via CPU offloading: Resolves the memory imbalance caused by adding an extra LM head to the first pipeline stage by offloading token embeddings to host CPU RAM, achieving stage symmetry at the cost of only minor lookup latencies.
Limitations & Future Work¶
- Overhead of auxiliary model training: Deploying Spexis necessitates training both a drafting adapter and a length prediction MLP on target domains. While training cost is modest (~12โ15 GPU hours), it introduces an extra offline maintenance step in rapid model iteration cycles.
- Sensitivity to out-of-distribution shifts: System throughput relies heavily on stable token acceptance rates and calibrated length predictions. Severe distribution shifts (e.g., complex coding or obscure language queries) can degrade drafting accuracy, eroding throughput advantages.
- Exploration in deep multi-stage pipelines: Current empirical validations center on two-to-four stage pipelines. Extending speculative parallelism into deep multi-node clusters with high stage depths warrants further theoretical and scheduling investigations.
Related Work & Insights¶
- vs. Conventional Pipeline Parallelism (GPipe, Orca): Traditional systems split batches into independent micro-batches to mitigate bubbles, incurring severe KV cache inflation. Spexis avoids this completely by sharing physical KV cache between speculation and verification stages.
- vs. Self-Speculative Decoding (Kangaroo, LayerSkip, Draft&Verify): Prior self-speculative approaches focus on single-GPU sequential draft-then-verify execution. Spexis is the first to map this structure directly across physical multi-GPU pipeline boundaries for true concurrent execution.
- vs. Length-Aware Schedulers (S3, LTR, TRAIL): Existing length prediction methods prioritize short jobs (SJF) to reduce queue wait times. Spexis specifically targets lookahead KV cache saturation and formulates an explicit cost-benefit model (\(G\)) balancing delayed admission against eviction recomputation.
Rating¶
- Novelty: โญโญโญโญโญ [Pioneering integration of speculative decoding as an orthogonal multi-GPU scheduling primitive]
- Experimental Thoroughness: โญโญโญโญโ [Evaluated across diverse hardware topologies, PCIe/SXM fabrics, and backed by validated cost models]
- Writing Quality: โญโญโญโญโญ [Clear motivation, rigorous mathematical derivations, and structured execution pipeline]
- Value: โญโญโญโญโญ [Directly tackles the fundamental tension between communication and memory in large-scale LLM serving systems]