outlets output length prediction from speculative decoding backbones
content_hash: 9f6fe20f4fa14be6 ---# OUTLETS: Output-Length Prediction from Speculative Decoding Backbones
Conference: EMNLP 2026
Paper: ACL Anthology
Area: Model Compression
Keywords: Speculative Decoding, Output-Length Prediction, Cluster Scheduling, Representation Reuse, LLM Serving
TL;DR¶
OUTLETS reveals an intrinsic structural connection between speculative lookahead representations and sequence termination forecasting, attaching a lightweight log-space regression head to a shared speculative decoding backbone to deliver accurate length predictions that slash short-request P99 latency by 34.8% in disaggregated serving.
Background & Motivation¶
In production Large Language Model (LLM) serving, autoregressive generation exhibits an extreme heavy-tailed output length distribution, with token lengths varying across multiple orders of magnitude across applications (such as summarization vs. code generation) and even within the same model family. Modern high-concurrency systems increasingly adopt disaggregated serving architectures that decouple prefill and decode stages onto specialized hardware nodes. However, when concurrency slots or KV-cache capacity reach saturation, conventional First-Come-First-Served (FCFS) admission and Round-Robin (RR) instance dispatch create severe system bottlenecks: short, latency-sensitive requests become trapped behind unpredictably long requests, triggering severe Head-of-Line (HOL) blocking that deteriorates tail latency and induces severe inter-instance load imbalance.
To enable Shortest-Job-First (SJF) prioritization and proactive workload-balanced routing, accurate output-length prediction has emerged as a fundamental primitive for scheduling control planes. Yet existing predictors face an unavoidable trade-off between runtime overhead and predictive fidelity: external proxy models (such as BERT or classification models) introduce non-trivial memory footprints and inter-process latency while lacking visibility into target model internal dynamics; meanwhile, internal state-based probes (such as shallow MLPs mounted on frozen target hidden states) only observe representations tightly specialized for immediate next-token prediction, lacking the temporal foresight required to anticipate multi-step continuation trajectories.
Concurrently, speculative decoding (SD) accelerates autoregressive serving by deploying a drafter that iteratively rolls representations forward to propose future token trajectories, which inherently encodes macro-level structural signals regarding sequence evolution and eventual termination. Core idea: repurpose the speculative decoding backbone as a trajectory-aware length predictor, attaching a lightweight log-space regression head to the shared draft decoder representations to supply high-fidelity length signals to cluster schedulers at minimal marginal computational cost.
Method¶
Overall Architecture¶
OUTLETS unifies token drafting for speculative acceleration and sequence length regression within a single shared computational graph. At prefill completion or during each decoding iteration, multi-level hidden states are extracted across frozen target LLM layers and projected into a compact draft embedding space via a feature fusion layer; subsequently, a lightweight Transformer draft decoder equipped with Gated Attention rolls the representations forward, capturing both fine-grained lexical transitions and long-range termination trends; finally, a dual-head formulation decodes the forward states into vocabulary candidate probabilities via the draft head and scalar remaining-length predictions via the length regression head.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Target LLM multi-level hidden states<br/>H = {h^(2), h^(N/2), h^(N-2)}"] --> B["1. Multi-Level Hidden State Feature Fusion<br/>Cross-depth concatenation and projection to fctx"]
B --> C["2. Gated Attention Draft Decoder<br/>Forward lookahead state hdraft_t without attention sink"]
C --> D["3. Dual-Head Decoupled Co-Prediction<br/>Joint draft token distribution & log-length regression"]
D -->|Scheduling Control Plane| E["Instance Load Balancing (LB)<br/>& Queue Shortest-Job-First (SJF)"]
Key Designs¶
1. Multi-Level Hidden State Feature Fusion: capturing cross-depth semantic and syntactic priors Relying exclusively on the final hidden layer of the target model yields representations hyper-specialized for immediate next-token vocabulary selection, stripping away broader hierarchical abstractions needed for trajectory forecasting. OUTLETS selects a subset of target layers \(I = \{2, \lfloor N/2 \rfloor, N - 2\}\) encompassing shallow lexical features, intermediate syntactic patterns, and deep semantic abstractions, concatenating them along the feature dimension and projecting into draft dimension \(d\): $\(f_{\text{ctx}} = W_{\text{proj}} \left[ \text{Concat}\left(\{h^{(i)}\}_{i \in I}\right) \right]\)$ The resulting fused feature \(f_{\text{ctx}}\) provides a static contextual anchor that supports both candidate token generation and global sequence-termination estimation.
2. Gated Attention Draft Decoder: simulating future trajectories without attention sink To transcend the expressivity limits of shallow target probes that observe only current step activations, OUTLETS leverages a lightweight Transformer decoder \(M_{\text{draft}}\). Rather than adopting standard Llama attention as in EAGLE-3, OUTLETS incorporates Gated Attention into the draft decoder layers. The gating mechanism mitigates attention sink anomalies during autoregressive rolling, preserving long-range termination cues and stabilizing lookahead latent states \(h_t^{\text{draft}}\) across multiple speculative rollout steps.
3. Dual-Head Decoupled Co-Prediction: decoupling local vocabulary logits from global log-length regression On top of the shared forward draft state \(h_t^{\text{draft}}\), OUTLETS bifurcates inference into two specialized task heads. The draft model head applies a linear projection \(W_{\text{lm}}\) to generate draft token distribution \(P_{\text{draft}}(y_{t+1} \mid y_{1:t})\); concurrently, the length regression head addresses the severe heavy-tailed length distribution by mapping the target into log-space \(z = \log(1 + \text{length})\). A multi-layer perceptron \(\text{MLP}_{\text{len}}\) with progressively shrinking hidden dimensions maps the draft state to a scalar prediction \(\hat{z}_t = \text{MLP}_{\text{len}}(h_t^{\text{draft}})\). This formulation avoids large numerical gradient instability and naturally prioritizes relative prediction error over absolute deviation.
Loss & Training¶
The framework is optimized end-to-end using a composite multi-task objective: $\(\mathcal{L} = \mathcal{L}_{\text{SD}} + \lambda \mathcal{L}_{\text{len}} + \gamma \|\theta_{\text{MLP}}\|_2^2\)$ where \(\mathcal{L}_{\text{SD}}\) denotes the speculative drafting KL-divergence, \(\mathcal{L}_{\text{len}}\) is the mean squared error (MSE) evaluated in log-space, and the final term imposes L2 regularization on regression MLP parameters \(\theta_{\text{MLP}}\) (\(\gamma = 10^{-5}\)). Empirical evaluation shows that a fixed weight \(\lambda = 0.1\) outperforms dynamic schemes like Uncertainty Weighting and GradNorm, providing robust structural supervision without compromising speculative acceptance. The model is trained via AdamW with learning rate \(1\times 10^{-5}\), weight decay 0.02, and BF16 mixed precision across 10 epochs.
Key Experimental Results¶
Main Results¶
Evaluating Mean Absolute Error (MAE, lower is better) on ShareGPT, Alpaca, and LMSYS across static prefill (\(t=0\)) and dynamic decoding stages:
| Model | Scenario | Dataset | Target Mean (Avg. Len) | OUTLETS | MLP Baseline | BERT Proxy | OPT Proxy | PIA Prompting |
|---|---|---|---|---|---|---|---|---|
| Llama-3.2-1B | Static | ShareGPT | 324.4 ± 268.9 | 100.1 | 109.5 | 128.6 | 130.6 | 204.3 |
| Llama-3.2-1B | Static | Alpaca | 237.6 ± 198.3 | 53.4 | 56.9 | 70.0 | 70.4 | 132.7 |
| Llama-3.2-1B | Static | LMSYS | 246.6 ± 276.8 | 69.6 | 73.6 | 83.7 | 84.5 | 171.8 |
| Llama-3.2-1B | Dynamic | ShareGPT | 274.2 ± 226.7 | 63.8 | 72.5 | — | — | — |
| Llama-3.2-1B | Dynamic | Alpaca | 202.1 ± 175.8 | 31.8 | 39.1 | — | — | — |
| Llama-3.2-1B | Dynamic | LMSYS | 279.1 ± 256.7 | 41.7 | 50.3 | — | — | — |
| Llama-3.1-8B | Static | ShareGPT | 355.9 ± 276.7 | 80.6 | 84.9 | 132.8 | 135.2 | 283.3 |
| Llama-3.1-8B | Dynamic | ShareGPT | 286.0 ± 232.9 | 67.6 | 90.4 | — | — | — |
| Qwen3-30B-A3B | Static | ShareGPT | 1053.9 ± 536.8 | 186.7 | 210.8 | 262.2 | 255.2 | 936.2 |
| Qwen3-30B-A3B | Dynamic | ShareGPT | 664.1 ± 460.8 | 117.1 | 158.4 | — | — | — |
End-to-end serving performance in a disaggregated cluster (1 prefill + 3 decode instances) under 100 QPS saturation stress testing:
| Scheduling & Prediction Policy | System Throughput (tokens/s) | Avg Latency (s) | Short P99 (s) [<800 tok] | Medium P99 (s) [800-1300 tok] | Long P99 (s) [≥1300 tok] |
|---|---|---|---|---|---|
| RR + FCFS (vLLM Baseline) | 17,840.4 | 51.6 | 59.8 | 74.6 | 103.0 |
| RR + SJF (OUTLETS) | 18,108.2 | 50.8 | 38.0 | 61.7 | 100.8 |
| LB + SJF (OUTLETS) | 18,434.7 | 50.5 | 39.0 | 60.1 | 100.1 |
Ablation Study¶
Comparing joint training against specialized single-task length predictors (LP-ONLY) across model families:
| Model Config | Scenario | Dataset | OUTLETS (Joint) | Length-Only Model (LP-ONLY) | Note |
|---|---|---|---|---|---|
| Llama-3.2-1B | Static | ShareGPT | 100.1 | 97.1 | Sparse supervision; performance is closely matched |
| Llama-3.2-1B | Static | Alpaca | 53.4 | 54.1 | SD regularization mitigates static overfitting |
| Llama-3.2-1B | Static | LMSYS | 69.6 | 71.7 | OUTLETS demonstrates superior generalization |
| Llama-3.2-1B | Dynamic | ShareGPT | 63.8 | 60.3 | Dense step supervision gives LP-ONLY slight edge |
| Llama-3.1-8B | Static | ShareGPT | 80.6 | 81.1 | Joint training preserves representation versatility |
| Llama-3.1-8B | Dynamic | ShareGPT | 67.6 | 65.3 | Modest gap (only 2.3 MAE) |
| Qwen3-30B-A3B | Static | ShareGPT | 186.7 | 195.0 | Joint training wins on large MoE reasoning model |
| Qwen3-30B-A3B | Dynamic | ShareGPT | 117.1 | 103.3 | Dedicated capacity provides slight advantage |
Key Findings¶
- Zero degradation in speculative drafting quality: Acceptance rate curves across drafting steps 1–3 on the test set show that OUTLETS and the draft-only model (SD-ONLY) overlap almost identically, confirming the draft decoder has ample capacity to host both next-token generation and trajectory termination signals.
- Draft decoder layer is the key contributor: Ablation of EAGLE-3 architectural components shows that removing the draft decoder incurs the largest increase in length prediction MAE, confirming that static fused hidden states must undergo temporal decoding to manifest predictive termination signals.
- Direct regression outperforms bucketed classification: Continuous regression in log-space preserves ordinal distance metrics, consistently outperforming 50-bin classification while avoiding boundary quantization artifacts.
- Prioritizing short jobs does not starve long jobs: Under LB + SJF, while short-request P99 latency drops by 34.8%, long-request P99 latency also slightly decreases from 103.0s to 100.1s, demonstrating that eliminating head-of-line blocking cleanses queue fragmentation and benefits the entire cluster.
Highlights & Insights¶
- Structural synergy between drafting and length forecasting: Token-level speculative drafting inherently builds prospective multi-step sequences, naturally embedding sequence termination signals into its latent representations.
- Negligible marginal cost for deployed SD setups: In serving clusters already using speculative decoding, OUTLETS introduces only 2.6M parameters and 0.7 ms runtime overhead per step to deliver high-quality scheduling primitives.
- Strong zero-shot cross-domain generalization: Trained solely on LMSYS dialogue, a Qwen3-30B predictor evaluated out-of-domain on GSM8K math reasoning (MAE 397.9, Avg. Len 1561.2) and HumanEval code generation (MAE 506.6, Avg. Len 2324.5) achieves relative errors around 22–25%.
Limitations & Future Work¶
- Dependency on speculative backbone presence: When speculative decoding is disabled or absent, deploying the standalone draft backbone introduces cold-start memory and compute overhead (72.3M params, 1.7 ms per step for Qwen3-30B).
- Unexplored dynamic rescheduling loop: The present system implementation restricts scheduling decisions to static prefill estimates (\(\hat{r}_0\)); online dynamic predictions (\(\hat{r}_t\)) have not yet been evaluated for active preemption or live request migration across instances.
- Trajectory length boundaries: Experiments evaluate outputs bounded within 2048/4096 tokens; behavior under ultra-long agentic chains, high sampling temperatures, or non-deterministic stopping criteria remains to be investigated.
Related Work & Insights¶
- vs TRAIL / ARES / Overclocking: Standard internal activation methods attach probes directly to the target LLM's top layers, constrained by states optimized for single-step token prediction; OUTLETS exploits the draft decoder's prospective trajectory modeling to achieve significantly lower MAE.
- vs S3 / BERT / OPT / ShuffleInfer: External proxy regressors require auxiliary prompt encoding and lack runtime target state context; OUTLETS leverages shared target representations, eliminating external proxy latency and memory overhead.
- vs PIA: Prompt-based self-prediction (PIA) alters prompt templates and burns costly autoregressive generation cycles; OUTLETS functions as a non-intrusive control plane predictor entirely decoupled from generated text.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ First study to uncover and rigorously validate the structural connection between speculative decoding representations and output-length prediction.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 3 model scales, static/dynamic regimes, component ablations, and a 4-instance disaggregated vLLM cluster test.
- Writing Quality: ⭐⭐⭐⭐⭐ Clean mathematical formulation and clear narrative connecting representations to system throughput.
- Value: ⭐⭐⭐⭐⭐ Highly practical, drop-in optimization for modern high-concurrency LLM serving architectures.