⚡ LLM Efficiency¶
🧠 NeurIPS2026 · 79 paper notes
📌 Same area in other venues: 📷 CVPR2026 (8) · 🔬 ICLR2026 (171) · 💬 ACL2026 (23) · 🧪 ICML2026 (48) · 🤖 AAAI2026 (9) · 🧠 NeurIPS2025 (34)
🔥 Top topics: LLM ×15 · Diffusion Models ×9 · Agents ×3
- A Statistical Theory of Gated Attention through the Lens of Hierarchical Mixture of Experts
-
This paper establishes the first mathematical equivalence between multi-head self-attention/gated attention and three-level Hierarchical Mixture of Experts (HMoE), rigorously proving that non-linear gating reduces expert estimation sample complexity from exponential \(\mathcal{O}(\exp(\epsilon^{-1/\tau}))\) in standard attention to polynomial \(\mathcal{O}(\epsilon^{-4})\), and uncovering why gating Value or SDPA outputs eliminates degenerate PDE coupling.
- Accelerating Diffusion Language Models via Structured Suffix Modeling
-
Addressing the computational redundancy of full suffix attention and the loss of iterative denoising history in block-wise diffusion language models, this paper proposes Structured Suffix Modeling (SSM)—a training-free framework that allocates distinct token budgets across local, middle, and tail suffix regions while fusing previous-step decoding probabilities via soft embeddings, achieving up to a 72.81× end-to-end speedup while matching or improving generation accuracy.
- Attention-Discounted Adaptive Sampler for Masked Diffusion Language Models
-
Without training the model or changing the base sampler's stopping rule, ADAS dynamically discounts confidence using a candidate's attention to positions already selected in the current step and their uncertainty, improving parallel decoding at low numbers of denoiser evaluations; task-and-method-average gains are 9.11 and 10.46 percentage points for the two models.
- Block Sparse Flash Attention
-
BSFA first computes all causally visible QK scores exactly inside FlashAttention-2, then uses offline-calibrated block-maximum thresholds to skip V loads, PV, and softmax-state updates for low-scoring blocks, achieving a LongBench score of 39.78% versus the dense baseline's 40.24% on Llama-3.1-8B and a 1.13× end-to-end prefill speedup on the longest 10 samples.
- Communication-Efficient LLM Adaptation over Decentralized GPU Meshes
-
This paper proposes an asynchronous two-circuit training architecture for decentralized post-pretraining LLM adaptation, combining aggressive in-graph activation masking for pipeline parallelism with an off-critical-path anchor circuit and spectral correction optimizer to eliminate isotropic compression noise, achieving over 40× training throughput gains across commodity 200 Mbps internet links while fully matching dense adaptation performance.
- CommunityKV: Efficient Long-Context Decoding via Graph Partitioning
-
CommunityKV formulates training-free sparse attention as a community detection problem on an attention-induced token graph, combining direct and co-attention relationships with Leiden partitioning during prefill and constant-time greedy updates during decoding to enable sub-linear retrieval of semantically coherent KV chunks.
- CompactAttention: Accelerating Chunked Prefill with Block-Union KV Selection
-
CompactAttention tackles the kernel inefficiency and explicit KV-copy overhead in chunked prefill (\(Q \ll KV\)) by lowering 2D block-sparse masks into GQA-aware KV block tables through Q-block and intra-group unions, enabling zero-copy in-place execution on top of dense paged-attention engines.
- Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse
-
Addressing the substantial prefill overhead incurred when switching between different-sized models in an LLM family, this paper proposes a gradient-free, per-head closed-form ridge mapper with RoPE decoupling that achieves a 2.7–25× prefill speedup while retaining 73–98% of standalone downstream accuracy on matched-KV pairs.
- D-PACE: Dynamic Position-Aware Cross-Entropy for Parallel Speculative Drafting
-
Addressing the failure of fixed position-decay schedules in block-diffusion parallel drafters to match dynamic acceptance bottlenecks during training, D-PACE derives closed-form position weights by differentiating a smooth surrogate of expected accepted draft length, boosting wall-clock speedup and emitted length across multiple models with only 2.3% training overhead and zero inference modifications.
- DivMoE: Fine-Grained MoE Upcycling via Cross-Domain Expert Composition
-
Addressing the catastrophic routing collapse in single-source fine-grained MoE upcycling, DivMoE introduces domain-specialized expert slicing and hard diversity-constrained routing, strictly outperforming dense base models and six upcycling baselines across 15 benchmarks.
- Dooly: Configuration-Agnostic, Redundancy-Aware Profiling for LLM Inference Simulation
-
Addressing the bottleneck where existing LLM inference simulators hardcode operator sets and re-profile configurations from scratch, Dooly uses a single dummy-prompt inference pass with taint propagation to label tensor dimension origins and reuses serving engine context to isolate stateful kernels, delivering configuration-agnostic profiling that cuts GPU-hours by 56.4% across 12 models while maintaining simulation error under 8%.
- Dual Dimensionality for Local and Global Attention
-
Addressing the capacity redundancy of assigning uniform key-value representation dimensionality to all tokens in autoregressive Transformers, this paper proposes Distance-Adaptive Representation (DAR), which preserves full dimensionality within a local context window and applies a lower-dimensional bottleneck projection to distant tokens, matching or surpassing full-dimensional baselines across 70M–410M pretraining and 1B fine-tuning setups.
- Dynamic Delayed Tree Expansion for Improved Multi-Path Speculative Decoding
-
Addressing the issue of wasteful early branching in multi-path speculative decoding, this paper introduces delayed tree expansion along with a lightweight neural selector, enabling optimal-transport-based verification to outperform Traversal Verification for the first time with an average 5% throughput speedup.
- Dynamic Expert Sharing: Decoupling Memory from Parallelism in Mixture-of-Experts Diffusion LLMs
-
To tackle the "expert explosion" memory bottleneck in MoE diffusion LLMs during multi-token parallel decoding, Dynamic Expert Sharing (DES) dynamically elects a compact sequence-level coreset via saliency-aware voting, slashing unique expert activations by over 55% and latency by up to 38% with 99% accuracy retention.
- Efficient Serving for Dynamic Agent Workflows with Prediction-based KV-Cache Management
-
To tackle runtime execution uncertainty and loop dynamics in agentic workflows where traditional LRU and static DAG heuristics fail, this paper presents PBKV, a prediction-based KV-Cache serving system that fuses global call graphs, execution prefixes, and prefill semantics for multi-step agent prediction alongside hierarchical eviction and conservative prefetching, achieving up to 1.85× end-to-end speedup and 2.55× cache hit rate improvement.
- Efficiently Aligning Draft Models via Parameter- and Data-Efficient Adaptation
-
To resolve the speculative decoding degradation caused by distribution shifts after domain-specific target model fine-tuning, EDA introduces a parameter- and data-efficient adaptation framework with decoupled shared-private gated experts, target self-generation, and representation-shift sample selection, significantly restoring acceptance rates and speedups with minimal training overhead.
- Elastic Spectral State Space Models for Train-Once Budgeted Inference
-
Addressing the deployment bottleneck across heterogeneous platforms, ES-SSM leverages Hankel spectral operator approximation coupled with input-adaptive gating and log-scale budget dropout to enable train-once, export-many budgeted inference without search or fine-tuning.
- Emo: Pretraining Mixture of Experts for Emergent Modularity
-
Emo introduces a self-supervised MoE pretraining objective that constrains tokens within a document to route through a shared expert pool without requiring human-labeled domains, enabling coherent semantic modularity to emerge such that retaining only 12.5%–25% of experts incurs virtually no performance loss for domain-specific deployment.
- Fast and Stable Triangular Inversion for Delta-Rule Linear Transformers
-
Addressing the severe performance bottleneck and acute numerical instability of chunk-wise triangular matrix inversion in Delta-Rule Linear Transformers, this paper systematically analyzes direct and iterative inversion methods and proposes a Mixed Recursive Inversion algorithm (MXR) combining micro-block Cayley-Hamilton powers with unrolled Bunch-Hopcroft recursion, achieving up to 4.3x kernel speedup and 2.5x full-layer GDN speedup on NPUs with zero end-to-end accuracy loss.
- Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction
-
Addressing the severe trade-off between compression overhead and fidelity in existing KV cache eviction methods, Fast KVzip introduces lightweight sink-attention gating modules trained with frozen base LLM weights to directly predict KV importance from hidden states, enabling up to 70% KV cache eviction across prefill and decoding with near-lossless accuracy and minimal runtime overhead.
- Fingerprinting Inference Systems of Large Language Models
-
By crafting targeted prompt sets for prefill, chunked prefill, KV-cache reuse, and autoregressive decoding, this paper demonstrates that subtle numerical deviations caused by floating-point non-associativity expose the inference engine, attention backend, and GPU hardware platform using only black-box text responses.
- FLARE: Diffusion for Hybrid Language Model
-
FLARE presents the first diffusion conversion framework tailored for hybrid-attention (Softmax + Gated DeltaNet linear attention) LLMs, combining a token-balanced clean/noisy dual-stream objective, register-level hardware state scheduling kernels, and a unified serving engine to retain Qwen3.5 capabilities across 2B/4B/9B scales with only 10B transfer tokens while boosting single-GPU serving throughput up to 4.8x over open-source dLLM baselines.
- Flash-dLLM: IO-Aware KV Caching and Parallel Decoding for Fast, Memory-Efficient Diffusion LLMs
-
Flash-dLLM introduces a training-free inference acceleration framework for diffusion LLMs (dLLMs) that combines an IO-aware fused KV-cache kernel in SRAM, selective high-attention token tracking, and a self-contained draft-and-verify parallel decoding scheme, achieving up to 148.2× speedups on reasoning and code generation tasks.
- FlashMask-3: Efficient and Expressive Mask-Aware Distributed Attention
-
FlashMask-3 reveals the zero-communication clip–shift–max localization property of column-wise sparse masks, coupling a three-role Hopper warp-specialized pipeline with topology-aware compute-communication overlap to deliver highly expressive and fast distributed attention for long-context training.
- Fractional State Space Transition for Long Sequence Modeling
-
Frac approximates fractional long memory with finite exponential modes on a shared geometric timescale bank, then adds token-wise controls and independent read/write routing, preserving bounded recurrent state while raising the 1.3B model's average LongBench score from GDN's 16.0 to 17.9, without leading on every task or throughput measure.
- From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs
-
This paper frames the adaptation of autoregressive (AR) LLMs into diffusion models as a continuous progression under the Block-Diffusion paradigm, introducing context-causal attention, a one-pass parallel training mask with auxiliary AR guidance, and a gradual block growth curriculum to deliver NBDiff-7B, a state-of-the-art 7B diffusion model with 32K context and strong reasoning.
- Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
-
Addressing the architectural constraint of tied scalar gates for key-side erasing and value-side writing in delta-rule recurrent attention, Gated DeltaNet-2 introduces channel-wise erase gates \(\bm{b}_t\) and write gates \(\mathbf{w}_t\) along with a decay-absorbing chunkwise WY parallel training formulation, outperforming Mamba-2, KDA, and Mamba-3 across language modeling and long-context multi-key retrieval.
- Geometry-Aware Online Scheduling for LLM Serving: From Theoretical Bound to System Practice
-
To capture the 2D spatio-temporal geometric growth of KV cache footprints in autoregressive LLM inference, this paper proposes Smallest Volume First (SVF) and an ultra-lightweight 1-bit SVF online scheduling algorithm, establishes a worst-case approximation ratio approaching 3, and achieves significant reductions in average and tail latencies within vLLM.
- HACO: Hedged Agent Computing for Reliable LLM Systems
-
Addressing runtime binding failures in multi-agent workflows under fluctuating service regions and networks, HACO models role invocations as reliability-constrained hedge subset selections, achieving high robustness and low latency via adaptive redundancy and online experience harvesting.
- HAPS: Hierarchical LLM Routing with Joint Architecture and Parameter Search
-
Addressing the core limitation of conventional LLM routers that treat candidate models as static black boxes and restrict search to discrete architectures, HAPS introduces a hierarchical routing framework that jointly searches discrete architectures and continuous parameters via shared hypernetworks and reward-augmented reinforcement learning, substantially outperforming existing routing baselines while minimizing token costs.
- HYVE: Hybrid Views for LLM Context Engineering over Machine Data
-
Addressing context explosion and attention rot caused by nested, repetitive machine data in LLM prompts, HYVE adapts database management principles to build a request-scoped in-memory datastore with complementary row/column views, cutting token usage by 50–90% while drastically improving structured generation and range-query analytical accuracy through delayed query backfilling.
- Interleaved Head Attention
-
To overcome the linear scaling constraint of multi-head attention caused by head isolation, this paper introduces Interleaved Head Attention (IHA), which constructs pseudo-heads via cross-head linear mixing and interleaves them as an expanded virtual sequence, enabling up to quadratic attention patterns per head and significantly boosting multi-step reasoning and retrieval efficiency while remaining fully compatible with FlashAttention.
- Intrinsic Selection and Particle Resampling for Inference-Time Scaling Beyond Domain Verifiability
-
Addressing the bottleneck of expensive or unavailable external verifiers in non-verifiable and hard domains, this paper proposes exploiting intrinsic sample-set tail entropy for candidate selection (iS), step-level particle resampling (iPF), and privileged distillation (dPF) without trained reward models or ground-truth verification.
- It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs
-
Synth expands sourced encyclopedic seeds into heterogeneous instruction and reasoning data, enabling the final Baguettotron student to acquire instruction-following and factual capabilities from random initialization in one training process; the 600M variant reaches 79.3% precision on verifiable facts about seed entities, but this does not establish broad knowledge coverage or assume that teachers and data generation are free.
- LaCache: Robust Semantic Caching for LLM Serving
-
To eliminate cache-collision vulnerabilities in LLM serving where crafted queries inject adversarial outputs into semantic caches, LaCache shifts integrity verification from the query to the response via speculative lookahead checks on the first \(k\) tokens, slashing attack hit rates to near zero while retaining over 90% of benign cache utility with minimal serving overhead.
- LEAD: Length-Efficient Adaptive and Dynamic Reasoning for Large Language Models
-
To tackle verbose reasoning trajectories and overthinking in RL-trained LLMs, LEAD introduces an online dynamic reward balancing controller driven by Potential-Scaled Instability (PSI) and a per-problem target length derived from the mean of correct rollouts with symmetric efficiency penalties, achieving superior accuracy and Accuracy-Efficiency Scores across mathematical benchmarks.
- Learn from the Gap: Differential-Aware Advantage Pruning with Adaptive Rollout Sampling for GRPO
-
Addressing the computational bottleneck and gradient redundancy caused by multi-rollout sampling in Group Relative Policy Optimization (GRPO), this paper proposes FastRL—a framework that combines maximizing differential advantage pruning (MDAP) to eliminate collinear gradient updates with adaptive rollout sampling (ARS) to dynamically scale generation budgets, achieving a 2.07×-2.22× speedup and superior reasoning accuracy.
- Learnable Low-Rank Polynomial Sketch for Effective Linear Attention
-
The paper proposes Learnable Low-Rank Polynomial Sketch (LLoPS), which theoretically establishes the optimal linear polynomial sketch and factorizes it into lightweight projections and Hadamard products via rank-1 CP tensor decomposition, achieving strictly lower approximation error with \(O(ndmtp)\) complexity and outperforming existing linear attention baselines.
- Learning Evidence Highlighting for Frozen LLMs
-
HiLight proposes an evidence emphasis framework that trains a lightweight Emphasis Actor to insert minimal highlight tags into unaltered context without discarding raw text, decoupling evidence selection from downstream reasoning for frozen LLM solvers via weakly supervised reinforcement learning.
- Learning Subspace-Preserving Sparse Attention Graphs from Heterogeneous Multiview Data
-
To tackle subspace structure distortion and dense cross-subspace noise when extracting features from diverse pretrained models, this paper proposes Sparse Attention Graph Learning (SAGL), which combines bilinear asymmetric attention factorization, dynamic sparsity gating, and \(\alpha\)-entmax probability simplex projection to construct block-diagonal sparse directed graphs for residual information aggregation without costly iterative algebraic solvers.
- Load Balancing Mixture of Experts with Similarity Preserving Routers
-
Addressing the issue where conventional auxiliary load balancing losses enforce uniform expert usage and induce severe knowledge redundancy across experts, this paper introduces Similarity Preserving Routers (SimBal), applying soft orthogonality regularization to router weights to prevent routing collapse while preserving token semantic geometry, achieving 36% faster training convergence and substantially lower redundancy.
- MARS: Enabling Autoregressive Models Multi-Token Generation
-
MARS proposes a lightweight fine-tuning strategy requiring zero architectural modifications and no extra parameters, jointly training clean and masked streams under causal attention and right-shifted logits to equip standard autoregressive models with controllable multi-token generation while preserving single-token competence, delivering up to a 1.71× wall-clock speedup.
- Mean-Field Parallel Decoding for Discrete Diffusion Language Models
-
Addressing the joint inconsistency issue caused by independent marginal token updates in discrete diffusion language models, this paper introduces a training-free mean-field parallel decoding framework that structures commit selection via JSD-derived pairwise inhibitory potentials, achieving up to 8.62× speedup over entropy decoding in a single forward pass.
- Memory Inception: Latent-Space KV Cache Manipulation for Steering LLMs
-
Memory Inception (MI) is a training-free latent attention steering framework that encodes guidance text into compact key-value memory banks attached only at calibrated sensitive layers and heads, preserving strong steering fidelity and long-term persistence while cutting content-matched KV cache footprint by 6 to 118 times.
- MobileMoE: Scaling On-Device Mixture of Experts
-
MobileMoE establishes a sub-billion active parameter MoE family optimized via an on-device scaling law under mobile memory and compute constraints, achieving 1.8-3.8x faster prefill and 2.2-3.4x faster decode on flagship smartphones at comparable INT4 weight footprints.
- More Value per Key: Asymmetric Sparse Attention for Faster LLM Decoding
-
Addressing the computational asymmetry where sparse attention drastically cuts probability-value multiplication while leaving query-key multiplication as the dominant bottleneck, this work introduces SAGA to decouple key and value head counts alongside Atop-N sparse decoding, achieving over 2× end-to-end decoding speedups at long contexts while preserving model capacity.
- MulCLIP: A Multi-level Alignment Framework for Enhancing Fine-grained Long-context CLIP
-
Addressing standard CLIP's limitation on short coarse captions and existing long-text models' dependence on expensive external region proposals (such as SAM or YOLO), MulCLIP introduces an end-to-end multi-level alignment framework that unifies global contrastive learning, locally calibrated word-patch reconstruction, and subcaption-aggregated patch alignment, achieving superior fine-grained retrieval without segmentation overhead.
- Multi-Token Residual Prediction
-
Multi-Token Residual Prediction (MRP) exploits the high similarity of logit distributions between adjacent diffusion denoising steps to predict step-wise residuals using a lightweight 3-layer module, delivering up to \(1.40\times\) lossless speculative speedup in static decoding and recovering up to 22.6 accuracy points via remasking in aggressive dynamic decoding.
- N-vium: Mixture-of-Exits Transformer for Accelerated Exact Generation
-
N-vium reformulates autoregressive transformers as a learned mixture-of-exits distribution, enabling exact two-step sampling from intermediate prediction heads while recovering complete KV caches by deferring and piggybacking upper-layer computation with subsequent tokens, achieving up to 57.9% wall-clock decoding speedup at 1.5B scale with zero perplexity degradation.
- Neural Garbage Collection: Learning to Forget while Learning to Reason
-
Introduces Neural Garbage Collection (NGC), casting KV cache eviction as discrete actions sampled from the language model and trained end-to-end via policy gradients using only outcome-based task rewards, substantially outperforming heuristic eviction baselines at 2–3x peak cache reductions.
- Orthros: Phase-Aware Heterogeneous Attention for Efficient Transformers
-
Orthros proposes a phase-aware heterogeneous attention mechanism that shifts efficiency allocation from macro-structural mixing to phase decoupling: applying linear sliding window attention across the full sequence to construct the KV cache while selectively executing full cross-attention exclusively at next-token prediction positions, achieving up to 21.9x prefilling speedup with zero recall degradation in long-context processing.
- Orthrus: Memory-Efficient Parallel Token Generation via Dual-View Diffusion
-
Orthrus injects a lightweight diffusion attention module into a frozen autoregressive LLM to pioneer a dual-view architecture sharing a single KV cache, achieving up to 7.8× strictly lossless inference acceleration with constant \(O(1)\) memory overhead via single-step parallel block projection and consensus verification.
- PatchKV: Weight-Space Compensation of KV Cache
-
In single-context multi-query serving, PatchKV compensates for the severe information loss of aggressively compressed KV caches by deriving closed-form ridge regression patches on MLP output projections using context-derived reference queries, recovering substantial full-cache accuracy with zero per-query serving overhead.
- PithTrain: A Compact and Agent-Native MoE Training System
-
Addressing the steep overhead of AI coding agents navigating bloated production MoE frameworks with heavy implicit indirection, PithTrain introduces four agent-native principles (compactness, Python-native, no implicit indirection, agent skills) to match production throughput while slashing agent turns by up to 62% and active GPU debugging time by 64%.
- Positional LSH: Binary Block Matrix Approximation for Attention with Linear Biases
-
This paper views attention with positional bias through the lens of locality-sensitive hashing (LSH), proving that the ALiBi bias matrix is the exact expectation of binary contiguous block-diagonal masks sampled via random binning features on positions, thereby deriving a near-linear time uniform approximation algorithm for long-context attention.
- Prism Attention: Proposal-Refined Index Sharing Mechanism for Efficient LLMs Inference
-
Prism Attention addresses the severe accuracy degradation in static cross-layer index sharing caused by inter-layer attention drift by introducing a coarse-to-fine proposal-refinement paradigm, which leverages INT2-quantized scoring to build an expanded candidate pool and performs context-length-independent local refinement at subsequent layers, achieving a 3.4x end-to-end decoding speedup at 128K context while matching full attention accuracy.
- Proteus: Incremental Memory Activation for Long-Context Sequence Modeling
-
Proteus addresses the fundamental flaw of static recurrent memory—where early tokens overfit without compression pressure while later tokens suffer from severe interference and overwriting—by introducing an incremental memory activation paradigm that progressively unlocks capacity over context progression at zero parameter or computational cost.
- Qrita: High-performance Top-k and Top-p using Pivot-based Truncation and Selection
-
Qrita replaces costly full-vocabulary sorting and lossy rejection sampling in LLM decoding with Gaussian \(\sigma\)-truncation and quaternary pivot search with prefix-sum duplication handling, cutting memory usage by half while boosting end-to-end serving throughput by up to 1.4\(\times\) with exact deterministic outputs.
- Reformulating KV Cache Eviction Problem for Long-Context LLM Inference
-
Addressing the memory bottleneck of long-context LLM inference and the localized shortcomings of heuristic eviction methods, this paper reformulates KV cache eviction into a layer-wise matrix multiplication approximation problem, introducing the output-aware LaProx metric based on attention and projected value norms alongside layer-wise normalization for model-wide global selection without offline calibration.
- RheoSampling: Resolving the One-Hot Dilemma in Stochastic Dynamic-Tree Speculative Decoding
-
RheoSampling resolves the one-hot probability collapse in stochastic dynamic-tree speculative decoding by decoupling a sampled token's role into a proxy probability for tree expansion/pruning and its true sampling probability for verification, achieving strictly lossless acceleration with optimal transport verification.
- Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts
-
This paper reveals an inherent "geometric coupling" between routers and their corresponding experts in Sparse Mixture-of-Experts (SMoE) driven by backpropagation chain rules, demonstrates how conventional auxiliary load-balancing losses collapse router geometry via global interference gradients, and validates the centrality of this coupling via a parameter-free online K-Means router.
- RubiConv - Efficient Boundary-Respecting Convolutions
-
Addressing boundary bleed-over and periodic wrap-around artifacts caused by document packing in large-scale long-convolution training, RubiConv reformulates packed convolutions via block-diagonal DFTs and adaptive twiddle modulation into a single parallel GEMM pipeline, matching Splash Attention throughput while strictly isolating cross-document interactions.
- SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking
-
Addressing the gradient blockage and ranking misalignment caused by discrete Top-K selection and layer-wise attention distillation in post-training attention sparsification, SAS injects normalized log-space selector scores directly into attention Softmax, training the context selector end-to-end via standard language modeling loss to achieve superior performance under tight attention budgets.
- SEED: Self-Speculative Decoding via Implicit Encoder–Decoder
-
SEED trains the last two layers of an existing decoder-only model as a drafter that reuses deep KV caches from the previous full-model verification, achieving average decoding speedups of 2.6×/2.7× over AR on Qwen3-1.7B/4B while maintaining or improving task quality under task-specific fine-tuning and adaptive tree verification.
- Short-Context dominance: How Much Local Context Natural Language Actually Needs?
-
This paper formulates and confirms the "short-context dominance hypothesis," showing that 75%–80% of natural language tokens need only the final 32–96 tokens for accurate prediction, and designs an unlabelled long-context detector (LSDS) alongside a targeted boosting decoding algorithm (TaBoo) that consistently outperforms vanilla nucleus sampling and CAD.
- SimSD: Simple Speculative Decoding in Diffusion Language Models
-
SimSD proposes a plug-and-play, training-free speculative decoding framework for diffusion language models by pairing reference data with mask prediction slots, enforcing a token-level temporal causal mask, and aligning RoPE indices, enabling a target model to verify multiple drafted tokens in a single forward pass with up to 7.46× throughput speedup.
- SMoA: Spectrum Modulation Adapter for Parameter-Efficient Fine-Tuning
-
Addressing the fundamental loss of informative singular-value tails in standard LoRA, SMoA aligns and partitions pretrained weights into spectral blocks modulated via in-block Hadamard low-rank branches, achieving higher effective rank capacity and superior performance with only \(1/K\) of LoRA's trainable parameters.
- Sparse Attention as Compact Kernel Regression
-
This paper establishes a formal theoretical connection between sparse attention and compact-support kernel regression via the Nadaraya-Watson estimator, proving that normalized ReLU, sparsemax, and \(\alpha\)-entmax correspond to Epanechnikov and rectified polynomial kernels, while introducing \(r\)-ReLUmax to eliminate zero-denominator degeneracies.
- SpecBlock: Block-Iterative Speculative Decoding with Dynamic Tree Drafting
-
SpecBlock introduces a block-iterative speculative decoding framework that generates \(K\) dependent positions per forward via layer-wise hidden-state shift and dynamically branches candidate trees with a co-trained rank head, cutting drafting cost by ~50% while outperforming EAGLE-3 in wall-clock speedup.
- STILL: Selecting Tokens for Intra-Layer Hybrid Attention to Linearize LLMs
-
Addressing token routing position-bias and feature magnitude distortion in pretrained LLM linearization, STILL introduces a causal Self-Saliency Score to route key historical tokens to high-fidelity Softmax Attention while summarizing background context via a Norm-Preserved Feature Map (NP-Map), achieving subquadratic complexity with lossless long-context memory.
- Tackling the Data-Parallel Load Balancing Bottleneck in LLM Serving: Practical Online Routing at Scale
-
Addressing the step-wise synchronization barriers imposed by tensor/expert parallel collective communication across replicated LLM instances, this paper proposes BalanceRoute—a family of practical online routing algorithms that exploits a piecewise-linear F-score to capture the sharp margin-overflow asymmetry and leverages a short-horizon termination classifier to dramatically reduce data-parallel imbalance with sub-millisecond dispatch latency.
- TALON: Confidence-Aware Speculative Decoding with Adaptive Token Trees
-
Addressing the structural rigidity of static-width and static-depth draft trees, TALON introduces a training-free, budget-driven tree expansion framework that combines root Top-K initialization with deep confidence gating to dynamically morph the draft tree into deep-and-narrow chains for deterministic contexts and shallow-and-wide branches for uncertain ones, achieving up to 5.16× end-to-end acceleration.
- Test-Time Speculation
-
Addressing the severe degradation of acceptance length in speculative decoding for long-response tasks where acceptance rates approach 1, TTS leverages target model verification distributions as free online distillation signals to continuously adapt the draft model at test-time, improving mean acceptance length by 41%.
- To Copy or Not to Copy: Controlling Speculative Decoding via Intrinsic Model Signals
-
To eliminate the severe speculation tax caused by accidental n-gram repetitions in context-based drafting, SwitchSD trains an unbiased linear probe on intermediate post-attention representations to detect genuine copy-intent (AUC > 0.99), enabling dynamic, model-aware switching between context copying and neural draft models (e.g., EAGLE3) for up to 15% throughput gains across LLaMA-3 and Qwen-3 families.
- TokenRouter: Efficient Serving System for Token-Level LLM Routing
-
TokenRouter addresses severe step desynchronization and batch admission latency in token-level LLM routing by introducing request-centric programming with model-centric decoupled execution, a KV-cache-preserving handoff-resume mechanism, and a DTMC-modeled delayed-batching scheduler, achieving 2.01–64.15× higher decoding throughput over existing systems.
- TreeGraft: Adaptive Multi-Drafter Grafting for Tree-Based Speculative Decoding
-
TreeGraft replaces the single-drafter paradigm in tree-based speculative decoding with a multi-drafter collaborative framework, leveraging expanded grafting position selection, non-destructive grafting, and a value-distilled online scheduler to boost inference speedup by 15.1% over the best single-drafter baseline across 10 model pairs.
- UniVer: A Unified Perspective for Multi-step and Multi-draft Speculative Decoding
-
UniVer formulates tree-structured speculative verification as a conditional optimal transport (OT) problem, using prefix acceptance probabilities as dynamic scaling factors to unify horizontal multi-draft selection and vertical multi-step coordination with provable losslessness and up to 8.5% acceptance length gains.
- ValuSpec: Plug-and-Play Candidate Valuation before Target Verification for Tree-Based Speculative Decoding
-
ValuSpec introduces a plug-and-play candidate valuation and pruning mechanism prior to target verification, estimating confidence transfer thresholds from target greedy trajectories on an offline calibration split to discard low-value candidate branches before expensive verification, significantly cutting wasted compute while guaranteeing exact mathematical equivalence to greedy autoregressive decoding.
- Verification-Aware Training for Speculative Decoding
-
VAT simulates target model sequential verification during draft training, introduces a lightweight verification head to supervise survival status, and re-anchors the positional loss decay to the first rejection point, providing plug-and-play acceptance length and wall-clock speedup gains across EAGLE-3 and DFlash.