Skip to content

โšก LLM Efficiency

๐Ÿ’ฌ EMNLP2026 ยท 5 paper notes

๐Ÿ“Œ Same area in other venues: ๐Ÿง  NeurIPS2026 (79) ยท ๐Ÿ“ท CVPR2026 (8) ยท ๐Ÿ”ฌ ICLR2026 (171) ยท ๐Ÿ’ฌ ACL2026 (23) ยท ๐Ÿงช ICML2026 (48) ยท ๐Ÿค– AAAI2026 (9)

๐Ÿ”ฅ Top topics: LLM ร—2 ยท Reasoning ร—2

Compressing Sequences in the Latent Embedding Space: K-Token Merging for Large Language Models

This paper proposes K-Token Merging, a latent-space soft prompt compression method that fuses contiguous blocks of \(K\) token embeddings into a single compressed embedding via an average-initialized residual MLP with LoRA fine-tuning, achieving up to 75% prompt length reduction and a 27.8% throughput speedup with negligible accuracy degradation.

Efficient Iterative Retrieval with Heterogeneous Batching

Addressing the complementary nature of compute-bound embedding and memory-bound generation in iterative RAG, ORTHRUS introduces a unified inference loop with heterogeneous batching, chunked embedding with incremental pooling, and workload-aware intra-batch scheduling to eliminate static GPU partitioning bubbles, achieving 1.28ร—โ€“4.52ร— higher throughput and cutting end-to-end p99 latency by up to 55.8%.

SpecSteer: Synergizing Local Context and Global Reasoning for Efficient Personalized Generation

Addressing the core tension between the reasoning capacity deficit of on-device models and the privacy risks of cloud-centralized LLMs in personalized generation, SpecSteer introduces an asymmetric collaborative speculative decoding framework where an on-device private specialist drafts candidate sequences, a cloud generalist conducts non-private ratio-based verification, and a steering recovery mechanism corrects rejected tokens, closing the reasoning gap while delivering a 2.36ร— inference speedup.

Spexis: Speculative Lookahead Scheduling for LLM Inference

Spexis introduces speculative parallelism (SP) as an orthogonal scheduling dimension for multi-GPU LLM inference that overlaps self-speculative drafting with downstream pipeline verification while sharing the same KV cache, augmented with confidence-based selective speculation and lookahead remaining-length prediction to achieve up to 34% serving throughput speedup over optimized TP+PP baselines.

Survival-Guided Length Control for Efficient Reasoning

This paper proposes a plug-in, training-free survival-guided length predictor for masked diffusion language models (DLMs) that recasts sequence termination as a discrete-time survival process over the end-of-sequence token, deriving a closed-form expected length from single-pass EOS probabilities and delivering up to 7ร— decoding speedups across reasoning and code benchmarks with zero accuracy loss.