Skip to content

📦 Model Compression

💬 EMNLP2026 · 85 paper notes

📌 Same area in other venues: 🧠 NeurIPS2026 (205) · 🎞️ ECCV2026 (94) · 📷 CVPR2026 (111) · 🔬 ICLR2026 (243) · 💬 ACL2026 (59) · 🧪 ICML2026 (117)

🔥 Top topics: LLM ×18 · Model Compression ×16 · Compression ×14 · Reasoning ×4 · Alignment/RLHF ×3

A^3: Attention-Aware Accurate KV Cache Fusion for Fast Large Language Model Serving

To tackle accuracy degradation in cross-request KV cache reuse caused by position mismatch and missing cross-attention, \(A^3\) leverages shallow query-aware attention to guide selective token recomputation and pairs with adaptive token eviction, achieving a 2× reduction in time-to-first-token (TTFT) with near-lossless accuracy.

A Target-Centric Survey of Quantization-Aware Training

Addressing the limitation of prior quantization surveys that overlook the distinct numerical properties and lifecycles of different tensor targets, this paper presents the first target-centric survey of Quantization-Aware Training (QAT), deconstructing weight, activation, joint W+A, KV cache, and gradient quantization to formulate cross-target transferability rules, empirical evaluation protocols, and a rigorous minimal reporting checklist for real hardware deployment.

Accelerating Neural Architecture Search with Code Semantics

This paper introduces ECO-SCE, the first code-semantic transition estimator for neural architecture search that models parent-to-proposal evolutionary utility via semantic shifts, structural edits, and local evidence calibration, filtering low-gain code rewrites prior to costly evaluation and cutting downstream NAS GPU hours by 70.88% zero-shot.

AdaCode: Adaptive Lossless Compression for BFloat16 KV Cache

Addressing the severe memory bottleneck of KV caches in long-context generation without suffering from the approximation errors of lossy quantization, this paper reveals that BFloat16 exponent values are inherently concentrated and proposes AdaCode, a bit-exact lossless compression framework that encodes frequent exponents into short fixed-length codes while retaining outliers explicitly, achieving an average 71.78% compression ratio across diverse LLMs and 70.26% on vision-language models with zero accuracy loss.

Adaptive Layer Allocation for Efficient Transformer Fine-Tuning

Grounded in Moral Foundations Theory (MFT), this work uncovers a cross-lingual shared yet culturally nuanced moral subspace in LLMs, extracts causally steerable Moral Vectors, and introduces Adaptive Moral Fusion (AMF) to dramatically slash HarmBench jailbreak rates while minimizing XSTest over-refusals without degrading core utility.

Rethinking Stepwise Model Routing: A Cost-Efficient Table Reasoning Perspective

EcoTab presents a table-aware stepwise model routing framework that decouples table structure tokens from natural language text tokens via a word-level Trie, independently maps their uncertainties to next-step failure risks using offline counterfactual suffix replacement, and fuses the risks via Noisy-OR logic to guide small/large model selection without generating extra verification tokens.

HLoRA: Hierarchical Regularization on Combined Parameter Space for LoRA Fine-Tuning

Addressing the failure of conventional continual learning regularizers under LoRA's factorization ambiguity, HLoRA formulates path-integral importance directly over the combined weight space \(\Delta W = BA\) alongside element-wise and layer-wise hierarchical regularization, preventing catastrophic forgetting while speeding up importance calculation by ~20x and reducing storage by 85%–90%.

CulturePRM: A Process Reward Model for Mitigating Cultural Overriding in Cultural Reasoning

To tackle cultural overriding—a prevalent failure mode where reasoning models acknowledge local cultural norms but subsequently erode or substitute them with universal values or Western defaults—this paper introduces CulturePRM, a lightweight 1B process reward model that scores intermediate reasoning steps to guide Best-of-N reranking, yielding substantial accuracy gains and narrowing cross-cultural disparities.

Small Models, Fewer Heads, More Forgetting: Sparse Attention Fine-Tuning for SLM Continual Learning

Addressing the structural bottleneck where small language models (SLMs) suffer severe catastrophic forgetting due to their scarce attention heads causing premature subspace overlap, LightLoRA introduces a parameter-free sparse fine-tuning framework that scores attention heads via SVD parameter variation and directional gradient conflict penalties, selectively updating high-contribution orthogonal heads while preserving historical knowledge.

Popular but Wrong: Understanding and Mitigating LLM Overconfidence through Knowledge Popularity

This paper investigates the root cause of LLM overconfidence in factual question answering through the lens of knowledge popularity, revealing that hallucinated answers systematically drift toward globally popular or frequently co-occurring entities, and introduces a lightweight popularity-aware MLP calibrator that slashes average confidence on incorrect answers from 0.765 to 0.254 while reducing Expected Calibration Error (ECE) from 0.356 to 0.050.

SSR: Symbolic-Neural Soft Reasoning for Robust and Verifiable Thinking Chains

To tackle auto-formalization noise and the brittleness of strict determinism in neuro-symbolic inference, SSR integrates confidence-weighted MaxSAT relaxation, dual-system targeted retrieval under ambiguity, and verifiable forward/backward reasoning tree generation, delivering both high accuracy and fully verifiable thinking chains.

Advantageous Parameter EXpansion Training Makes Better Large Language Models

Addressing the severe structural redundancy where high-activation advantageous parameters disproportionately govern LLM capabilities while underutilized parameters remain dormant, this paper proposes Advantageous Parameter EXpansion Training (APEX), which progressively expands advantageous parameter representations into disadvantageous ones across multi-stage training, outperforming full-parameter fine-tuning with only 52% trainable parameters.

ADWIN: Adaptive Windows for Horizon-Aware On-Policy Distillation

ADWIN recasts rollout length in on-policy distillation (OPD) as an online prefix-admissibility problem, performing synchronous updates on short teacher-anchored prefixes while calibrating the horizon via asynchronous full-rollout gradient probes, reducing end-to-end training FLOPs by up to 4.1× with equal or superior reasoning accuracy.

All for 1-Bit: Towards Genuine 1-Bit Post-Training Quantization for LLMs

AF1 (All for 1-Bit) establishes the first genuine 1-bit post-training quantization (PTQ) framework for LLMs operating under a strict 1.0 effective-BPW storage budget without auxiliary metadata, pairing Null-space-Aware Binary Factorization (NABF) and Hierarchical Shapley Allocation (HiSA) to surpass existing binary PTQ baselines while delivering an average 2.5× inference speedup and over 90% memory reduction.

ANCHORS: Adaptive Normalization for Continual High-Order Retention via Sigmoid-Guided Magnitude Control

Drawing inspiration from homeostatic synaptic plasticity in biological neural networks, ANCHORS integrates CABR inverse low-rank singular subspace decomposition with a closed-loop Sigmoid-guided dynamic magnitude normalization valve (S-MagNorm), significantly boosting downstream adaptation accuracy while fundamentally solving the catastrophic forgetting dilemma in LLM PEFT without requiring memory replay.

AWARe: Mitigating Catastrophic Forgetting via Activation-Weighted Adaptive REtention

Addressing catastrophic forgetting in Multimodal Large Language Models (MLLMs) during downstream adaptation, AWARe identifies critical neurons via activation saliency on a small calibration set and globally freezes the top 30% most salient weights in self-attention and projection layers, updating only ~17.5% of parameters while achieving superior stability and plasticity.

bayesian sparse low rank adaptation for large language model uncertainty estimat

DALorRA shifts variational Bayesian inference from dense parameter spaces to the lightweight rank dimension of LoRA adapters using stochastic Bernoulli masks, dramatically mitigating LLM overconfidence while training faster than standard LoRA with negligible parameter overhead.

Beyond: Better-than-Full-Precision KV Caches via Learnable Non-Uniform Quantization as Implicit Regularization

Beyond overturns the traditional local distortion minimization paradigm by optimizing non-uniform representation levels and decision thresholds against downstream next-token negative log-likelihood (NLL) with the backbone frozen, using a half-wave rectified STE pseudo-gradient to turn 4-bit discretization into a benign implicit regularizer that surpasses full-precision baselines while delivering 3.2× memory compression.

Beyond Outliers: A Data-Free Layer-wise Mixed-Precision Quantization Approach Driven by Numerical and Structural Dual-Sensitivity

Existing data-free layer-wise mixed-precision quantization (LMPQ) methods uniformly treat all intra-layer weight modules and rely solely on single numerical outlier metrics while overlooking structural expressiveness; this paper proposes NSDS, which mechanistically decomposes layers into Detectors and Writers to quantify numerical vulnerability and role-aware structural expressiveness, achieving superior low-bit quantization performance without any calibration data.

Beyond Retraining-Free MoE Compression: A Cost-Normalized Study of Post-Compression Adjustment

This paper revisits the prevailing paradigm of retraining-free MoE compression by arguing that compressed checkpoints are compressed initializations rather than final deployable artifacts; across 28 benchmarks and measured GPU budgets, a single-epoch full-parameter causal LM fine-tuning with only 3,000 examples recovers 37.3% of the original-to-compressed degradation gap.

Beyond Task Boundaries: Instruction-Level Parameter Isolation for Supervised Fine-Tuning

Addressing the unit mismatch between coarse task-level isolation and instance-level optimization in heterogeneous supervised fine-tuning (SFT), this paper proposes Instruction-Level Parameter Isolation (IPI). By constructing multi-view capability representations and a lightweight low-rank router, IPI predicts instance-conditioned sparse update masks, while an identity-trick reparameterization preserves dense pretrained forward computation while routing sparse gradients to the base model and task-loss saliency signals to the router, consistently mitigating destructive gradient interactions and outperforming existing isolation baselines across diverse LLMs.

Beyond Tokens: Semantic-Aware Speculative Decoding for Efficient Inference by Probing Internal States

Addressing the frequent false rejections caused by strict token-level lexical matching in traditional speculative decoding, SemanticSpec probes the internal hidden states of LLMs to estimate semantic probabilities, enabling sequence-level semantic verification that achieves up to a 2.7x speedup on DeepSeekR1-32B while preserving high reasoning accuracy.

Beyond Top-k : Mass-Aware Support Sampling for On-Policy Distillation

To address the tail-token neglect and support-mass discrepancy caused by deterministic top-\(k\) truncation in on-policy reverse-KL distillation, this paper proposes Stochastic Support Sampling (SSS) combined with Teacher Mass Reweighting (TMR), achieving +1.4 hard-math average gains on Qwen3 and +2.8 code average gains on Llama-3.1 under identical support budgets.

Calibrated Tree-Structured Speculative Decoding: Mitigating Overconfidence via Path-Level Calibration

To alleviate severe draft model overconfidence in tree-structured speculative decoding—which squeezes valid paths out of verification budgets and triggers wasteful shallow subtree expansions—CATS-SD introduces training-time dynamic tree construction with depth-decayed path-level calibration, boosting acceptance length by up to 11.3% and end-to-end speedup from 2.51× to 2.71× on GSM8K.

Calibrated Tree-Structured Speculative Decoding: Mitigating Overconfidence via Path-Level Calibration

To resolve the critical bottleneck of draft model overconfidence triggering wasteful subtree expansion and incorrect branch pruning in tree-structured speculative decoding, CATS-SD introduces a depth-weighted path-level tree calibration loss paired with BiLD loss during training, and proposes the lookahead APC metric, achieving up to 11.3% higher acceptance length and a 2.71× end-to-end speedup.

CentralKV: Compressing Long-Context LLMs via Attention-Graph Centrality

CentralKV is a training-free KV cache compression framework that casts attention into a dynamic sparse causal graph and deploys an online Stream-Influence centrality algorithm using a non-linear max operator, accurately retaining low-frequency but structurally pivotal bridge tokens to preserve multi-step reasoning chains under severe memory budgets.

clustering based balanced sampling and allocation with data parallelism for high

Addressing sample redundancy and excessive mini-batch gradient variance in large-batch distributed instruction tuning, CluSTER combines lightweight gradient-proxy clustering, dual-level balanced sampling, and cluster-weighted updates to cut training time by up to 69.6% while preserving original data distribution and model accuracy.

CoDe-LoRA: Mitigating the Orthogonality Dilemma in Continual Learning of LLMs via Knowledge Consolidation and Decoupling

Addressing the "Orthogonality Dilemma" where strict geometric parameter isolation blocks transferable representation reuse across related tasks, CoDe-LoRA decouples continual adaptation into a shared consolidation branch stabilized by adaptive null-space projection with dynamic scaling and a task-specific expert pool equipped with prototype confidence fallback, consistently outperforming prior SOTA under replay-free constraints.

Cognitive Chunking for Soft Prompts: Accelerating Compressor Learning via Curvature Alignment

Inspired by cognitive chunking in human working memory and empirical observations of spatial specialization, Parallelized Iterative Compression (PIC) introduces a block-wise causal attention mask to strictly enforce sequential local chunk compression within a single parallel forward pass, reducing pre-training compute by 67% while significantly outperforming competitive soft prompt compression baselines on RAG and ICL tasks.

Cognitive Chunking for Soft Prompts: Accelerating Compressor Learning via Block-wise Causal Masking

Inspired by human working memory chunking, Parallelized Iterative Compression (PIC) introduces a block-wise causal attention mask that restricts memory tokens to local sequential context blocks, accelerating compressor pre-training convergence by 67% while boosting downstream QA and reasoning accuracy without inference latency penalties.

Complexity-Aware Routing: Detecting and Allocating Dynamic Compute for Efficient Mixture-of-Experts Large Language Models

To resolve the computational waste and under-fitting in conventional static Top-k MoE models, CAR decouples offline dual-signal complexity perception (Shannon Surprisal and Predictive Entropy) from discrete dynamic expert allocation. Operating under a frozen backbone, it reduces active parameters on Qwen3-30B-A3B by approximately 10% while lifting GSM8K accuracy by 1.57 points over static routing.

Compressing Sequences in the Latent Embedding Space: K-Token Merging for Large Language Models

To overcome the severe information loss of hard token pruning and the combinatorial tail explosion of vocabulary expansion, K-Token Merging introduces a lightweight mean-residual MLP encoder that condenses contiguous blocks of \(K\) token embeddings into single latent vectors during prefill, coupled with LoRA adaptation to achieve up to 75% sequence length reduction with near-lossless generation quality.

OptiMer: Optimal Distribution Vector Merging Is Better than Data Mixing for Continual Pre-Training

Addressing the rigid pre-training bottleneck and prohibitive tuning overhead of data mixture ratios in continual pre-training, OptiMer decouples ratio selection from model training by extracting independent distribution vectors per dataset and searching continuous merge weights post-hoc via TPE Bayesian optimization, outperforming joint data mixing and uniform model averaging across 16 benchmarks with 15–35× lower search cost.

CoQuant: Joint Weight-Activation Subspace Projection for Mixed-Precision LLMs

Addressing the flaw in prior subspace-based mixed-precision quantization that relies solely on activation statistics while overlooking weight noise, CoQuant derives a first-order joint error expectation model that yields a closed-form weighted PCA solution balancing activation and weight covariances, substantially reducing layer-wise perturbation and achieving state-of-the-art perplexity and zero-shot reasoning accuracy on Llama-3.2 and Qwen2.5.

Counterfactual Constraint-Conditioned On-Policy Distillation for Multi-Constraint Instruction Following

To alleviate constraint dilution in on-policy distillation where a frozen teacher's probability mass is stretched over joint prompt requirements, CC-OPD forces the teacher to rescore the student's rollout under leave-one-out constraint contexts, extracting dense per-constraint token-level likelihood differences as a clipped shaping reward that allows a 1.5B student to surpass its 7B RL-trained teacher.

CRISP: Cliff-awaRe Input-adaptive Sparse Prefilling with Structural-Mass-Motivated Routing

Addressing the quadratic bottleneck of self-attention prefilling in long-context LLMs, CRISP identifies the overhead in JSD-based dynamic routing and mathematically formalizes the post-softmax mass cliff where cumulative coverage thresholds suffer from \(O(n)\) noise accumulation and sink-only collapse, introducing the \(O(1)\) structural routing proxy \(C_{\text{struct}}\) alongside a sink-aware noise-floor threshold to achieve parity with dense attention on retrieval tasks and up to a 5.30x attention speedup at 512k tokens.

Cross-Decoder Retargeting of Soft Compressor for Long-Context Compression

Addressing the issue where soft text compressors are tightly coupled to their training decoders and cannot be ported across models, this paper systematically explores decoder-interface portability, reveals that compressed vectors are decoder-conditioned continuous inputs rather than universal semantic codes, and demonstrates that trainable retargeting and generator adaptation successfully recover target-native reconstruction fidelity.

DART: Low-Latency Parallel Drafting with Continuity-Aware Tree Pruning for Speculative Decoding

DART eliminates the sequential drafting bottleneck in speculative decoding through a single-forward parallel shifted-logits drafter coupled with N-gram continuity-aware tree pruning, reducing drafting latency by up to 6.8× and delivering 2.03×–3.44× end-to-end lossless speedups over standard autoregressive decoding.

Do Benchmarks Measure the Same Thing? Evaluation-Free Calibration for Compressed Models

This paper introduces an evaluation-free benchmark diagnosis and calibration framework based on Sparse Autoencoder (SAE) feature coverage, treating internal monosemantic feature overlap as a proxy for behavioral equivalence to quantify sample redundancy and cross-benchmark overlap without running costly model evaluations, achieving representative near-zero variance subset compression and suite curation.

Draft-OPD: On-Policy Distillation for Speculative Draft Models

Addressing the rapid performance plateau of speculative draft models trained via offline supervised fine-tuning (SFT) on static target trajectories, Draft-OPD introduces on-policy distillation using target-assisted rollouts with error-position replay and an acceptance-aware bidirectional KL objective, delivering up to 5.60× lossless end-to-end inference speedup on Qwen3 models.

E^3-Pruner: Towards Efficient, Economical, and Effective Layer Pruning for Large Language Models

Addressing the trilemma of severe performance degradation, prohibitive search costs, and limited practical acceleration in structured pruning, this paper presents E^3-Pruner—a two-stage framework that couples differentiable Gumbel-TopK search with progressive scheduling and token-entropy-weighted adaptive distillation, achieving state-of-the-art capability recovery under a tiny 0.5B token budget.

EarlySelectKV: Anticipatory KV Selection via Cross-Layer Query Proxies

Addressing the critical pre-attention serialization bottleneck in query-conditioned dynamic KV selection where selectors must wait for the target-layer query, EarlySelectKV introduces a training-free transformation that uses proxy queries derived from preceding hidden states to precompute selected KV indices asynchronously, matching native accuracy across LongBench and RULER while improving end-to-end throughput by up to 12.4% and cutting GPU routing memory by 11.1%.

ECHO: Early-layer Collaborative Hierarchical Orchestration with Bonus Logits in Speculative Decoding

ECHO mitigates draft quality decay and the verification wall in draft-model-free speculative decoding through a dual-loop hierarchical architecture, bifurcating inference into agile early-layer tree exploration driven by early bonus logits and authoritative subsequent-layer verification via state reuse, achieving a 2.4× to 3.01× lossless speedup without auxiliary deployment parameters.

Enabling KV Caching of Shared Prefix for Diffusion Language Models

To overcome the severe accuracy collapse to near-zero caused by naively porting autoregressive shared-prefix caching to diffusion language models (DLMs), BICACHE introduces the first bidirectional prefix caching framework, which dynamically determines safe shallow layers based on the shared prefix ratio and applies periodic refresh in deep layers, boosting throughput by 36.3%–98.3% while keeping accuracy drops within 0%–1.8%.

Every Time I Hire a Linguist, Inference Costs Go Down: On Linguistic Rules as Effective Prompt Compressors

LingComp leverages offline LLM-guided evolutionary search over multi-level linguistic seeds to discover deterministic rule-based prompt compressors that execute entirely on CPU via spaCy, matching neural compression performance without any deployment LM forward passes.

Frequency Matters: Fast Model-Agnostic Data Curation for Pruning and Quantization

Addressing the heavy computational burden of forward-pass model-aware calibration curation for LLM pruning and quantization, this paper proposes ZipCal, a model-agnostic greedy lexical diversity curation strategy grounded in Zipfian power laws that matches or surpasses SOTA methods (e.g., COLA) while providing 150×–400× speedups.

Garbage Attention in Large Language Models: Sink Heads and Sink-Aware Pruning

This paper establishes the <BOS> attention sink phenomenon as a mechanistic driver of functional redundancy in LLMs, reveals that deep sink heads act as internal dumping grounds for surplus attention probability, and introduces a zero-recalibration structured pruning strategy that reliably removes redundant heads and layers while maintaining downstream task performance.

Gradient Starvation in Low-Rank Language Model Pretraining

This paper reveals that the representational and performance deficit in low-rank language model pretraining is not inherited from initialisation, but is sustained by an escalating deficit in gradient effective rank (gradient starvation), while demonstrating that optimisation gains (perplexity) dissociate from representational alignment.

HEAL: Latent Head Alignment for Post-Training Quantization of Large Language Models

Addressing the issue where conventional block-level reconstruction in LLM PTQ masks individual head distortions through cross-head error cancellation, this paper proposes HEAL, a variational latent head alignment framework with stop-gradient targets and stochastic token-window calibration that significantly lowers perplexity and improves zero-shot accuracy under low-bit weight-activation quantization.

HLoRA: Hierarchical Regularization on Combined Parameter Space for LoRA Fine-Tuning

Addressing the unreliability of per-parameter importance estimation under low-rank factorization, HLoRA formulates path-integral importance directly on the combined weight update space \(\Delta W = BA\) and imposes hierarchical element-wise and layer-wise regularization, mitigating catastrophic forgetting while delivering a 20x speedup and saving 85%–90% storage over previous methods.

InfoQuant: Shaping Activation Distributions for Low-Bit Weight-Activation Quantization in LLMs

InfoQuant recasts activation transformation as quantizer-facing distribution design from an information-theoretic perspective, combining train-free Peak Suppression Orthogonal Transformation (PSOT), adaptive outlier-token selection (ASOT), and learnable activation clipping (LAC) to preserve 97% of floating-point accuracy under W4A4KV4.

Is Knowledge Distillation Actually Greener? A Case Study in Machine Translation

By incorporating the Machine Learning Life Cycle Assessment (MLCA) framework into neural machine translation distillation, this paper decomposes carbon footprints across teacher training, distillation overhead, and student inference—accounting for both operational and amortized embodied hardware emissions—revealing that knowledge distillation is not inherently green, as its break-even deployment volume is highly serving-dependent and shifts by multiple orders of magnitude under batched inference.

Jasper-Flash: Elastic Token Compression and Progressive Distillation for Inference-Scalable Text Embedding Models

To alleviate the sequence-length-dependent quadratic computational bottleneck in text embedding models, Jasper-Flash introduces Elastic Token Compression (ETC) via an MLP and 1D adaptive average pooling, paired with Compression-Adaptive Progressive Distillation (CAPD), cutting 1,024-token encoding latency from 24.24 ms to 4.48 ms on an RTX 4090 while retaining a competitive 74.22 MTEB score.

KV Admission: Learning What to Write for Efficient LLM KV Cache Compression

Introducing Write-Gated KV (WG-KV), an end-to-end learnable KV admission mechanism that predicts token utility prior to cache write and couples sliding local windows with paged global cache, achieving near-lossless long-context inference at extreme sparsity and delivering multi-fold wall-clock speedups.

last but not least boundary attention calibration for multimodal kv cache compre

Addressing the critical flaw where observation window attention in multimodal KV cache compression dilutes sparse, answer-critical visual evidence, this paper proposes BACON, a plug-and-play boundary attention calibration method that harnesses the prompt's last query while filtering noise through intra-layer coherence and inter-layer persistence to substantially recover critical tokens under tight budgets.

LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Eviction

LKV reframes KV cache eviction as an end-to-end differentiable optimization problem, unifying global head-wise budget learning (LKV-H) with matrix-free intrinsic token selection (LKV-T) to achieve near-lossless long-context fidelity with only 15% KV cache retention and up to 3.19× decoding throughput speedup.

LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Eviction

LKV reformulates LLM KV cache eviction as an end-to-end differentiable optimization problem, leveraging a closed-form Soft-TopK operator to jointly learn global head-wise budgets (LKV-H) and matrix-free token utility (LKV-T), achieving near-lossless long-context generation with only 15% KV cache retention and up to 3.19× decoding speedup.

LongFlow: Efficient KV Cache Compression for Reasoning Models

Addressing severe KV cache memory inflation and bandwidth bottlenecks in long Chain-of-Thought reasoning models, LongFlow proposes a zero-history importance metric derived directly from intermediate attention contribution vectors using only the current query, and co-designs a fused Triton kernel integrating FlashAttention, token scoring, and in-place eviction to achieve an 80% KV cache reduction and up to an 11.8x throughput speedup while preserving reasoning accuracy.

MAPLE: Masked Adapter Prototype Learning for OOD generalization

Addressing the issue where existing LoRA prototypes learned in isolation encode dominant noisy subspaces that yield indistinguishable in-distribution and out-of-distribution routing entropies, MAPLE introduces a simple training paradigm that masks the target-task adapter during prototype refinement, effectively purging spurious shared noise and achieving consistent zero-shot OOD accuracy improvements across diverse benchmarks on Phi-2 and Llama 3.

Mitigating Model Merging Conflicts via Knowledge-Aware Task Vector Purification

PAVE projects fine-tuned model parameters into a knowledge-aware subspace via context-oriented SVD guided by activation covariance matrices, pruning task-irrelevant redundant components and adaptively allocating preserved ranks across models to resolve multi-task parameter conflicts in a plug-and-play manner.

MOAQ: A Multi-Scale Outlier-Aware Framework for Extreme Quantization of Large Language Models

MOAQ is a finetuning-free weight-only post-training quantization framework that resolves the 2-bit/3-bit degradation bottleneck by reorganizing disordered outliers with clustering-optimized Hadamard transforms, searching structured fractional bits, and dynamically compensating for cross-layer error enrichment.

MoARa: Module-Aware Rank Allocation and Structure-Preserving Decomposition for Low-Rank LLM Pre-training

By identifying the structural sensitivity disparity across Transformer linear modules and mitigating the joint attenuation of gradient magnitude and direction, MoARa introduces static profiling-based rank reallocation alongside head-dimension-aligned block-wise decomposition, achieving 37% fewer optimization steps and 34% wall-clock pre-training speedup on Llama 2 7B without expanding the total rank budget.

NestedKV: Nested Memory Routing for Long-Context KV Cache Compression

NestedKV reframes training-free KV cache compression as bounded continuum-memory maintenance, capturing token anomalies across stable (global), episodic (block), and current (sliding-window) key anchors, and dynamically routing scores via head-adaptive priors and compression-induced surprise gating to preserve long-context retrieval at extreme eviction ratios.

nosa native and offloadable sparse attention

Addressing the severe training-inference mismatch of training-free KV offloading and the communication-bound PCIe bottleneck of unconstrained trainable sparse attention, NOSA introduces a natively offloadable sparse attention mechanism with a theoretical locality lower bound alongside a co-designed inference system NOSI, achieving up to 5.04× decoding throughput speedup while maintaining lossless long-generation quality.

OCGQuant: Outlier-Companion Grouping for NVFP4 Quantization

Addressing the challenge where activation outliers dominate block scales in native NVFP4 microscaling formats, OCGQuant introduces Outlier-Companion Grouping to adaptively pair outlier channels with low-magnitude companion channels, eliminating collateral quantization error while sustaining RTN-level prefill speedups via fused kernel optimization.

One Size Does Not Fit All: Token-Wise Adaptive Compression for KV Cache

DynaKV presents the first post-training low-rank KV cache compression framework that dynamically allocates retention dimensions to individual tokens based on their semantic importance, maintaining robust generation quality even under aggressive compression budgets.

outlets output length prediction from speculative decoding backbones

OUTLETS reveals an intrinsic structural connection between speculative lookahead representations and sequence termination forecasting, attaching a lightweight log-space regression head to a shared speculative decoding backbone to deliver accurate length predictions that slash short-request P99 latency by 34.8% in disaggregated serving.

Prefilling-dLLM: Predictive Prefilling for Long-Context Inference in Diffusion Language Models

Addressing the quadratic recomputation bottleneck in long-context diffusion language models where the entire prefix is re-encoded at every denoising iteration, Prefilling-dLLM introduces a training-free prefill-decode disaggregation framework that executes one-time chunked prefilling, selects Top-K salient chunks via self-information and pseudo-label scoring, applies intra-chunk token sparsity, and eliminates the lost-in-the-middle phenomenon with chunk-level BOS attention anchors while yielding 9.1–28.0× end-to-end speedups across 8K–32K contexts.

PrimeTTT: Priming Test-Time Training for LLMs via Meta-Learned Initialization

PrimeTTT addresses the marginal gains and misalignment failures of vanilla test-time training on LLMs by employing a compact hypernetwork trained with second-order MAML to prime prompt-specific LoRA initializations, delivering substantial long-context gains with only ~2% inference overhead.

PRISM: Private Rank-Invariant Subspace Memory for Federated Continual Fine-Tuning under Local Differential Privacy

PRISM introduces a fixed-rank continual adaptation framework that divides LoRA capacity into persistent memory and active plastic subspaces with noise-floor shrinkage gating, stabilizing strict LDP adaptation and cutting uplink communication by 42.5%.

PRKV: Page Restructed KV Cache for High Accuracy and Efficiency LLM Generation

To tackle inaccurate coarse-grained token selection and high data-transfer communication latency in offloaded KV cache retrieval for long-context LLM serving, PRKV introduces a co-designed page restructuring framework that couples an adaptive hybrid static-dynamic retrieval algorithm with HND contiguous memory indexing and pinned-buffer batched transfers, achieving up to 6.75x decoding speedup while preserving full-cache accuracy.

Reservoir of Importance: Learning Semi-Structured Sparsity with Differentiable Masking

To eliminate the combinatorial parameter and memory overhead of modeling full categorical distributions in learnable N:M pruning, RoI formulates mask learning as differentiable subset sampling via Weighted Reservoir Sampling and Gumbel-Top-K relaxation, reducing trainable parameters from \(O(\binom{M}{N})\) to \(O(M)\) while achieving superior perplexity and downstream accuracy across Qwen2.5 and Gemma-2.

Routing-Free Mixture of Experts

Routing-Free MoE completely eliminates centralized routers, Softmax, and TopK selection by allowing each expert to autonomously determine its activation via internal low-rank projections and learnable thresholds, combined with a unified dual-axis soft load-balancing framework that unlocks emergent layer-heterogeneous sparsity and barrier-free expert parallelism.

Rubric-Aware Multi-Expert Online Policy Distillation for Chinese Elementary School Essay Writing

Addressing the challenges of rubric compliance, genre routing, and child-appropriate register in primary school essay writing, RA-MOPD introduces a four-stage pipeline that distills genre-specific GRPO specialists into a single 4B model using reward-model perturbation-based Rubric-Weighted KL (RW-KL) and parameter-free Cross-Genre expert Mixing (CGM), reaching a 4.21/5.00 rubric score and 97.8% routing accuracy.

RW-TTT: Batched Serving for Request-Owned Test-Time Training State

Addressing the throughput collapse in test-time training (TTT) caused by mutable model state breaking static-weight batching, RW-TTT formalizes a read-write serving contract that tags decode steps with owner, version, and READ/WRITE effects, groups only compatible phases, and commits updates exclusively to owning requests, achieving a 9.31x throughput speedup on a single GPU while preserving long-context task accuracy.

SARATHI: Decoupled Data-Free Probing and Adaptive Reconstruction for Structured LLM Pruning

Addressing calibration domain bias and memory wall collapses caused by coupling neuron selection and weight reconstruction in post-training structured pruning, SARATHI completely decouples data-free NMF residual subspace probing from localized \(O(d^2)\) peak-memory Adaptive OBS reconstruction, while introducing closed-form bias shift compensation to eliminate perplexity collapse in post-LayerNorm architectures.

Scaling Laws for Mixed Quantization

Presents the first unified scaling law for post-training mixed-precision quantization in large language models, modeling perplexity degradation as a separable multiplicative function of model parameter count, low-precision parameter ratio, and block size, establishing that larger models tolerate higher low-precision ratios while the absolute loss benefit of fine-grained block scaling diminishes with scale.

SUGAR: Spectral and Geometry-Aware Alignment for Matryoshka Representation Distillation

SUGAR formalizes the task of Matryoshka Representation Distillation and proposes a multi-objective framework combining spectral effective-rank matching, Grassmannian chordal-distance subspace alignment, and orthogonal band regularization, enabling compact student encoders to retain high accuracy across aggressive dimensional truncations.

TableCache: Primary Foreign Key Guided KV Cache Precomputation for Low-Latency Text-to-SQL

Addressing the severe prefill latency induced by extensive database schemas in Text-to-SQL and the failure of traditional prefix caching under dynamic table orders, TableCache precomputes table-level KV caches offline guided by primary-foreign key DAG topological sorting, indexes them in a Table Trie, and couples bitset-based query reranking with an asynchronous compute-loading pipeline, delivering up to a 3.62x TTFT speedup with negligible accuracy loss (within 1.0%-1.6%).

Towards a Comprehensive Scaling Law of Mixture-of-Experts: A Unified Frontier Perspective

Addressing the multi-factor coupling and non-monotonic performance dynamics in Mixture-of-Experts (MoE) architectures, this paper systematically decouples data size, total parameter count, activated parameter count, number of activated experts, and shared expert ratio across 450 controlled pre-training runs, deriving a unified joint MoE scaling law with closed-form optima indicating scale-invariant expert granularity and progressively sparser optimal activation ratios.

TRACER: An Information Entropy-Grounded Framework for Efficient Large-table Semantic Annotation

TRACER addresses the effectiveness–efficiency trade-off in large relational table semantic annotation by formulating row compression as a conditional entropy minimization objective, utilizing an unsupervised entropy-guided evidence ranker and hybrid selection to retain ontology-discriminative cues while reducing inference latency by up to 27x.

train overcomplete deploy compact scaling recovery capacity for structured llm p

Addressing the capacity-knowledge asymmetry bottleneck in structured LLM pruning where recovery modules lack capacity to reconstruct removed knowledge, OverRep temporarily constructs an overcomplete recovery module with an annealed activation function during training, which is then algebraically merged into compact linear projections for deployment without inference overhead.

VisCache: Visual KV Cache Pruning for Efficient Vision Large Language Models

VisCache is a training-free and plug-and-play coarse-to-fine visual KV cache pruning framework that filters temporal redundancy with a lightweight scout VLM and surgically compresses visual KV states via PruneKV—featuring a parabolic layer-wise budget schedule and asymmetric key pruning with value fusion—achieving up to 2.35× end-to-end speedup while retaining only 19–28% of the visual KV cache without degrading multimodal reasoning accuracy.

W2T: LoRA Weights Already Know What They Can Do

W2T (Weight-to-Token) resolves the continuous \(GL(r)\) reparameterization ambiguity of LoRA adapters via an efficient thin QR-SVD canonicalization and encodes the resulting singular components with a hierarchical Transformer, accurately reading semantics, performance, and cross-task similarity directly from raw checkpoint weights without training data or base model inference.

Who Speaks for the Pruned? Visual Token Pruning as Coverage Optimization

Addressing the fundamental flaw where conventional VLM pruning only considers which tokens are important—causing redundant selection of salient regions and discarding critical small details—CoverPruner reframes visual token pruning as Representational Coverage Maximization (RCM) in the projector output space, leveraging a lightweight layer-0 attention probe and greedy submodular selection to achieve substantial accuracy leads under aggressive compression budgets.