Skip to content

📦 Model Compression

🧠 NeurIPS2026 · 205 paper notes

📌 Same area in other venues: 🎞️ ECCV2026 (94) · 📷 CVPR2026 (111) · 🔬 ICLR2026 (243) · 💬 ACL2026 (59) · 🧪 ICML2026 (117) · 🤖 AAAI2026 (61)

🔥 Top topics: Model Compression ×48 · Compression ×37 · LLM ×21 · Diffusion Models ×11 · Reasoning ×10

A Composite Activation Function for Learning Stable Binary Representations

This paper introduces the Heavy-Tailed Activation Function (HTAF), constructed by composing sigmoid and hyperbolic tangent functions, which orthogonally decouples a steep origin gradient for accurate Heaviside approximation from slow tail gradient decay to prevent gradient vanishing, thereby enabling surrogate-free gradient-based training of SNNs, BNNs, and Deep Heaviside Networks while unlocking annotation-free Implicit Concept Bottleneck Models without accuracy loss.

A Free Lunch in LLM Compression: Revisiting Retraining after Pruning

Revisiting the perceived impossibility of post-pruning adaptation in LLMs, this work demonstrates that local reconstruction matching intermediate dense activations matches full PEFT retraining with an order of magnitude less data and compute, revealing a granularity-invariant "free-lunch" regime that makes simple pruning criteria competitive at scale.

A Self-Evolving Framework for Efficient Terminal Agents via Observational Context Compression

TACO introduces the first plug-and-play, self-evolving observational context compression framework for CLI agents, dynamically refining structured filtering rules via intra-task implicit feedback and cross-task global pooling to cut token consumption by 12%–27% while boosting task success across six diverse benchmarks without human rules or model tuning.

AC/DC on a Budget – Alternating Sparse Phases

This paper reveals that dense phases in AC/DC sparse training drive critical topology rewiring and parameter sign corrections, and proposes zero initialization, shortened dense phases, and the Alternating Rewiring Compression (ARC) algorithm—which updates only active parameters alongside a small high-gradient fraction of out-of-mask weights—matching or exceeding full-dense AC/DC performance across CIFAR100, ImageNet, and GPT-2 at substantially lower theoretical FLOPs.

Active Learning for Conditional Generative Compressed Sensing

This paper introduces an active learning framework for conditional generative compressed sensing with subsampled Fourier measurements, decoupling prompts into sampling, recovery, and ground-truth roles and proving stable recovery guarantees and explicit sample complexity penalties governed by a prompt compatibility factor.

AdaPreLoRA: Adafactor-Preconditioned Low-Rank Adaptation

Addressing the fundamental obstacle that the rank deficiency of LoRA's generator Jacobian prevents pulling full-weight preconditioners back to factor space, AdaPreLoRA unifies existing optimizers under a constrained affine inversion framework and derives a closed-form factor update minimizing an Adafactor-weighted factor imbalance, achieving the optimal low-rank preconditioned step at \(\mathcal{O}((m+n)r)\) memory.

Adaptive Mass-Segmented KV Compression for Long-Form Reasoning

AMS leaves token-importance scorers unchanged and instead uses attention-derived quality mass to form adaptive segments and allocate retention quotas before local selection, mitigating contiguous context loss in long-form reasoning; on Math500 with a 256-token cache budget, AMS-Expected improves over AdaKV-ExpE2 by 16.0 percentage points.

AdvJudge-Zero: Binary Decision Flips in LLM-as-a-Judge via Adversarial Control Tokens

AdvJudge-Zero leverages a zero-seed beam search over an LLM judge's own conditional next-token distribution to uncover low-perplexity adversarial control tokens that flip binary rejections to approvals, and introduces a 9-class mechanism-stratified LoRA hardening recipe that completely prevents reward hacking collapses under GRPO.

Alignment Collapse Under KV Cache Quantization: Diagnosis and Mitigation

Reveals that low-bit KV cache quantization in LLMs silently destroys safety alignment while leaving perplexity virtually unchanged, identifies activation channel geometry as the root cause, and introduces the Per-Channel Reduction (PCR) diagnostic framework to prescribe training-free mitigations that recover up to 97% of lost alignment.

Amortized-Precision Quantization for Early-Exit Vision Transformers

Addressing the vulnerability where low-bit quantization noise disrupts early-exit decisions and causes severe accuracy collapse or delayed exits, this paper proposes Amortized-Precision Quantization (APQ) and the MAQEE bi-level optimization framework to jointly optimize exit thresholds and layer-wise bit-widths, slashing BOPs by up to 95% while matching full-precision accuracy.

ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scaling

Addressing the issue where deep layers in scaled foundation models often degrade into underutilized near-identity mappings, this paper theoretically proves that residual shortcut placement causes an exponential gap in convergence rates, and proposes Adaptive Neural Connection Reassignment (ANCRe) to dynamically learn inter-layer skip connections via ingoing normalization with under 1% overhead.

AR1-ZO: Topology-Aware Rank-1 Zeroth-Order Queries for High-Rank LoRA Fine-Tuning

Addressing the rank paradox in zeroth-order (ZO) high-rank LoRA fine-tuning where perturbation dimension explodes and finite-difference SNR collapses, AR1-ZO introduces alternating rank-1 atom queries paired with topology-aware scaling \(\gamma = \alpha r\), restoring a rank-invariant active signal under the standard two-forward-pass query budget.

ASAP: Assembly-Source Aligned Pseudocode Refinement for Binary Decompilation

To fix heuristic errors in decompiler pseudocode and prevent LLMs from succumbing to pseudocode attention bias, ASAP aligns assembly with source semantics via two-level contrastive learning, compresses chunked features into fixed tokens with a Q-Former, and enforces assembly usage via stochastic masking and an advantage loss, boosting re-execution from 64.7% to 71.9%.

ASAP: Attention Sink Anchored Pruning

By reframing the attention sink as a mass-accumulating attractor of a Lazy Random Walk, ASAP computes diffusion distances to the sink on the cumulative transition manifold, enabling single-shot adaptive background compression and foreground preservation with up to 48% speedup without any training.

AsyncOPD: How Stale Can On-Policy Distillation Be?

Addressing the system bottleneck where on-policy distillation (OPD) for long chain-of-thought reasoning is stalled by synchronous generation barriers, this paper reveals the fragility of reverse KL under finite teacher-score caches, proposes an unclipped current-advantage surrogate with Multi-Sample Monte Carlo (MC) estimation, and achieves 1.6× to 3.8× throughput speedups without sacrificing reasoning accuracy.

Attention Sinks and Outliers in Attention Residuals

To tackle severe attention sinks, activation outliers, and low-bit quantization collapse caused by dual-Softmax normalization in AttnResidual, this paper proposes OASIS, introducing Softmax1-based explicit null routes at both token and depth levels and coupling token-level null evidence to penalize uninformative depth branches, dramatically boosting post-quantization performance without altering the training objective.

Attention Transfer Is Not Universally Effective for Vision Transformers

This paper revisits Attention Transfer across a comprehensive benchmark of 20 pre-trained teachers from 11 ViT families, revealing that Attention Transfer is not universally effective: 4 major families consistently fail (underperforming from-scratch training by up to 5.1%), and proves that this failure is driven by teacher-student architectural mismatch, which can be completely reversed by adding randomly initialized teacher-native components (e.g., LayerScale, PreLayerNorm) to the student.

AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signals

To tackle distribution shifts and ungeneralizable artifacts from committing to a single privileged teacher view, AVSD extracts stable update directions from cross-view geometric consensus and selectively integrates arithmetic marginal residuals via an alignment- and magnitude-constrained gate for on-policy self-distillation.

Balancing Image Compression and Generation with Bootstrapped Tokenization

SelfBootTok introduces a self-bootstrapped 1D image tokenizer that cleanly decomposes representations into compact global semantic tokens and bootstrapped multi-granularity local tokens aligned via optimal transport, enabling downstream generators to model only 64 global tokens and achieve state-of-the-art 1.56 gFID while cutting overall training compute by 40%.

Beyond Adjacent Layers: Graph-Guided Layer Fusion for Compressing Large Language Models

Challenging the conventional assumption that functional redundancy in LLMs is confined to adjacent or contiguous layers, this paper introduces Fusion via Community Mining (FCM), a global graph-guided depth compression framework that clusters non-adjacent functionally equivalent layers and preserves phase-covering, high-Block-Influence representatives, achieving state-of-the-art 2× depth compression across 13 open-weight models.

Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation

To resolve task-dependent performance trade-offs between full fine-tuning and LoRA, this paper introduces MoLF (Mixture of LoRA and Full Fine-Tuning) and its memory-efficient variant MoLF-Efficient, which route physical weight updates across dense and low-rank pathways at the optimizer level using an Expected Preconditioned Descent (EPD) metric.

Beyond Masked Sparsity: SNACK Enables Truly Sparse Neural Networks on GPU

SNACK is a truly sparse GPU layer for PyTorch that maintains weights in COO format throughout training and inference—featuring a custom batch-to-SM mapped SpMM kernel, support-restricted sparse outer-product backward pass, and rejection-sampled uniform initializer—delivering up to 40% peak memory savings and 4.8x inference acceleration on commodity GPUs without ever materializing dense weight tensors.

Beyond Parameter Arithmetic: Sparse Complementary Fusion for Distribution-Aware Model Merging

To eliminate catastrophic repetition and output incoherence caused by dense parameter-space perturbation in model merging, this paper proposes Sparse Complementary Fusion with Reverse KL (SCF-RKL)—a data-free, distribution-aware framework that selectively injects 5%–15% high-impact weights to preserve base manifold stability while seamlessly absorbing specialized capabilities.

BLADE: Scalable Bi-Level Adaptive Data Selection for LLM Training

Addressing the computational intractability of inverse-Hessian operations in influence-based data selection and the representational lag of static reference models in excess-loss filtering, BLADE reformulates the bi-level selection objective into a penalized single-level surrogate via Lagrange multipliers, pairing dynamic reference model synchronization with a memoryless randomized block-coordinate Frank-Wolfe online token selector to deliver first-order convergence and substantial benchmark gains across diverse LLM training regimes.

Block Sphere Vector Quantization

Addressing the geometric mismatch in rotation-based quantizers that rely on coordinate-wise scalar codebooks, this paper unifies the theoretical trade-offs among EDEN, RabitQ, and TurboQuant, and proposes Block-Sphere Quantization (BlockQuant) using exact spherical block marginal distributions to achieve strictly superior MSE distortion, expected inner-product error, and KV-cache compression performance.

\(\boldsymbol{f}\)-OPD: Stabilizing Long-Horizon On-Policy Distillation with Freshness-Aware Control

f-OPD: Stabilizing Long-Horizon On-Policy Distillation with Freshness-Aware Importance Weights

Addressing the severe training collapse caused by stale buffer samples in asynchronous long-horizon on-policy distillation (OPD) of LLMs, this paper decomposes the asynchronous objective discrepancy into student rollout drift and teacher supervision drift, and proposes \(f\)-OPD with freshness-aware sample weighting, rollout-anchored regularization, and adaptive buffer refreshing to approach synchronous accuracy while retaining asynchronous throughput gains.

Bound-Conditioned Latent Inference for Progressive Image Compression

Reformulating progressive trit-plane coding as sequential latent inference under shrinking interval constraints, BLI jointly refines Gaussian entropy parameters and latent estimates using a shared model, reducing BD-rate by 17.78% over DPICT with 5.3x fewer parameters and lower latency than CTC.

Bounding Global and Local Compression Error of Signal Parameterizations

By establishing three structural properties of optimal error curves (monotonicity, discrete convexity, and finite zero error), this paper introduces a theoretically principled and computationally efficient framework to bound and predict global and local compression errors of differentiable signal parameterizations without requiring ground-truth signals.

Breaking the Group Size Barrier: Parameter-Efficient Group Dance Generation with Spatial-Temporal Guidance

Addressing group-size rigidity and identity entanglement in joint group dance modeling, ChainDance reformulates multi-dancer synthesis into a sequential Chain-of-Dancers conditional decomposition on a frozen single-dancer diffusion backbone using lightweight role and motion graph encoders alongside training-free inference noise optimization, enabling parameter-efficient generation across arbitrary group sizes without retraining.

C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models

C-LoRA identifies that catastrophic forgetting in sequential LoRA fine-tuning is driven by coupled gradient amplification on critical rank-one subspaces rather than capacity bottlenecks, and resolves it via a decoupled routing matrix with stop-gradient base retention and importance gating in a single fixed-budget adapter.

Can Model Merging Improve Aggregation in DiLoCo?

This paper demonstrates an exact mathematical equivalence between pseudo-gradient aggregation in DiLoCo and task arithmetic model merging, proposing IsoLoCo—an outer optimization scheme combining isotropic singular value equalization with Nesterov momentum that eliminates negative parameter interference as worker counts scale.

CASS: Contribution-Aware Structured Sparsity for Model Merging

CASS uses unlabeled task samples to identify attention heads and FFN neurons with prominent contributions, then filters task vectors or constrains fine-tuning gradients to reduce merging interference, improving Task Arithmetic's average accuracy across 20 tasks on ViT-B/16 from 65.02% to 69.99%.

ChunkFT: Byte-Streamed Optimization for Memory-Efficient Full Fine-Tuning

ChunkFT introduces a memory-efficient full-parameter fine-tuning framework organized around dynamically activated, byte-budget balanced chunks and custom chunk-aware backward operators that materialize gradients and optimizer states only for active sub-tensors, enabling 8B model full fine-tuning on a single RTX 4090 GPU while rivaling or outperforming dense AdamW.

Closed-Form Linear-Probe Dataset Distillation for Pre-Trained Vision Models

Focusing on the frozen-backbone linear-probing transfer paradigm, this paper presents Closed-Form Linear-Probe Dataset Distillation (CLP-DD), which computes the exact linear probe induced by synthetic data via a sample-space kernel ridge solver and optimizes synthetic pixels with a discriminative class-anchor cross-entropy loss, matching or surpassing trajectory-matching methods while accelerating distillation by ~14x and reducing GPU memory to less than one-eighth.

Co-Evolving Policy Distillation

Addressing gradient conflicts in mixed RLVR and low absorption efficiency caused by teacher-student behavioral drift in static multi-teacher distillation, this paper proposes Co-Evolving Policy Distillation (CoPD), which alternates branch-specific RLVR with cross-branch mutual on-policy distillation to keep behavioral overlap high throughout training, breaking the conventional ceiling where a unified student cannot surpass its domain-specific experts.

CODA: Rewriting Transformer Blocks as GEMM-Epilogue Programs

CODA introduces a GPU kernel abstraction for Transformers that reparameterizes non-attention memory-bound operations as composable epilogue programs attached to fixed high-performance GEMM mainloops, eliminating intermediate tensor materialization in global memory.

Codebook-Guided Cross-Modal Knowledge Distillation for Structurally Heterogeneous Features

To address cross-modal feature heterogeneity where visual spatial grids and audio/text sequences lack unit-level correspondence, this paper proposes distilling intermediate knowledge via a vector-quantized codebook with task-relevance and student-compatibility evaluation, achieving state-of-the-art transfer across five multimodal benchmarks.

Collaborative Reasoning Distillation via Cross-Feedback and Coherent Curation

Proposes Collaborative Reasoning Distillation (CRD), an end-to-end framework combining interactive multi-teacher cross-feedback, fine-grained step-wise quality assessment, and coherence-aware Viterbi path stitching to empower compact models on MATH-500 and AIME'25 using only 50K training samples.

Communication-Efficient Personalized Adaptation via Federated-Local Model Merging

To resolve the fundamental conflict between cross-client task heterogeneity and heavy communication overhead in federated large model fine-tuning, Potara introduces a principled model merging framework that formulates client personalization as a convex combination of federated and local task vectors, deriving closed-form optimal mixing weights by minimizing a variance-trace excess loss upper bound grounded in linear mode connectivity.

Compact SO(3) Equivariant Atomistic Foundation Models via Structural Pruning

Addressing the severe computational bottleneck of higher-order tensor operations in SO(3) equivariant atomistic foundation models, this paper proposes a structural pruning framework operating on complete irreducible representation blocks combined with an energy-force sensitivity criterion, strictly preserving physical equivariance while achieving up to 4× parameter reduction and 2.7× inference speedup with accuracy surpassing from-scratch trained models.

Conflict-Aware Logit Adapters for Utility-Preserving Anti-Distillation

To prevent unauthorized downstream distillation from black-box reasoning traces without incurring decoding overhead, Conflict-Aware Logit Adapters (CALA) trains a lightweight residual adapter via virtual proxy-student ascent and conflict-aware gradient projection, preserving teacher utility and surface fluency while substantially impairing student imitation.

ConQuR: Corner Aligned Activation Quantization via Optimized Rotations for LLMs

ConQuR formulates activation outlier suppression as aligning normalized activation vectors with the vertices of an inscribed unit hypercube, resolving rotation calibration via closed-form orthogonal Procrustes updates within an online streaming procedure that eliminates activation storage overhead and accounts for quantization drift.

Continuous-Time Distribution Matching for Few-Step Diffusion Distillation

Addressing the rigid discrete anchoring and numerical truncation drift in conventional distribution matching distillation, this paper introduces Continuous-Time Distribution Matching (CDM), which employs dynamic continuous scheduling and velocity-driven off-trajectory extrapolation to achieve state-of-the-art 4-step image synthesis without any auxiliary GAN or reward models.

Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillation

This paper uncovers the persistent high-loss "Rock Tokens" phenomenon in On-Policy Distillation (OPD), proves via causal knockout that the vast majority are functionally neutral structural residuals, and introduces a window-aware selective gradient freezing strategy that achieves a 1.7× training wall-clock speedup with negligible accuracy degradation.

CORP: Closed-Form One-Shot Representation-Preserving Structured Pruning for Transformers

Addressing the accuracy collapse in post-training structured pruning caused by uncorrected cascading representation errors, CORP introduces a gradient-free, fine-tuning-free, closed-form compensation framework that removes MLP hidden channels and attention Q/K dimensions using unlabeled calibration data, achieving 83.27% Top-1 on DeiT-Huge with 50% joint sparsity and a 1.64× throughput speedup.

Crowded in B-Space: Calibrating Shared Directions for LoRA Merging

This paper demonstrates that LoRA merge interference is structurally concentrated in the output-side matrix \(B\) across shared directions, and proposes Pico, a data-free pre-merge method that downscales over-shared directions in \(B\) and rescales overall update magnitude to achieve superior multi-task merge performance.

D\(^2\)Quant: Accurate Low-bit Post-Training Weight Quantization for LLMs

Addressing the drastic degradation of LLMs under sub-4-bit weight-only quantization, D2Quant introduces a zero-cost absorbable Dual-Scale Quantizer (DSQ) to relieve the down-projection bottleneck and Deviation-Aware Correction (DAC) to rectify systematic mean shift in post-attention LayerNorm, achieving superior sub-4-bit accuracy without extra bit budget or runtime overhead.

D2Quant: Accurate Low-bit Post-Training Weight Quantization for LLMs

To tackle severe accuracy degradation in sub-4-bit weight-only post-training quantization (PTQ) for LLMs, D2Quant proposes a Dual-Scale Quantizer (DSQ) with an absorbable scaling factor tailored to down-projections and Deviation-Aware Correction (DAC) for LayerNorm mean shifts, delivering state-of-the-art 2-bit accuracy without extra inference overhead.

DC-SAE: Deep Compression Semantic Autoencoder for Faster Diffusion Convergence

DC-SAE introduces a decoupled dual-stream autoencoder combining an input-level pre-merged frozen semantic branch with a learnable pixel branch alongside a decoder-side Spatial DeMerger, simultaneously achieving high-fidelity image reconstruction and rapid diffusion convergence under \(32\times\) spatial compression.

DD-Ranking: Rethinking the Evaluation of Dataset Distillation

Uncovering how soft labels and aggressive data augmentations inflate test accuracy in dataset distillation (DD), DD-Ranking establishes a unified evaluation benchmark with Label-Robust Score (LRS) and Augmentation-Robust Score (ARS) to reveal that synthetic images often underperform naive random sampling.

DDMS: Discriminative Distillation of Multi-view Foundational Features into Single-view Models

DDMS introduces a 3D-aware distillation framework that constructs a multi-view teacher by fusing frozen multi-view geometric foundation features with 2D backbone features under a discriminative ranking and semantic anchoring objective, distilling it into a single-view student to achieve strong 3D consistency and local discriminability without sacrificing semantic transferability.

Differentiable Bit-Widths: Co-optimizing Pruning and Quantization via SVD for Ultra-Efficient LLM Compression

DBW co-optimizes SVD rank truncation and mixed-precision quantization within an end-to-end differentiable framework by assigning component-level and matrix-level bit-widths including zero bits, achieving state-of-the-art performance under aggressive 1.61 bpw compression.

Differentially Private Model Merging

To address shifting differential privacy (DP) requirements during real-world deployment, this paper proposes two training-free, post-processing merging mechanisms—random selection (RS) and linear combination (LC)—along with rigorous, tight privacy accounting based on Rényi differential privacy (RDP) and privacy loss distributions (PLD).

diffusion fine-tuning with rewarded moment matching distillation

The paper introduces Rewarded Moment Matching Distillation (RMMD), a two-stage post-training framework that repurposes intermediate moment-matching distillation loss as an on-policy integral regularizer with single-step gradient backpropagation, achieving a 7.5× inference speedup on the GenCast weather model while outperforming the teacher across 93% of physical variables.

DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling

Addressing the severe quality degradation in masked discrete diffusion language models when decoding multiple tokens in parallel due to independent marginal assumptions, DiLaDiff couples a semantically regularized continuous latent space with MeanFlow ODE self-distillation, matching 200-step teacher performance in only 5 latent steps while achieving a 7× wall-time generation speedup.

DiRotQ: Rotation-Aware Quantization for 4-bit Diffusion Transformers

DiRotQ employs time-averaged PCA to decompose DiT activations into a high-variance high-precision subspace and a residual subspace, applying random orthogonal rotation to equalize residual channel variance with offline inverse rotation fused into static weights, achieving near-lossless W4A4 inference across PixArt-Σ, FLUX.1-dev, and SANA-1.6B with up to 9.2× end-to-end acceleration.

DiRotQ: Rotation-Aware Quantization for 4-bit Diffusion Transformers

Addressing severe quality degradation in 4-bit Diffusion Transformers caused by activation outliers and temporal variance shifts, DiRotQ introduces a PCA-guided rotation framework that isolates high-variance outliers into a high-precision subspace while quantizing the rotation-equalized residual subspace to 4-bit, delivering lossless W4A4 generation and a 2.3× speedup on FLUX.1-dev.

Distillation of Foundation Models for Time-dependent PDEs

To overcome the severe computational overhead and lack of physical inductive biases in time-dependent PDE foundation models, this paper introduces Teacher Rollout Extension (TREX), a distillation framework that enriches low-data downstream trajectories via noisy autoregressive rollouts to train a lightweight, translation-equivariant student with over 3,000x fewer parameters.

Distilling Graph Geometry: Knowledge Gap from GNNs to MLPs

Addressing the spectral underfit on sparse graphs and spectral overfit on dense graphs caused by topology-unaware uniform distillation, this paper introduces G2MLP, an Ollivier-Ricci curvature-guided Wasserstein alignment framework that achieves up to a 58x inference speedup while consistently outperforming both graph teachers and baseline students.

Distilling LLM Feedback for Lean Theorem Proving

Addressing the sparse rewards and mode collapse inherent to traditional reinforcement learning (such as GRPO) in formal theorem proving, this paper proposes Feedback Distillation, which trains a model on its own sampled trajectories to match token-level probability distributions conditioned on privileged language model feedback, maintaining high policy entropy while delivering an effective warm-up initialization.

Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models

CaRE-KD selects Forward or Reverse KL per token using relative teacher–student confidence and uses MC-dropout BALD to reject supervision when the teacher is uncertain but the student is comparatively certain, improving distillation quality in the tested settings and reaching 62.4 on MBPP and 73.9 on GSM8K, although gains are not universal and training becomes substantially more expensive.

Distribution Matching Distillation without Fake Score Network

To address the severe memory and optimization overhead of auxiliary fake-score networks in Distribution Matching Distillation (DMD), FSF-DMD leverages the endpoint pseudo-velocity of flow-map generators as a direct analytical surrogate for fake velocity, achieving superior 2-step ImageNet synthesis with reduced training cost across both distillation and from-scratch settings.

Distributionally Robust Mixture-of-Experts Training

Proposes Distributionally Robust MoE Training (DRMoET), which treats routed experts across layers as endogenous robustness groups and optimizes high-loss routing outcomes via activation-weighted credit and entropy-regularized mirror ascent, substantially mitigating degradation from imperfect routing without modifying forward inference.

dMoE: dLLMs with Learnable Block Experts

Addressing the severe memory-bound bottleneck in diffusion LLMs caused by independent token-level routing during block parallel decoding, dMoE aggregates token routing scores into a unified block-level distribution and dynamically selects an adaptive expert coreset via Top-\(p\) thresholding, reducing uniquely activated experts from 69.5 to 16.5, slashing memory footprint by over 74%, and achieving up to 2.04× speedup with 99.55% performance retention.

DropKV: Decoupling Residual-Output Perturbation for Near-Optimal KV-Cache Eviction

DropKV tackles the NP-hard combinatorial problem of output-perturbation minimization in long-context KV cache eviction by decoupling it into an independent per-token residual scoring objective with provable constant-factor guarantees under empirically verified cone conditions, accompanied by a fused Triton kernel that eliminates attention matrix materialization and delivers a 19.8x kernel speedup.

Dynamic Model Merging Made Slim

Addressing the severe parameter imbalance between shared and expert components in dynamic model merging, DiDi-Merging jointly optimizes low-rank capacities via differentiable continuous rank selection followed by data-free parameter-space refinement, matching independent fine-tuning accuracy with an ultra-compact 1.24x backbone storage footprint.

Dynamical Adapter Fusion: Constructing A Global Adapter for Pre-Trained Model-based Class-Incremental Learning

This paper proposes Dynamical Adapter Fusion (DAF), which derives optimal, closed-form fusion coefficients via PAC-Bayes generalization bounds and Lagrange multipliers to dynamically merge task-specific adapters, a robust historical prior, and the previous global adapter into a single persistent global adapter, completely eliminating test-time retrieval overhead and historical module storage.

DyRA: Dynamic Residual Approximation for Efficient Matrix Multiplication in DNNs

Addressing the issue where small weight-space approximation errors are severely magnified by input activations in deep neural networks, DyRA dynamically approximates and compensates output residual errors during inference using one-step alternating least squares (ALS), cutting accuracy degradation by more than 3× under matched FLOP budgets and delivering a 1.5× end-to-end GPU speedup on DINOv3.

Exact Instance Compression for Convex Empirical Risk Minimization via Color Refinement

Addressing the computational bottleneck of solving convex empirical risk minimization on large-scale datasets, this paper introduces an exact, lossless instance and feature compression framework based on color refinement from graph isomorphism, provably preserving global optimality and convexity while dramatically shrinking problem dimensions and cutting end-to-end training runtime.

Exact Unlearning via Quantized Sufficient Statistics

Quantized Sufficient Statistics (QSS) decomposes machine learning models into a small frozen schema and additive quantized accumulators, reducing exact unlearning to constant-time arithmetic subtractions and achieving near-SISA accuracy alongside up to 483× unlearning speedups at retention ratio \(\rho=0.5\%\).

FacePhys: State of the Heart Learning

FacePhys formulates camera-based physiological sensing as a neural controlled differential equation discretized via temporal-spatial state space duality, embedding complex oscillatory priors into a diagonal state transition matrix to achieve real-time streaming rPPG estimation with a 3.6 MB memory footprint and 9.46 ms per-frame latency.

FeatCal: Feature Calibration for Post-Merging Models

To bridge the persistent performance gap between merged models and task-specific experts, FeatCal theoretically decomposes layer-wise feature drift into upstream propagation and local mismatch, calibrating linear weights via closed-form ridge updates in forward order without gradient iterations or inference-time plugins.

Few-Step Diffusion Language Models via Trajectory Self-Distillation

To resolve the catastrophic degradation of masked diffusion language models under step compression caused by token factorization approximations, this paper introduces T3D, a trajectory self-distillation framework that matches the teacher's full denoising trajectory via reverse-KL Direct Discriminative Optimization (DDO) and early-token path-consistency regularization, closing the gap between few-step and full-step diffusion decoding.

FiLoRA: Focus-and-Ignore LoRA for Controllable Feature Reliance

FiLoRA decomposes low-rank adaptation into functionally aligned module groups and modulates their forward contribution via instruction-conditioned gating, enabling natural language instructions to steer internal feature reliance while keeping the task objective and base model weights frozen.

Flash PD-SSM: Memory-Optimized Structured Sparse State-Space Models

Addressing the prohibitive \(\mathcal{O}(LN^2)\) memory and compute bottleneck in constructing input-dependent transition matrices for structured sparse state-space models, Flash PD-SSM introduces a discrete dictionary hard routing mechanism coupled with a slope-annealed straight-through estimator to reduce memory complexity to \(\mathcal{O}(LN)\), achieving training throughput exceeding Mamba2 and SoTA state-tracking accuracy across synthetic automata, long-range multivariate time-series, and hybrid LLMs.

Fractional Power-of-Two Quantization for Efficient and Effective Multiplier-Free LLM Inference

FPoT proposes a base-\(2\sqrt{2}\) (half-octave) fractional power-of-two grid paired with uniform integer activations and 1-adder approximate-grid hardware calibration, enabling multiplier-free shift-add LLM inference with near-lossless 4-bit performance on par with uniform INT4.

From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs

Addressing the severe intra-expert channel activation sparsity in MoE architectures, NSFT refines parameter-efficient adaptation from entire experts to sub-experts (structured channel groups) equipped with learning-rate compensation and dynamic gradient scaling, achieving superior domain specialization and general capability retention with significantly fewer trainable parameters.

From “Weak” Signals to Strong Models: Preference Delta Aggregation with LoRA Merging

Addressing the scarcity of high-quality supervision, this paper introduces the PDA framework to parameterize preference deltas from diverse weak-weaker model pairs into modular LoRA adapters and develops GAM, a Grassmannian orthogonal Procrustes alignment technique that mitigates subspace interference for constructive model merging.

FuRA: Full-Rank Parameter-Efficient Fine-Tuning with Spectral Preconditioning

Addressing the issue where standard fine-tuning and LoRA perturb pretrained spectral geometry via noisy gradients, FuRA introduces a block tensor-train parameterization that projects weight updates into the pretrained column space with adaptive spectral preconditioning, achieving full-rank adaptation at LoRA-level compute and memory while consistently outperforming Full Fine-Tuning.

GeneZip: Region-Aware Compression for Long-Context DNA Modeling

Addressing the prohibitive token-mixing cost and biological mismatch of uniform downsampling in long-context genomics, GeneZip couples static gene-structure-guided Region-Aware Ratio (RAR) loss with bounded dynamic routing to compress raw DNA up to 137.6 BPT without test annotations, securing top average rank across DNALongBench benchmarks.

GeoPair: Geometry-Preserving Cross-Layer Factorization for Training-Free Transformer Compression

GeoPair is a principled, training-free cross-layer factorization framework that replaces heuristic covariance averaging and adjacent layer pairing with exact generalized Sylvester equation solvers and global graph matching, preserving layer-specific activation geometries and achieving state-of-the-art compression fidelity without fine-tuning.

GeoSprint: Geometric Redundancy-Aware Step Pruning for Inference in Diffusion Transformers

GeoSPRINT is a training-free acceleration framework that detects geometric redundancy along diffusion trajectories via thin-QR hyperplanarity testing and constructs non-uniform sampling schedules by blending log-SNR spacing with trajectory curvature, allowing a first-order DDIM solver to match or outperform higher-order ODE solvers.

Grid Games: The Power of Multiple Grids for Quantizing Large Language Models

This paper introduces the power-of-two-grids (PO2) problem for microscaled 4-bit LLM quantization, theoretically proving the concavity advantage of adaptive dual grids in small-group regimes and their asymptotic irrelevance in large-group regimes, while presenting the asymmetric PO2(Split87) pair and the TensorCore-compatible shifted grid triple SFP4.

GRINQH: Graded Input-Based Quantization Hierarchy for Efficient LLM Generation

Targeting the GPU memory bandwidth wall during batch-size-one autoregressive decoding, GRINQH introduces a decoding-centric dynamic quantization framework that allocates graded bit widths (0 to 8 bits) per input channel based on runtime activation magnitudes, pairing with a hierarchical bit-planar layout to establish a new Pareto frontier at 2–3 effective bits and accelerate decoding throughput by up to 28% over 4-bit MARLIN.

Hestia: A Hessian-Guided Differentiable Quantization-Aware Training Framework for Extremely Low-Bit LLMs

Addressing gradient mismatch and dead-zone stagnation caused by the Straight-Through Estimator (STE) in ultra-low-bit LLMs, Hestia introduces a temperature-controlled Softmax relaxation with a dynamic pressure continuation path, and adaptively schedules tensor-wise annealing via lightweight Hessian-trace curvature to achieve state-of-the-art ternary performance on 10B training tokens.

HiLoRA: Adaptive Hierarchical LoRA Routing for Training-Free Domain Generalization

HiLoRA introduces a training-free hierarchical routing framework that decomposes LoRA adapters into rank-one components (ROCs), adaptively allocating LoRA candidates and ROC budgets via sequence-level Gaussian likelihoods and refining token-level activations via down-projection responsiveness for robust domain generalization.

Hybrid-LoRA: Bridging Full Fine-Tuning and Low-Rank Adaptation for Post-Training

Addressing the excessive GPU memory overhead of full fine-tuning (FFT) and the noticeable performance degradation of Low-Rank Adaptation (LoRA) in reasoning-oriented RLVR post-training, this paper presents Hybrid-LoRA, which ranks candidate linear modules by their first-order gradient sensitivity via a novel Hybrid-LoRA Score to selectively assign the most adaptation-sensitive modules (~10%) to full fine-tuning while adapting the remaining components with LoRA, closely matching or even exceeding full fine-tuning performance.

Importance-Aware OBS Pruning for Diffusion Models

The paper injects prompt-related spatial importance into the layer-wise OBS reconstruction objective to improve subject fidelity in highly sparse diffusion models without fine-tuning, but its global mask-coefficient ablation has an unexplained inconsistency with the stated exact equations.

Inner Product Aware Quantization: Provably Fast, Accurate, and Adaptive Algorithms

Addressing the frequent inner product computations in vector search and neural weights, this paper formulates worst-case and average-case inner product aware quantization objectives, proves the optimality and hardness boundaries of stochastic rounding, and develops provably fast, adaptive algorithms achieving 2–10× speedups over prior state-of-the-art methods.

Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

This paper introduces "Intelligence per Watt" (IPW) as a unified metric for measuring the efficiency of on-device LLM inference, demonstrating across 20+ local models, 8 hardware accelerators, and 1M+ real-world queries that local LMs improved IPW by 5.3x from 2023 to 2025 while hybrid edge-cloud routing slashes energy and serving costs by 60%–80%.

KL for a KL: On-Policy Distillation with Control Variate Baseline

Addressing the severe training instability in On-Policy Distillation (OPD) caused by high gradient variance in single-sample Monte Carlo estimation, this paper casts OPD into policy-gradient RL, proves that its value function admits a closed form as the per-token negative reverse KL divergence, and introduces vOPD—an unbiased, critic-free control variate baseline with an efficient top-\(k\) approximation that matches full-vocabulary performance while slashing computational overhead.

KV-COBRA: KV Cache Compression via Co-Optimized Bit-Rank Allocation

Addressing the severe performance collapse of existing KV cache compressors at extreme low bit-rates (0.5–2 bpd) caused by uniform hyperparameter assignment, KV-COBRA formulates KV cache compression as a joint rate-distortion problem, co-optimizing per-head rank truncation and scalar quantization bit-widths via calibration-time water-filling and attention-KL basis reordering, achieving near-lossless generation with zero runtime overhead.

KV-COBRA: KV Cache Compression via Co-Optimized Bit-Rank Allocation

Addressing accuracy collapse at extreme compression rates, KV-COBRA demonstrates that the bottleneck lies in head-wise budget allocation rather than the quantization kernel itself, co-optimizing per-head rank truncation and bit-width along with attention-KL importance reordering at calibration time with zero inference overhead.

Label-Efficient Dataset Pruning via Semi-Supervised Pseudo-Labeling

SemiPrune introduces a label-efficient dataset pruning framework that leverages a small labeled subset (e.g., 10% or even 1%–5%) via semi-supervised learning to generate target-adaptive pseudo-labels, overcoming the fundamental failure of label-free pruning on domain shifts, corruptions, and long-tailed distributions.

LAMP: Look-Ahead Mixed-Precision Inference of Large Language Models

LAMP introduces an adaptive look-ahead mixed-precision evaluation paradigm for deep composite functions that identifies numerically sensitive intermediate components amplified by subsequent nonlinearities and recomputes them in high precision, reducing LLM attention rounding errors by up to two orders of magnitude with only ~1% recomputations.

LAQuant: A Simple Overhead-free Large Reasoning Model Quantization by Layer-wise Lookahead Loss

Addressing the severe accuracy collapse of large reasoning models during long-chain autoregressive decoding, this paper proposes LAQuant—a runtime-overhead-free layer-wise quantization-aware training recipe that couples a single-layer lookahead loss for implicit cross-layer co-adaptation with Hessian-subspace-aligned calibration data to preserve critical KV-cache fidelity.

LARK: Learnability-Grounded Trajectory Selection for Efficient Reasoning Distillation

Addressing the heuristic flaws of filtering teacher chain-of-thought trajectories solely by surface quality or student likelihood in reasoning distillation, LARK leverages optimization theory to propose a backward-free forward proxy score \(\hat{g}_k\) for anchor-time gradient flow loss decay and a closed-form \(\chi^2\)-regularized budgeted weighting rule, achieving highly efficient and generalization-preserving supervision.

Layerwise Progressive Freezing: A Training Scaffold for Depth-Scalable Binary Networks

Addressing the severe degradation and training collapse of deep binary neural networks (BNNs), this paper identifies the mechanism of activation-induced gradient blockades and proposes StoMPP, an input-to-output layerwise progressive freezing procedure with soft refresh that flattens depth degradation in both STE-free and STE-augmented regimes.

Layerwise Progressive Freezing: A Training Scaffold for Depth-Scalable Binary Networks

Addressing the severe accuracy collapse of full binary neural networks (BNNs) under depth scaling caused by forward/backward mismatch in the straight-through estimator (STE), this paper proposes StoMPP, an input-to-output layerwise progressive freezing scaffold that eliminates activation-induced gradient blockades and enables depth-scalable binary networks without requiring surrogate gradients.

Learning from the Self-future: On-policy Self-distillation for dLLMs

Tailored for the arbitrary-order non-autoregressive decoding of diffusion LLMs, d-OPSD introduces the first on-policy self-distillation framework by conditioning self-teachers on self-generated suffix future answers and providing step-level divergence supervision, achieving superior reasoning performance with only ~10% of RLVR optimization steps.

Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-Tuning

Through a systematic audit of 64 published LoRA studies and extensive hyperparameter sweeps across diverse LLM scales, this paper demonstrates that vanilla LoRA matches advanced variants within a 1%–2% margin once learning rates are properly tuned, with optimal ranges explained by Hessian loss curvature.

Learning Visual Speech Representations via Cross-Modal Distillation and Joint Face-Lip Modeling

FLARE leverages Whisper acoustic features as supervision targets via random-projection quantization and integrates dual-stream face-lip ConvNeXt frontends with an enhanced Conformer backbone, achieving state-of-the-art lipreading robustness with dramatically reduced data dependency.

Learning What to Remember: Test-Time Training via Context Distillation

Addressing the failure of existing test-time training (TTT) and recurrent models to distinguish information with future utility from irrelevant details, this paper proposes Test-Time Context Distillation (TTCD), which leverages the hidden-state discrepancy between a long-window teacher and a short-window student to update MLP fast weights via an in-place self-supervised objective.

Leech Lattice Vector Quantization for Efficient LLM Compression

This paper introduces LLVQ, a codebook-free vector quantization framework using the 24-dimensional Leech lattice with multi-shell angular search, invertible Golay-based indexing, and norm-preserving Spherical GPTQ, establishing new SOTA at 2 bits per weight while enabling rotation-free PTQ.

LensVLM: Selective Context Expansion for Compressed Visual Representation of Text

LensVLM compresses long documents into low-resolution rendered images for efficient global scanning and trains the VLM to selectively expand critical regions into uncompressed text via an Expand tool, matching full-text QA accuracy at 4.3x effective compression while outperforming retrieval and token pruning baselines.

LinearARD: Linear-Memory Attention Distillation for RoPE Restoration

Addressing catastrophic short-context degradation in RoPE-scaled LLMs, LinearARD introduces self-distillation over top-layer Q/K/V self-relation distributions paired with an exact linear-memory KL kernel, recovering 93%–94% of native short-text accuracy and high long-context robustness using just 4.25M tokens (1.6% of baseline compute).

Locking Pretrained Weights via Deep Low-Rank Residual Distillation

To defend open-weight models against unauthorized fine-tuning and safety jailbreaking, DLR-Lock exploits the physical memory asymmetry between inference and training in automatic differentiation by replacing pretrained wide MLPs with parameter-matched deep low-rank residual networks (DLR-Nets), maintaining inference utility and FLOPs while forcing linear activation memory expansion and severe optimization barriers during backpropagation.

LOFT: Low-Rank Orthogonal Fine-Tuning via Task-Aware Support Selection

LOFT introduces a unified framework that explicitly decomposes parameter-efficient orthogonal fine-tuning into adaptation support selection and in-subspace transformation, deriving a task-aware support selection rule (SkewGrad) via first-order gradient dynamics that significantly outperforms weight-principal baselines while preserving pretrained geometry.

LoRaQ: Optimized Low Rank Approximation for 4-bit Quantization

Addressing the severe generative degradation of diffusion transformers under aggressive 4-bit quantization, this paper introduces LoRaQ, a data-free post-training quantization method that directly shapes weight errors into low-rank structures and enables the first fully sub-16 bit mixed-precision pipeline for both residual and auxiliary branches, outperforming SVDQuant under identical memory budgets.

LoRASpace: A Pool-Wide Shared Substrate for Static and Dynamic Multi-LoRA Composition

Addressing the trade-off between output distortion in static merging and linear computational scaling in composite execution, LoRASpace exploits pool-wide low-rank subspace overlap to build a training-free shared substrate that matches composite-level visual quality at merge inference cost while achieving up to 6.1× dynamic speedup.

LOTION: Smoothing the Optimization Landscape for Quantized Training

Addressing vanishing gradients and straight-through estimator (STE) instability caused by piecewise-constant quantizers in low-bit training, LOTION directly smooths the quantized loss surface using the expectation under unbiased randomized-rounding noise, preserving original global minima and yielding an equivalent diagonal Gauss-Newton curvature regularizer that significantly outperforms conventional QAT on 150M/300M language models in INT4 and FP4 precision.

LowRankArena: A Standardized Evaluation Platform for SVD-Based LLM Compression

Addressing benchmark fragmentation and disconnected efficiency claims in SVD-based LLM compression, this paper presents LowRankArena, a standardized evaluation platform with unified task suites, uniform-precision budgets, and end-to-end vLLM serving across 3+ TiB of released checkpoints, revealing that method rankings are architecture-dependent and decode-stage speedups remain constrained by memory bandwidth.

LSVD: Loss-Aware Low-Rank Approximation for Efficient Low-Precision Vision-Language Models

To alleviate the severe computational and memory-bandwidth bottlenecks in deploying vision-language models under low precision, LSVD introduces curvature-weighted loss-aware low-rank decomposition, calibration-gradient-based cross-layer rank allocation, and a hybrid low-rank quantized FFN, achieving 2.3× to 4.7× decoding speedup while preserving near-lossless accuracy.

Making LoRA Identifiable: Orthogonal Alignment for Continual Learning

To resolve coordinate misalignment and representational interference in single-adapter continual learning caused by the non-identifiability of low-rank factorization, this paper proposes SOAR, a framework that constrains the input matrix onto the Stiefel manifold to collapse the latent transformation ambiguity to orthogonal rotations, coupled with closed-form Orthogonal Procrustes alignment and per-rank retention gating for conflict-free consolidation under constant memory.

MemCoRe: Recovering Evidence from Progressively Compressed Factual Knowledge for Agent Memory

Addressing the fundamental tension in LLM agent memory between over-retention obscuring evidence and aggressive compression discarding it, MemCoRe introduces an information-bottleneck-principled hierarchical compression framework spanning notes, keywords, and topic groups, coupled with gated construction and tri-pathway iterative retrieval to compress redundancy while ensuring accurate evidence recovery.

MetaCluster: Enabling Deep Compression of Kolmogorov-Arnold Network

MetaCluster proposes a three-stage compression framework that merges meta-learned manifold shaping with weight sharing for Kolmogorov-Arnold Networks (KANs), constraining per-edge basis coefficient vectors to a compact, clusterable subspace and achieving up to 80× and 124.1× storage reduction on vision and equation modeling tasks without accuracy loss.

Minimax Rates and Spectral Distillation for Tree Ensembles

Adopting an operator-theoretic spectral viewpoint, this paper proves the first minimax-optimal convergence rate for adaptive random forest regression under polynomial kernel eigenvalue decay and introduces SCATE, a spectral distillation scheme that compresses tree ensembles into resource-exact MLPs achieving orders-of-magnitude footprint reduction.

Mixture-of-Control: State-Aware Fine-Tuning for Transformer-based Models

Addressing the trade-off between isolated block adaptation in state-based tuning and high compute/memory overhead in full-block cross-layer routing, Mixture-of-Control (MoC) routes sparse control experts across transformer layers to fuse local and global control signals, achieving superior adaptation accuracy while retaining state-based memory efficiency.

Mixture-Trained Merging for Unified Multi-Objective Models

Addressing sharp cross-domain accuracy trade-offs and think/non-think mode collapse in naive expert merging, Mixture-Trained Merging (MTM) trains soft-expert branches on objective-biased data mixtures and iteratively couples proxy merge evaluation, base promotion, and mixture refinement via multi-objective Bayesian optimization to construct unified multi-capability models.

MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Any-Precision LLM

Addressing cross-precision generalization degradation and outlier migration in post-training quantization, MoBiQuant introduces a recursive residual bit-slicing framework (MoBiSlice) coupled with a token-adaptive router (MoBiRoute), achieving repacking-free, kernel-relaunch-free any-precision LLM inference under unified calibration parameters.

MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Any-Precision LLM Inference

MoBiQuant tackles poor cross-bit generalization and hardware overheads in any-precision LLM quantization by combining recursive residual bit-slicing (MoBiSlice) with token-adaptive gating (MoBiRoute), achieving seamless runtime precision switching and up to 1.34× throughput gain with a single set of calibration parameters.

MoEZip: Routing-Aware KV Cache Compression for Sparse Mixture-of-Experts LLMs

Targeting the catastrophic expert routing failure in long-context sparse MoE LLMs under aggressive KV cache eviction, MoEZip introduces a closed-form routing fragility metric based on the Fisher information matrix trace, combining it with instruction attention to adaptively safeguard routing anchors and achieving 2.3× to 4.9× higher benchmark scores over prior SoTA at 10% KV cache retention.

MPQ: A Message-Passing View of Post-Training Quantization

To overcome the premature hard-decision trap in classical greedy PTQ solvers, this paper formulates row-wise quantization as Gibbs posterior inference over discrete codebooks, introducing a two-stage solver MPQ that unites finite-temperature vector approximate message passing (VAMP) with exact block coordinate descent (CD) to achieve substantial perplexity and accuracy gains in aggressive 2-bit lattice and 3/4-bit scalar regimes.

NADS: Navigator-Guided Data Selection for Mitigating Catastrophic Forgetting in Fine-Tuning

Addressing catastrophic forgetting in LLM fine-tuning when pretraining data is unavailable, NADS leverages predictive divergence from an unconstrained navigator model to identify forgetting-vulnerable candidates, applies a scalable whitened-diagonal DPP algorithm to efficiently select high-utility and diverse synthetic constraints, and fine-tunes with knowledge distillation to preserve general capabilities.

Next-Latent Prediction Transformers Learn Compact World Models

NextLat trains autoregressive transformers to predict their own next hidden states via a lightweight latent dynamics model and stop-gradient self-distillation, provably inducing compact belief-state world models that yield up to 3.3× variable-length self-speculative decoding speedups with zero architectural alterations.

Normalized Architectures are Natively 4-Bit

This paper demonstrates that nGPT, an architecture constraining weights and hidden representations to the unit hypersphere, is natively robust to 4-bit arithmetic, enabling stable end-to-end NVFP4 training on Blackwell GPUs without randomized Hadamard transforms or per-tensor scaling because the hypersphere constraint induces weak positive correlations that constructively amplify signal accumulation across the dot product rather than suppressing local quantization noise.

Not All Layers Need Tuning: Selective Layer Restoration Recovers Diversity

Addressing the widespread issue of mode collapse caused by post-training in large language models, this paper proposes Selective Layer Restoration (SLR), a training-free method with zero inference overhead that uses a discrete Constrained Random Character (CRC) proxy task to search for Pareto-optimal layer intervals and restore them to pre-trained weights, substantially boosting output diversity across creative writing, open-ended QA, and multi-step reasoning.

Not All Tasks Quantize Equally: Fisher-Guided Quantization for Visual Geometry Transformer

FGQ weights channel reconstruction errors in block-wise post-training quantization of VGGT using normalized squared gradients from its geometric tasks, prioritizing sensitive features during calibration; under W4A4, mean completeness error on 7-Scenes falls from QuantVGGT's 0.085 to 0.059, although not every metric recovers FP16 performance.

NP-LoRA: Null Space Projection for Subject-Style LoRA Fusion

NP-LoRA is a training-free framework that projects content LoRAs into the complementary null space of style LoRAs via an exact closed-form soft projection, effectively eliminating conflicting updates in overlapping low-rank subspaces while preserving subject identity.

NPUsper: Eliminating Redundant Computation for Real-Time Whisper on Mobile NPUs

To overcome heavy padding overheads used to suppress hallucinations and the structural mismatch between dynamic autoregressive decoding and static-graph NPU execution in streaming Whisper, NPUsper introduces an online cross-attention temporal hallucination detector paired with a controlled unrolling chunk-graph decoding architecture, drastically slashing redundant computation and power consumption while preserving transcription accuracy.

NuMuon: Nuclear-Norm-Constrained Muon for Compressible LLM Training

Addressing the severe performance degradation of Muon-trained models under aggressive post-training compression, this work introduces NuMuon—an optimizer that augments Muon's spectral-norm LMO with a nuclear-norm constraint, yielding a closed-form top-\(k\) singular update that retains pretraining convergence while improving post-compression quality by up to 55.8%.

OCTOPUS: Optimized KV Cache for Transformers via Octahedral Parametrization Under Optimal Squared Error Quantization

Addressing memory bandwidth and footprint bottlenecks in long-context autoregressive transformers, OCTOPUS introduces joint quantization of rotated coordinate triplets mapped onto a 2D square via octahedral parameterization, achieving state-of-the-art compression fidelity across text, video, and audio without data-dependent calibration.

On the Construction and Implications of Low-Loss Valleys in LoRA-based Bayesian Inference

On the Construction and Implications of Low-Loss Valleys in LoRA-based Bayesian Inference

LoRA-Curve constructs continuous, segmented Bézier paths connecting distinct LoRA optima into low-loss valleys, enabling smooth parameter-space traversals and superior Bayesian model averaging without the prohibitive compute of traditional ensembling or the restrictive single-mode assumptions of variational inference.

Overcoming Kernel Redundancy for Scaling Logic Gate Networks

Addressing performance saturation in differentiable logic gate networks caused by kernel redundancy during width scaling, this paper proposes a Boolean-routed Dynamic Logic Kernel (DLK) and an Early-stage Dynamic Logic Kernel (EDLK), achieving higher representation diversity and parameter efficiency within a fully logic-gate-based inference path.

PACE: Pareto-Adaptive Compression for Efficient Native MLLMs

PACE introduces Spatially Decoupled Importance Estimation (SDIE) and Pareto Adaptive Compression Scheduling (PACS) for ViT-free native MLLMs with Multimodal-RoPE, achieving near-lossless or improved accuracy across multimodal benchmarks at 25%–50% visual token retention while reducing inference FLOPs by approximately 2×.

PATCH: Learnable Tile-Level Hybrid Sparsity for LLMs

PATCH introduces a learnable tile-level hybrid sparsity framework that partitions weight matrices into hardware-friendly tiles and learns to assign each tile as either dense or 2:4 sparse via Gumbel-Softmax optimization over frozen weights, enabling continuous 0%–50% global sparsity ratios and achieving 1.18×–1.38× measured end-to-end inference speedups.

PermuQuant: Lowering Per-Group Quantization Error by Reordering Channels for Diffusion Models

Addressing the critical issue where outlier channels dominate shared group scales in low-bit diffusion models, PermuQuant introduces a post-training channel reordering framework driven by a joint second-moment criterion and calibration-based acceptance rule, achieving runtime-overhead-free low-bit quantization.

PerQ: Inverse Generative Modeling for Neural Image Compression via Quantization Error Compensation

PerQ constrains generative reconstruction within a rate-determined quantization cell via a decoder-side flow-matching compensator attached to a frozen multi-rate backbone, matching generative perceptual realism while preserving VVC-level pixel fidelity and fast 4-step decoding in a single model.

PGSB: Pretrained-Guided Shared Basis for LoRA Model Merging

PGSB proposes a two-stage framework for LoRA model merging that first extracts shared bilateral low-rank support subspaces and then refines the merge coordinates using singular-value-weighted pretrained weight interactions, substantially reducing coordinate mixing in coordinate-sensitive merge operators at near-zero extra cost.

Polestar: Drift-Aware Cache Calibration and Token Commitment for Efficient Inference of Diffusion LLMs

Addressing the dual bottlenecks in masked diffusion LLMs (dLLMs) where bidirectional attention prevents efficient KV-cache reuse and parallel decoding suffers from confidence instability, Polestar leverages KL-divergence-based token representation drift as a unified physical signal to guide sparse KV-cache refreshes and early token commitments, delivering up to 3.7× higher throughput and 3.67 tokens per forward pass without fine-tuning.

POP: Online Structural Pruning Enables Efficient Inference of Large Foundation Models

Addressing the severe generation collapse in prior structural pruning caused by static masks during autoregressive decoding, POP proposes a partition-guided online structural pruning framework that divides channels into retained, pruned, and candidate sets during prefilling, dynamically evaluating only the candidate set per decoding step with under 4% FFN FLOPs overhead, significantly recovering generative accuracy with 1.14×–1.38× end-to-end speedups without any offline calibration.

Precision As You Need: Stochastic Computing Is a Dense Adaptive Quantizer

Reinterprets stochastic computing (SC) as a dense adaptive quantizer whose precision is continuously governed by stream length, coupling a dynamic per-row mixed-precision policy with fused GPU Triton kernels to match full precision across diverse vision transformers without retraining.

Predicting Quantization Price for Selecting PTQ Configurations Before Deployment

Addressing the fundamental challenge that global degradation in post-training quantization (PTQ) remains hidden until the complete model is built, this paper formulates weight-space PTQ as a pre-deployment configuration selection problem priced by layer-output error covariance and downstream curvature, enabling ultra-fast budgeted configuration search during calibration.

Prediction Under Imperfect Compression: A Theory of Approximate MDL

This paper formally establishes the first comprehensive theory of approximate Minimum Description Length (Approx-MDL) sequence prediction, proving that additive optimization slack preserves finite cumulative squared error for regularization strength \(\lambda \ge 1\), while identifying a sharp threshold below which weak regularization (\(\lambda < 1\)) or multiplicative slack induces catastrophic overfitting.

Procedural Memory Distillation: Online Reflection for Self-Improving Language Models

Addressing the limitation of reinforcement learning with verifiable rewards (RLVR) and self-distillation methods that discard episode-level experiences, this paper proposes Procedural Memory Distillation (PMD), which constructs a three-level procedural memory hierarchy (experience, insight, behavior) that co-evolves with the policy online, distilling cross-episode reflection into model weights via reverse-KL to achieve native, memory-free self-improvement at inference time.

Progress-Aware Distillation for Mitigating Stagnation in Small Language Model Agents

To tackle progress stagnation where small language model agents become trapped in unproductive reasoning-action loops during multi-turn interactions, this paper introduces Progress-Aware Distillation (ProD), which encodes teacher-student stagnation discrepancies into an adaptive margin to dynamically penalize redundant cycles, substantially reducing stagnation while boosting task accuracy and interaction efficiency.

Pruning and Distilling Mixture-of-Experts into Dense Language Models

The first systematic framework for converting trained MoE language models into standard fully dense architectures via D-optimal diversity-aware expert scoring, unmerged pure pruning concatenation, and forward-KL knowledge distillation, significantly outperforming matched dense-to-dense pruning.

RA-CFGCache: From Branch-Level Criteria to Guided-Risk Control under Classifier-Free Guidance

Addressing the branch–guided mismatch and local–final propagation mismatch inherent in classifier-free guidance (CFG), RA-CFGCache introduces a training-free caching framework that composes branch proxies via exact CFG error geometry and rescales local risks using empirical propagation priors to govern joint branch reuse without changing the sampling trajectory.

RaZeR: Pushing the Limits of NVFP4 Quantization with Redundant Zero Remapping

RaZeR uncovers unused sign and exponent bits within FP8 block scaling factors to repurpose the redundant FP4 negative zero into adaptive high-accuracy special quantization values, reducing perplexity degradation by 31%~35% under identical memory footprint and co-designing high-performance GPU kernels with lightweight Tensor Core microarchitecture.

Re-examining Low Rank adaptation for private LLM fine-tuning

Reveals that isotropic Gaussian noise in DP-SGD inflates gradient singular values and disrupts their natural rapid decay, acting as the primary optimization bottleneck; proposes a single-step RMT-based spectral denoising method combined with phase-transition thresholding and block-norm correction that speeds up training by 15.6%–75.0% and saves significant privacy budget without compromising formal DP guarantees.

Reasoning as Compression: Unifying Budget Forcing via the Conditional Information Bottleneck

This paper reframes reasoning chain compression as a lossy source coding problem under the Conditional Information Bottleneck (CIB) with side information, introducing a language model surprisal-based semantic prior that unifies existing budget forcing heuristics and achieves Pareto-optimal efficiency-accuracy trade-offs.

Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVR

Addressing the drawback where self-distillation overwrites a model's independent reasoning on successful rollouts, this paper proposes RLRT to invert teacher guidance on correct trajectories by amplifying self-driven exploration tokens, substantially outperforming GRPO and conventional self-distillation across mathematical reasoning benchmarks.

Recovery Guarantees for Posterior Sampling of One-Bit Compressed Sensing

Addressing the unknown geometric complexity of data priors and severe information loss in noisy one-bit compressed sensing, this paper establishes matching sample complexity upper and lower bounds governed by the approximate covering number \(\log \mathrm{Cov}_{\eta, \delta}(R)\) with Wasserstein prior mismatch robustness, and develops a split Gibbs diffusion plug-and-play sampler (PnP-OneBit) that outperforms deterministic point estimators.

Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs

Addressing the operational burden of maintaining isolated checkpoints across diverse deployment constraints, this paper proposes Recurrent Residual Quantization (RRQ), representing weights as an ultralight low-bit base coupled with cascaded residual corrections to construct calibration-free 2/4/6/8-bit representations from a single package.

ReSET: Accurate Latency-Critical NVFP4 Reasoning via Step-Aware Temperature Scaling

ReSET addresses the twin bottlenecks of low-precision NVFP4 reasoning—accuracy degradation at locally deterministic symbolic tokens and hardware underutilization in latency-critical small-batch decoding—via step-aware dynamic temperature scaling combined with a specialized CUDA-core NVFP4 GEMV kernel, outperforming PTQ baselines and achieving up to a 2× end-to-end decoding speedup on NVIDIA B200 GPUs.

Resource-Aware Parameter-Efficient Model Adaptation for Onboard High-Dimensional Data

To overcome the severe uplink bandwidth bottleneck and complex spectral-spatial nonlinearities in onboard hyperspectral model updating on LEO satellites, NE-LoRA introduces a GELU-activated auxiliary branch coupled with matrix-differentiated learning rates, achieving parity with or superiority over full fine-tuning while retaining standard LoRA uplink deployment costs.

Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets?

Through extensive benchmarks across standardized hard-label and soft-label protocols on ImageNet benchmarks, this work demonstrates that SOTA dataset distillation methods fail to outperform coresets without auxiliary teacher supervision, while incurring orders-of-magnitude higher construction costs and severe mode collapse.

Rethinking Layer-wise Model Merging through Chain of Merges

Addressing the Merging Covariate Shift (MCS) induced by inter-layer independence assumptions in layer-wise model merging, Chain of Merges (CoM) recursively propagates activations through merged layers and applies correlation-weighted closed-form regression, achieving state-of-the-art multi-task performance without retraining.

Rethinking LoRA Initialization for Robust Asymmetric Learning Rates

Addressing the issue where LoRA+ asymmetric learning rates suffer from training instability and ratio-dependent performance collapse caused by the amplification of initial random projections, this paper proposes the minimalist Init[A^2] scheme (\(\sigma_A^2 = \Theta(n^{-2})\)), which suppresses initialization noise energy by a factor of \(1/n\) while preserving zero initial adapter output, thereby eliminating high-ratio collapse and significantly expanding the stable hyperparameter search region.

Rethinking On-Policy Self-Distillation for Thinking Models

Reveals that privileged-context on-policy self-distillation (OPSD) severely undermines long reasoning trajectories in thinking models, driven by a mechanistic failure mode where privileged teachers penalize exploratory forking positions and eliminate the model's test-time search and backtracking capabilities.

Revisiting On-Policy Adversarial Black-Box Distillation: Calibrating Groupwise Re-Ranking

To tackle the groupwise reward geometry mismatch in on-policy adversarial black-box distillation—where the critic's Bradley-Terry objective naturally favors zero within-group dispersion and corrupts GRPO advantage estimation—GRGC introduces critic-side Gaussian optimal transport calibration and policy-side group power modulation, yielding substantial distillation gains across multiple model scales with under 0.11% runtime overhead.

Reviving Stale Updates: Data-Free Knowledge Distillation for Asynchronous Federated Learning

To tackle update staleness and optimization instability in asynchronous federated learning (AFL), FedRevive extracts predictive dark knowledge from stale client updates via server-side data-free knowledge distillation (DFKD) and fuses parameter-space updates with multi-teacher distilled updates through an adaptive staleness-dependent schedule, achieving up to 29.7% faster wall-clock convergence and 11.4 percentage points higher accuracy without raw data or extra client communication.

Robust and Efficient Finetuning of Vision Foundation Models via Implicit Ensembling

GMixout reinterprets Mixout as a single-run weight-sharing implicit ensemble, introducing an evolving exponential moving average (EMA) anchor, a controlled resampling period, and sparse CUDA kernels to achieve state-of-the-art out-of-distribution robustness with zero inference overhead and only 2.6GB training VRAM.

RotaryQuant: Fitting 120B MoE Models on Consumer Hardware via Fused Compressed-Space Attention

RotaryQuant introduces a three-axis compression system (role-based weight quantization, LRU expert offloading, and IsoQuant 3-bit KV cache compression using WHT plus SO(4) block rotations) paired with a four-kernel fused Metal pipeline that computes attention directly on packed 3-bit data without tensor materialization, fitting a 120B MoE model into 32GB consumer unified memory with interactive throughput.

Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models

Addressing the excessive GPU memory residency of Mixture-of-Experts (MoE) models and the prohibitive cost of full fine-tuning for pruning, this paper uses lightweight adaptation (training as few as 0.002% parameters via router-only LoRA) as a diagnostic instrument to identify prunable experts via router norm shifts, achieving quasi-linear accuracy decay and cutting memory by 49% under 50% expert removal.

RPFQ-ViT: Rotated Phase-Frame Quantization for Extremely Low-Bit Weights in Vision Transformers

RPFQ-ViT addresses the severe directional distortion of Vision Transformer projections under extremely low-bit (e.g., 2-bit) weights by reparameterizing even-dimensional channels into 2D phase planes with group-wise learnable rotation and phase anchors, achieving 79.33% Top-1 on ImageNet-1K (W2/A4) while preserving the standard real-valued computation graph and speeding up on-device mobile inference.

SAFE-SVD: Sensitivity-Aware Fidelity-Enforcing SVD for Physics Foundation Models

Addressing the vulnerability where low-rank compression destroys higher-order derivatives and physical conservation laws in physics foundation models, this paper proposes SAFE-SVD, the first physics-aware SVD compression framework that models global sensitivity in Sobolev space and combines sequential error compensation with greedy rank allocation to preserve physical structures even under an aggressive 0.2 parameter retention ratio.

Scalable Distributed Stochastic Optimization via Bidirectional Compression: Beyond Pessimistic Limits

Addressing the pessimistic lower bound in centralized distributed stochastic optimization where downlink communication latency fails to scale with the number of workers, this paper introduces a structural \((L_A, L_B)\)-smoothness assumption reflecting the local rate of Hessian variation and proposes Inkheart SGD and M4, achieving the first state-of-the-art time complexities that scale with \(n\) under non-negligible bidirectional communication costs.

Scalable Variational Bayesian Fine-Tuning of LLMs via Orthogonalized Low-Rank Adapters

Addressing both rank collapse in standard LoRA that degrades distance-aware feature geometry and the heavy multi-pass backbone Monte Carlo sampling cost of existing Bayesian fine-tuning methods, this paper proposes PoLAR-VBLL—a unified framework combining Stiefel-constrained polar adapters with a closed-form Jensen-tightened variational Bayesian last layer and optional Laplace refinement, achieving state-of-the-art predictive accuracy and calibration under a single backbone forward pass.

ScaleBITS: Scalable Bitwidth Search for Hardware-Aligned Mixed-Precision LLMs

ScaleBITS clusters sensitive weights into hardware-aligned computation blocks via bi-directional channel reordering and introduces a scalable batched greedy search algorithm guided by first-order gradients computed at quantized reference points, achieving high-performing 2–3 bit mixed-precision LLM compression without runtime latency overhead.

SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation

SDS-LoRA identifies that non-uniform singular values in standard LoRA cause anisotropic gradient scaling that collapses the effective gradient rank, and resolves this by structurally decoupling singular values from the backward pass via orthonormal basis projection, achieving worst-case optimal gradient approximation and condition-number-independent convergence.

SEAD: Competence-Aware On-Policy Distillation via Entropy-Guided Supervision

Addressing the fundamental vulnerability in on-policy distillation where teacher supervision quality degrades on student incoherence, SEAD leverages joint teacher-student entropy as a unified probe across token, temporal, and prompt scales, skipping ~50% of redundant tokens while significantly outperforming vanilla OPD by +4.8 points across mathematical reasoning benchmarks.

SHRED: Retain-Set-Free Unlearning via Self-Distillation with Logit Demotion

Addressing the critical dependency of LLM machine unlearning on curated retain sets to prevent capability degradation, SHRED selectively demotes the logits of high-surprisal tokens in forget sequences while employing low-surprisal tokens as an intrinsic anchor within a unified top-K self-distillation objective.

Signed-Permutation Coordinate Transport for RMSNorm Transformers

Reveals that RMSNorm transformer residual streams possess a signed-permutation \(B_d\) discrete gauge symmetry rather than merely the permutation group \(S_d\), and proposes sign-marginalized Hungarian matching alongside parallel transport along training trajectories to enable high-fidelity cross-checkpoint transfer of sparse autoencoders, steering vectors, LoRA adapters, and optimizer states.

SliMOO: Interpretable Multi-Objective Evolutionary Search for LLM Depth Pruning

SliMOO addresses the severe task-dependent functional heterogeneity across LLM transformer layers by establishing an interpretable single-layer removal "task-layer importance map" as a search prior and performing NSGA-II-guided multi-objective evolutionary search, achieving Pareto-superior trade-offs across language, math, code, and knowledge benchmarks under fixed pruning budgets.

Stabilizing the Dynamic Low-Rank Training

SDLRT retains the leading singular directions discarded in the previous iteration and combines compensated basis augmentation with negative feedback on truncation tolerance to mitigate rank collapse under aggressive compression, achieving 85.15% average accuracy on six SuperGLUE validation tasks.

StableVQ: Practical Guidelines for Stable Vector-Quantized Tokenizer Training

StableVQ identifies that the entanglement between encoder-decoder and codebook optimization causes severe training instability in discrete visual tokenizers, and resolves it through Dynamic STE, Region VQ Loss, and Decoupled Schedule—achieving 100% codebook utilization and superior reconstruction quality with a parameter-free single linear projection.

Structural Rationale Distillation via Reasoning Space Compression

Addressing the rationale divergence and noisy supervision when distilling reasoning from large language models into smaller ones, D-RPC constrains the teacher to follow a compact, dynamically maintained bank of canonical reasoning paths and balances bank capacity and coverage via PAC-Bayes generalization bounds, significantly outperforming freeform and template-heavy distillation baselines across five math and commonsense benchmarks.

Structured Transforms for Low-Overhead Quantization of Language Models

This paper proposes an alternating greedy Kashin decomposition algorithm using sign-randomized Discrete Cosine Transforms (DCT) that factorizes LLM weights into two bounded infinity-norm components with guaranteed four-peak distributions in \(\mathcal{O}(N \log N)\) complexity, eliminating multi-restart \(k\)-means via closed-form centroid initialization while offering outstanding numerical stability and favorable wall-clock scaling.

Subliminal Learning as Trait-Direction Drift: A Mechanism and Targeted Control under SFT Distillation

This paper uncovers the micro-dynamic mechanism of subliminal learning during model distillation—showing that subtle log-likelihood preference gaps in semantically innocuous sequences accumulate into trait-direction representation drift under SFT—and designs probe-space corridor regularization to curb malicious transfer from 29.5% to 6.4% while preserving task accuracy.

Subliminal Learning Is Steering Vector Distillation

This paper demonstrates that subliminal learning in large language models is fundamentally an instance of steering vector distillation, showing that adaptive optimizers install the teacher's latent residual-stream steering direction into student parameters by suppressing outlier gradients.

Subliminal Transfer of Unsafe Behaviors in AI Agent Distillation

This paper provides the first empirical evidence that unsafe behavioral traits (such as destructive deletion and command-preference biases) can transfer subliminally during AI agent distillation, even when training trajectories are restricted to safe tasks and all explicit keywords and actions are rigorously sanitized.

TAPIOCA: Why Task-Aware Pruning Improves OOD Model Capability

Reveals that inference-time greedy layer pruning (TALE) yields no gain on in-distribution (ID) data but consistently boosts out-of-distribution (OOD) accuracy by eliminating layers acting as low-rank, high-gain amplifiers, thereby realigning distorted OOD residual representations with the task-adapted geometry.

The Geometry of Linear Program Compression: An Exact Characterization and Learning Algorithm

This paper uncovers the exact geometric foundation for lossless linear program (LP) compression in repeated solving, proposing an adaptive affine dimensionality reduction algorithm based on sample compression theory with a fast generalization certificate of \(1 - \widetilde{O}(d^{\star}/n)\) without requiring nondegeneracy.

The Hidden Power of Scaling Factor in LoRA Optimization

Revealing the fundamental optimization asymmetry between the scaling factor \(\alpha\) and learning rate \(\eta\) in LoRA, this paper introduces the LoRA-\(\alpha\) protocol driven by a square-root scaling law to restore suppressed curvature, allowing LoRA to directly reuse standard small learning rates from full fine-tuning with consistent cross-task gains.

The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes

This paper investigates the failure and success modes of on-policy distillation (OPD) and self-distillation (OPSD), identifies three core failure mechanisms—student prefix distortion, biased TopK reverse-KL gradients, and instance-specific PI marginalization—and proposes practical stabilizers including a stop-gradient TopK surrogate, RLVR-adapted teachers, and SFT student warm-up.

The Quantization Benefits of Residual-Free Transformers

This paper demonstrates that residual connections inherently amplify activation non-Gaussianity and outlier channels, and proposes a practical training recipe—combining orthogonal initialization, spectral second-order optimization, and depth-aware attention temperature scaling—that enables residual-free transformers to preserve near-Gaussian activations and achieve superior robustness under aggressive low-bit uniform quantization.

The risk of KV cache compression

This paper formalizes KV cache compression as sparse balancing of a probability measure in Hilbert space, proves tight minimax risk bounds governed by the response covariance spectrum, establishes a fundamental separation between query-aware and query-agnostic compression, and develops an efficient causal prefix compression scheme for prefill and autoregressive decoding.

ToPA: Block-wise Toeplitz Adaptation for Expressive and Efficient Fine-Tuning

Addressing the expressiveness bottlenecks of existing reparameterization-based PEFT methods caused by low-rank or fixed sparsity constraints, ToPA proposes a block-wise Toeplitz adaptation framework with an identity residual design, achieving dense and high-rank parameter adaptation while maintaining compact parameter budgets and FFT-accelerated efficiency.

TopoPrune: Robust Data Pruning via Unified Latent Space Topology

To overcome the "geometric brittleness" of traditional data pruning relying on extrinsic distances, TopoPrune couples global topological manifold projection with local differentiable multiparameter persistent homology optimization, creating a unified dual-scale importance score that delivers state-of-the-art accuracy and minimal variance under 90% extreme pruning, severe representation noise, and cross-architecture transfer.

Training Transformers for KV Cache Compressibility

Addressing the fundamental limitation that post-hoc KV cache compression is constrained by the frozen internal representations of pretrained models, this work formalizes KV compressibility, proves it is representation-dependent, and proposes KV-Compression Aware Training (KV-CAT) to incentivize compressible representations via dynamic train-time sparsification and self-distillation.

Training with Harnesses: On-Policy Harness Self-Distillation for Complex Reasoning

Introduces On-Policy Harness Self-Distillation (OPHSD), internalizing the procedural reasoning and verification signals of external inference harnesses into model weights via reverse-KL distillation, eliminating the need for complex runtime scaffolds while achieving superior standalone reasoning performance.

Transcoder Adapters for Reasoning-Model Diffing

To explain how post-training changes reasoning model internals, this paper introduces Transcoder Adapters that learn sparse approximations of MLP computation differences between base and fine-tuned models, achieving faithful rollouts at ultra-low sparsity (L0 0.1–10) and uncovering that hesitation behaviors can be ablated to cut response length by over 50% with virtually no accuracy penalty.

TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks

Addressing the accuracy collapse of vulnerable tokens under single-axis compression in long-context agentic workloads, TriAxialKV introduces a joint temporal-modal-semantic mixed-precision quantization scheme, expanding KV cache capacity by 4.5x and boosting end-to-end throughput by up to 1.52x on Qwen3-VL without compromising BF16 task accuracy.

TriSpec: Ternary Speculative Decoding via Lightweight Proxy Verification

TriSpec proposes a ternary speculative decoding paradigm that coordinates a lightweight single-layer drafter, a same-family proxy verifier, and the target model, using the proxy's top-1/top-2 probability margin to dynamically filter verification loads, reducing target model calls by over half and delivering up to 35% speedup with minimal accuracy loss.

Truth as a Compression Artifact in Language Model Training

Demonstrates that language models trained on contradictory corpora prefer correct answers not due to an inherent alignment with reality, but because next-token cross-entropy minimization seeks compact descriptions; when error systems possess internally coherent rules, this correctness preference completely vanishes.

Tunable Latent Generative Priors for Compressed Sensing

Addressing the rigid representation complexity of standard generative priors that leads to noise overfitting in inverse problems, this work trains tunable generative priors via nested dropout across VAEs, normalizing flows, and latent diffusion models, enabling inference-time latent dimension truncation \(k\) with proven optimal complexity and consistent gains over full-capacity baselines.

Tunable Latent Generative Priors for Compressed Sensing and Inverse Problems

By incorporating nested dropout into latent diffusion models, normalizing flows, and VAEs, this work creates tunable latent priors with ordered hierarchical representations, allowing dynamic inference-time latent dimension matching to measurement counts and noise levels without retraining.

UtoMe: Observation-Uncertainty-Guided Token Merging for Weather Foundation Models

Addressing the vulnerability of feature-similarity-only token merging to observation errors in reanalysis inputs, UtoMe integrates atmospheric dynamics saliency with ensemble data assimilation spread to steer anchor protection and uncertainty-calibrated bipartite token merging.

Visual Text Compression as Measure Transport

This paper formulates visual text compression (VTC) as a push-forward map within measure transport theory, decomposing encoding distortion into label-free precision and coverage costs to enable reliable visual/text path routing and transport-guided foveation.

WASD: Wasserstein-based Knowledge Distillation for Large Language Models

WASD introduces a debiased Sinkhorn divergence framework for autoregressive LLM distillation that leverages teacher token embedding geometry as transport cost and employs dual potential differences as stop-gradient loss weights for efficient, semantic-aware distribution alignment without auxiliary networks.

Your Teacher Can’t Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation

To tackle Supervision Fidelity Decay (SFD)—where teacher corrective signals collapse on long student rollouts—this paper introduces Lookahead Group Reward (LGR) and entropy-triggered tree attention to actively maintain supervision quality in on-policy distillation.

Patch Rebirth: Fast and Transferable Model Inversion of Vision Transformers

Addressing the prohibitive self-attention computational cost of Vision Transformer (ViT) model inversion and the severe information loss caused by pruning premature patches, this paper introduces Patch Rebirth Inversion (PRI), which progressively isolates informative patches from a single inversion trajectory and allows the remaining patches to accumulate rich semantics via the newly uncovered "Re-Birth effect," achieving up to 10× speedup over dense inversion and 2× over sparse inversion without sacrificing transferability.