Beyond Retraining-Free MoE Compression: A Cost-Normalized Study of Post-Compression Adjustment¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://aidaslab.github.io/Post-Compression-Adjustment/
Area: Model Compression
Keywords: Mixture-of-Experts, MoE Compression, Expert Pruning & Merging, Post-Compression Adjustment, Knowledge Distillation
TL;DR¶
This paper revisits the prevailing paradigm of retraining-free MoE compression by arguing that compressed checkpoints are compressed initializations rather than final deployable artifacts; across 28 benchmarks and measured GPU budgets, a single-epoch full-parameter causal LM fine-tuning with only 3,000 examples recovers 37.3% of the original-to-compressed degradation gap.
Background & Motivation¶
Mixture-of-Experts (MoE) architectures achieve high capacity with low per-token inference FLOPs by selectively activating only a small subset of experts per forward pass. However, standard inference engines must still reside all expert weights in GPU memory, which imposes massive memory footprints and creates critical hardware bottlenecks in resource-constrained environments. To resolve this challenge, recent literature has placed intense focus on retraining-free or calibration-free MoE compression—specifically expert pruning and expert merging. By evaluating expert importance or similarity over tiny calibration splits, these methods remove redundant experts or fuse expert weights into shared physical clusters, outputting a compressed checkpoint without executing full-scale model retraining.
Nonetheless, this paper identifies a fundamental conceptual blind spot in prior research: treating retraining-free compression as a zero-cost, one-shot procedure whose output is the final deployable artifact. In practice, calculating expert importance, routing statistics, and clustering affinities still consumes substantial GPU memory and compute cycles during calibration inference. More importantly, pruning or merging experts irreversibly alters the underlying expert manifold, causing significant residual errors such as routing probability shift, expert functional distortion, and router-expert interaction degradation. Given that modern compression pipelines already permit small data subsets and non-trivial GPU compute, allocating all available resources strictly to the compression phase while leaving downstream residual deviations unaddressed is fundamentally suboptimal.
To overcome this structural limitation, the authors reframe compressed MoE checkpoints as compressed initializations that naturally require a short post-compression recovery stage. The core idea is to reframe retraining-free MoE compression as a two-stage paradigm of "compressed initialization plus tiny-budget post-compression adjustment," demonstrating under a rigorous cost-normalized GPU protocol that full-parameter causal language modeling fine-tuning significantly outperforms teacher-based knowledge distillation and expert-subset updates in repairing residual errors across deep network layers.
Method¶
Overall Architecture¶
The proposed framework establishes a cost-normalized benchmarking protocol to dissect MoE compression error sources and evaluate post-compression adjustment (PCA) strategies across multiple dimensions. The pipeline consists of three systematic stages: First, representative retraining-free compressors (pruning algorithms REAP and AIMER; merging algorithms HC-SMoE and M-SMoE) are applied to instruction-tuned MoE backbones (Qwen3-30B-A3B and Gemma4-26B-A4B) across 50%, 62.5%, and 75% retention ratios to construct diverse compressed checkpoints. Second, an adjustment stage is executed using a minimal budget of 3,000 C4 general-domain text examples over a single epoch (effective batch size 8), comparing causal LM fine-tuning against token-level teacher KD and varying trainable parameter scopes. Third, both downstream task accuracy across 28 benchmarks and hardware costs (effective GPU-hours, time-integrated GPU memory, and electrical energy) are monitored end-to-end on two NVIDIA H200 GPUs.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Original Dense/MoE Checkpoints<br/>(Qwen3-30B / Gemma4-26B)"] --> B["Stage 1: Compression Residual Modeling & Decomposition<br/>(Separating Shared-Path Reweighting & Active-Path Replacement)"]
B --> C["Stage 2: Post-Compression Adjustment Objectives<br/>(Causal LM Likelihood vs Teacher-Guided Token-Level KD)"]
C --> D["Stage 3: Trainable Scope & Cost Normalization<br/>(Full Parameter vs Router-Only vs Top-k Expert Selection)"]
D --> E["Recovered Deployable MoE Artifacts<br/>(28 Downstream Tasks & Measured GPU-Cost Recovery)"]
Key Designs¶
1. Compression Residual Modeling & Decomposition: Isolating the mechanisms of downstream error propagation Addressing the lack of formal error analysis in post-compression recovery, the authors decompose the single-layer MoE discrepancy into orthogonal components. Let \(S\) denote the original top-\(k\) active expert indices, \(P\) the retained expert set, and \(S'\) the active expert set selected by the compressed layer for the same reference hidden input \(x\). Defining \(T = S \cap S'\) as the preserved overlap, \(D = S \setminus S'\) as the dropped experts, and \(R = S' \setminus S\) as the newly active replacements, the local residual vector is rigorously expressed as: $\(y_{\text{orig}}(x) - y_{\text{pruned}}(x) = e_{\text{route}}^p(x) + e_{\text{path}}^p(x)\)$ where the routing reweighting error \(e_{\text{route}}^p = \sum_{i \in T} (\tilde{g}_{o,S}^i - \tilde{g}_{p,S'}^i) E_i\) and the active-path replacement error \(e_{\text{path}}^p = \sum_{i \in D} \tilde{g}_{o,S}^i E_i - \sum_{i \in R} \tilde{g}_{p,S'}^i E_i\). Empirical measurement on ELI5 shows that the ideal scenario (\(S'=S\)) occurs in only 8%–32% of token-layer events, while partial overlap (\(0 < |T| < k\)) overwhelmingly dominates (67%–92%). Unrolling the first-order error recurrence \(\delta_L \approx \sum_{\ell=0}^{L-1} P_{\ell \to L} \epsilon_\ell\) demonstrates that local MoE perturbations inevitably propagate into subsequent attention, layer normalization, and dense feed-forward blocks, proving that updating routers alone is fundamentally insufficient.
2. Post-Compression Adjustment Objectives: Disentangling causal LM likelihood from teacher-guided knowledge distillation Targeting the question of whether imitation of the original model is the optimal recovery signal, this work directly contrasts causal language modeling (LM) fine-tuning against token-level teacher knowledge distillation (KD). For a sequence \(x = (x_1, \dots, x_L)\), causal LM optimizes the negative log-likelihood of observed text: $\(\mathcal{L}_{\text{LM}} = -\frac{1}{N_x} \sum_{t=1}^{L-1} m_{t+1} \log p_{\theta_c}(x_{t+1} \mid x_{\le t})\)$ In contrast, KD introduces the uncompressed base model as an active teacher, computing symmetric KL divergence over softened vocabulary logits \(\mathcal{L}_{\text{KD}} = \frac{\tau^2}{N_x} \sum_{t=1}^{L-1} m_{t+1} D_{\text{KL}}(p_T^{(t)} \parallel p_S^{(t)})\) with \(\tau=1.0\). Because compression distorts the student's internal expert manifold, forcing the student to match the teacher's probability distribution proves to be a less effective local recovery direction than directly maximizing next-token likelihood on natural text. Furthermore, KD requires executing forward passes through the full-sized teacher, nearly doubling inference latency and memory occupancy during adjustment.
3. Trainable Scope & Cost Normalization: Quantifying the trade-offs of full versus subset parameter updates To test whether selectively fine-tuning a small subset of parameters saves end-to-end computation, the authors evaluate four parameter scopes: Router-only, Router + Top-\(k\) Experts (\(k \in \{8, 16, 50\}\) identified by activation frequency on calibration data), Router + All-Experts (freezing dense non-expert layers), and Full-parameter adjustment. Crucially, the protocol records a comprehensive cost tuple: effective GPU-hours \(H_{\text{eff}} = G \cdot T_{\text{hr}} \cdot \frac{\bar{u}}{100}\), time-integrated GPU memory \(H_{\text{mem}} = T_{\text{hr}} \cdot \bar{M}_{\text{obs}}\), and electrical energy \(E_{\text{GPU}} = \frac{T_{\text{hr}}}{1000} \sum_{g=1}^G \bar{P}_g\). Unlike prior works that disregard calibration overhead, the measurement boundaries explicitly encompass both the pre-training expert-selection inference pass and the training loop. The empirical analysis reveals that Top-\(k\) strategies incur substantial selection inference costs that negate their backpropagation parameter savings, making Full FT the superior choice in both recovery magnitude and end-to-end efficiency.
Loss & Training¶
All post-compression adjustment runs follow a standardized setup: 3,000 text examples sampled from C4 tokenized to 512 tokens with seed 42; per-device batch size 2 with 4 gradient accumulation steps (effective batch size 8) for exactly 1 epoch; AdamW optimizer with learning rate \(5 \times 10^{-5}\), weight decay 0.0, and linear decay with a 3% warmup schedule; global gradient clipping norm 1.0; bfloat16 precision without gradient checkpointing; two NVIDIA H200 GPUs (140GB each). Evaluation is conducted via lm-evaluation-harness and vLLM using greedy decoding (temperature 0) with thinking mode disabled.
Key Experimental Results¶
Main Results¶
The main benchmark macro-averages results across 12 compressor-retention settings (2 backbones × 4 compression algorithms × 3 retention ratios) over 28 downstream tasks spanning General Reasoning, Mathematics, Coding, Chain-of-Thought (CoT), and Multiple-Choice QA (MCQA).
| Adjustment Strategy | 28-Task Score | Recovery Gain \(\Delta\) (pp) | Gap Recovered (%) | Energy \(E_{\text{GPU}}\) (kWh) | Eff. GPU-h \(H_{\text{eff}}\) (h) | Memory \(H_{\text{mem}}\) (GiB-h) |
|---|---|---|---|---|---|---|
| Original Uncompressed | 0.4837 | — | — | — | — | — |
| Compressed Only (Unadjusted) | 0.3901 | +0.00 | 0.0% | — | — | — |
| Full FT (Causal LM) | 0.4250 | +3.49 | 37.3% | 0.5767 | 0.6225 | 273.43 |
| Router All-Expert FT | 0.4131 | +2.31 | 24.6% | 0.5767 | 0.6115 | 249.15 |
| Router Top-50 FT | 0.4098 | +1.97 | 21.0% | 0.7351 | 0.7228 | 217.28 |
| Router Top-16 FT | 0.4035 | +1.35 | 14.4% | 0.6939 | 0.7053 | 159.43 |
| Router Top-8 FT | 0.4021 | +1.21 | 12.9% | 0.6650 | 0.6946 | 146.56 |
| Router-only FT | 0.3923 | +0.23 | 2.4% | 0.3378 | 0.2640 | 61.36 |
| Full KD (Distillation) | 0.4119 | +2.19 | 23.4% | 0.7868 | 0.7644 | 516.92 |
| Router All-Expert KD | 0.4052 | +1.51 | 16.2% | 0.7670 | 0.7574 | 484.34 |
| Router Top-50 KD | 0.4036 | +1.35 | 14.5% | 0.8989 | 0.8676 | 422.49 |
| Router Top-16 KD | 0.3993 | +0.92 | 9.9% | 0.8762 | 0.8475 | 330.21 |
| Router Top-8 KD | 0.3986 | +0.85 | 9.1% | 0.8869 | 0.8487 | 311.75 |
| Router-only KD | 0.3918 | +0.17 | 1.9% | 0.5317 | 0.4066 | 207.52 |
| Direct Router Logit Match (DRLM) | 0.3907 | +0.06 | 0.7% | 0.5218 | 0.3999 | 187.98 |
Ablation Study¶
The following table reports the benchmark-family breakdown for Full FT, along with budget reduction to 1,024 examples and domain shift robustness on OpenR1-Math-220k:
| Benchmark Family / Configuration | Original (Orig) | Compressed (Comp) | Full FT Adjusted | Absolute Gain \(\Delta\) (pp) | Core Finding |
|---|---|---|---|---|---|
| Chain-of-Thought (6 tasks) | 0.5889 | 0.4499 | 0.5406 | +9.07 | Major restoration of multi-step fluency and coherent reasoning |
| Multiple Choice (8 tasks) | 0.5296 | 0.4716 | 0.5149 | +4.33 | Strong recovery of knowledge retrieval and option disambiguation |
| General Reasoning (4 tasks) | 0.4510 | 0.4351 | 0.4503 | +1.52 | Nearly full recovery back to original uncompressed performance |
| Mathematical Reasoning (6 tasks) | 0.4123 | 0.2894 | 0.2972 | +0.78 | Marginal change; domain-specific syntax requires specialized data |
| Coding (4 tasks) | 0.3737 | 0.2430 | 0.2381 | -0.49 | Natural language C4 cannot repair specialized programming syntax |
| 1,024 C4 Budget (Macro Mean) | 0.4840 | 0.3900 | 0.4180 | +2.78 | Core rank ordering remains identical despite 66% data reduction |
| 1,024 OpenR1-Math Budget | 0.4840 | 0.3670 | 0.4130 | +4.58 | Cross-domain robustness with Spearman correlation \(\rho=0.973\) |
Key Findings¶
- Full-parameter fine-tuning yields superior Pareto efficiency: Full FT achieves the largest performance recovery (+3.49 pp, 37.3% gap recovered) while requiring lower electrical energy (0.5767 kWh) and fewer effective GPU-hours (0.6225 h) than Router Top-50 FT (0.7351 kWh, 0.7228 h). Subsetting experts introduces separate calibration inference that counteracts the parameter reduction benefits.
- Causal LM fine-tuning consistently outperforms knowledge distillation: Across all matched parameter scopes, causal LM fine-tuning beats token-level teacher KD. For example, Router Top-8 FT (+1.21 pp) outperforms Router Top-16 KD (+0.92 pp). Moreover, KD incurs a teacher forward pass that doubles memory residency and increases energy usage by ~36%.
- Degeneration in natural language generation is the primary failure mode: Post-compression performance collapse is heavily driven by repetitive token generation, premature termination, and syntax breakdown. A single epoch on 3,000 natural text examples restores generation stability, driving a dramatic +9.07 pp jump in CoT reasoning.
Highlights & Insights¶
- Reconceptualizing the compressed MoE artifact: By identifying compressed checkpoints as compressed initializations rather than terminal products, this paper corrects the unrealistic industry assumption that retraining-free compression is free of runtime or accuracy costs.
- Rigorous end-to-end cost accounting: By integrating expert-selection inference and teacher execution into the measured GPU cost tuple, the study eliminates the illusory efficiency advantage often attributed to selective parameter freezing.
- Propagation of MoE noise beyond sparse layers: First-order Taylor linearization analytically proves that sparse expert errors inevitably propagate into shared dense and attention projections, explaining why updating all model parameters is fundamentally necessary.
Limitations & Future Work¶
- Absence of system-level distributed training optimizations: Experiments were measured using a baseline PyTorch implementation; incorporating vLLM prefill acceleration, FSDP/DeepSpeed sharding, or fused Triton MoE kernels may alter fine-grained execution trade-offs.
- Scale restricted to sub-100B parameter regimes: Evaluations were performed on 30B-scale models (Qwen3 and Gemma4); whether models at the 100B+ scale display different routing redundancy and expert retention dynamics remains open.
- Safety and alignment considerations: Diagnostic behavioral testing indicates that post-compression adjustment restores both desirable reasoning capabilities and latent hazardous knowledge (such as elevated WMDP scores), highlighting the need for safety-conditioned adjustment objectives in future work.
Related Work & Insights¶
- vs Pure Retraining-Free Compressors (EAC-MoE, NAEE, REAP): While previous works focus exclusively on designing complex heuristics for one-shot pruning or merging, this paper demonstrates that single-shot compression hits an intrinsic performance ceiling that can be easily surpassed by an ultra-cheap adjustment phase.
- vs Method-Specific Recovery Schemes (MoE-Pruner, CD-MoE): Unlike prior works that tightly bind recovery training to specific compression algorithms, this work treats post-compression adjustment as an independent architectural axis, establishing that general causal LM loss is more cost-effective than standard teacher KD.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ Shifts the prevailing MoE compression paradigm from one-shot output to a principled two-stage framework.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluates 2 backbones, 4 compressors, 3 retention ratios, and 28 tasks with rigorous three-metric GPU cost profiling.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative, tight theoretical error decomposition, and seamless integration of analytical derivations with empirical findings.
- Value: ⭐⭐⭐⭐⭐ Provides a highly practical, reproducible, and cost-effective recipe (Full FT on 3,000 samples) for deploying compressed MoE models.