Skip to content

E^3-Pruner: Towards Efficient, Economical, and Effective Layer Pruning for Large Language Models

Conference: EMNLP 2026
Paper: ACL Anthology
Area: Multimodal VLM
Keywords: Large Language Models, Layer Pruning, Gumbel-TopK, Adaptive Knowledge Distillation, Structured Pruning

TL;DR

Addressing the trilemma of severe performance degradation, prohibitive search costs, and limited practical acceleration in structured pruning, this paper presents E^3-Prunerโ€”a two-stage framework that couples differentiable Gumbel-TopK search with progressive scheduling and token-entropy-weighted adaptive distillation, achieving state-of-the-art capability recovery under a tiny 0.5B token budget.

Background & Motivation

As large language models (LLMs) scale up to tens or hundreds of billions of parameters following scaling laws, substantial computational demands and memory footprints have become major bottlenecks for practical hardware deployment. In response, diverse model compression techniques have been extensively explored. Among existing paradigms, unstructured pruning retains high nominal accuracy via weight-level sparsity, but its irregular access patterns rarely translate to real-world latency speedups on standard hardware accelerators. Structured pruning across attention heads or MLP intermediate channels (e.g., LLM-Pruner, Sheared-LLaMA) offers improved hardware affinity, yet irregular dimensional shapes still restrict achievable speedups at the operator level. In contrast, layer pruningโ€”which excises entire Transformer blocksโ€”maintains uniform tensor architectures and yields deterministic wall-time acceleration proportional to the fraction of layers removed, making it an attractive compression paradigm.

Nonetheless, existing layer pruning methodologies face three fundamental, conflicting deployment hurdles. First, training-free calibration methods such as ShortGPT prioritize layer removal by measuring activation cosine similarity, but cause catastrophic, unrecoverable capability degradation at higher pruning ratios. Second, search-based schemes relying on neural architecture search (NAS) or evolutionary algorithms (such as DarwinLM) evaluate extensive candidate subnetworks through iterative training, consuming billions of tokens and incurring prohibitive computational costs. Third, differentiable soft-mask schemes regularized by continuous sparsity penalties fail to enforce exact target pruning rates and introduce up to \(5\times\) slower search overhead. Practical production environments urgently call for a paradigm that simultaneously satisfies task effectiveness, training economy, and inference efficiency.

This paper tackles these challenges by reformulating discrete layer selection as a continuous, differentiable optimization problem and smoothing structural transitions via progressive scheduling, coupled with an information-sensitive recovery protocol. The core idea is to employ a continuous Gumbel-TopK sampler with straight-through estimation for differentiable layer mask search under a progressive pruning schedule, followed by entropy-aware adaptive knowledge distillation with pre-cached teacher logits to restore complex reasoning capabilities under minimal training overhead.

Method

Overall Architecture

E^3-Pruner operates in two decoupled stages: the mask searching stage and the fine-tuning stage. In the searching stage, rather than directly optimizing discrete binary masks, the method assigns a learnable continuous importance score to each layer, initialized via output perturbation KL divergence. During forward passes, a differentiable Gumbel-TopK sampler continuously relaxes the discrete top-\(k\) operator, enabling gradient backpropagation through the Straight-Through Estimator (STE) while a progressive curriculum schedule smoothly scales the pruning ratio from 0 to the target. In the fine-tuning stage, the optimal discrete pruning mask is frozen, and an adaptive knowledge distillation objective is applied. By pre-storing top-\(K\) teacher logits offline, the framework completely sidesteps the GPU memory bottleneck of concurrent teacher-student model loading and re-weights tokens according to predictive entropy.

The overall architecture and pipeline are illustrated below:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    In["Original Pretrained / Aligned LLM"] --> Init["Layer Importance Initialization<br/>KL divergence under single-layer removal"]
    Init --> Search["Gumbel-TopK Differentiable Mask Search<br/>Continuous relaxation + temperature annealing + progressive schedule"]
    Search --> FixedMask["Frozen Discrete Binary Layer Mask<br/>Physically bypass removed Transformer blocks"]
    FixedMask --> Distill["Adaptive Knowledge Distillation Recovery<br/>Pre-cached Teacher Top-K Logits + Token-Entropy Reweighting"]
    Distill --> Out["Compact Accelerated Layer-Pruned Model"]

Key Designs

1. Layer Importance Initialization: Informative Priors for Stable Differentiable Search

Initializing layer importance scores randomly or from zeros leads to severe optimization instability across the discrete combinatorial search space, substantially inflating the token count required for convergence. To preserve economy, E^3-Pruner introduces an efficient calibration-based prior on a minimal set of calibration samples (e.g., 40 samples). Specifically, for the \(l\)-th layer, the initial importance score \(s_l\) is defined as the empirical Kullback-Leibler (KL) divergence between the output distribution of the original model \(\Theta\) and that of the perturbed model \(\Theta_{-l}\) with layer \(l\) removed:

\[s_l = \mathbb{E}_{x \sim \mathcal{C}} \left[ \mathcal{D}_{\text{KL}}\left( f(x; \Theta_{-l}) \parallel f(x; \Theta) \right) \right]\]

This initialization entails negligible computational cost, yet immediately separates critical layers (such as early feature extraction and late decision layers) from redundant middle layers, preventing early exploratory gradients from destabilizing foundational structures.

2. Gumbel-TopK Differentiable Mask Search: Exact and Efficient Mask Optimization

To overcome the non-differentiability of discrete hard \(\text{TopK}\) selection while avoiding the loose sparsity bounds of continuous relaxation, E^3-Pruner adopts a Gumbel-TopK continuous relaxation. Standard Gumbel noise \(g \sim \text{Gumbel}(0, 1)\) is injected into the layer importance vector \(S_1 = S + g\). Over \(k\) iterative selection steps, a soft multi-hot mask variable \(\tilde{M}\) is updated via temperature-scaled softmax, decaying selected probability mass at each iteration:

\[\tilde{M}_{i+1} = \tilde{M}_i + \sigma(S_i / \tau), \quad S_{i+1} = S_i + \log\left(1 - \sigma(S_i / \tau)\right)\]

As the temperature \(\tau \to 0\), the soft mask \(\tilde{M}_k\) weakly converges to the discrete \(\text{TopK}(S, k)\) selection. A linear annealing schedule \(\tau = 1 - \beta \cdot (t / T)\) governs temperature decay. To enforce layer skipping in the forward pass while permitting gradient backpropagation, the straight-through estimator (STE) is applied by binarizing the mask during forward evaluation and passing gradients unchanged during backward passes.

To prevent sudden internal representation collapse caused by immediate layer drop, a progressive curriculum schedule linearly reduces the number of retained layers \(k'\) from the total layer count \(L\) to target \(k\) over iterations \(t \in [1, T]\):

\[k' = L - \left\lceil (L - k) \cdot \frac{t}{T} \right\rceil\]

This progressive schedule allows retained layers to smoothly adapt to upstream structural shifts. Furthermore, the search stage consumes merely the first 10% of the overall training budget (e.g., 120 steps), ensuring remarkable training economy.

3. Adaptive Knowledge Distillation: Memory-Efficient and Entropy-Aware Recovery

Once the binary mask is identified, the model architecture is fixed for capability recovery. Supervised fine-tuning (SFT) often suffers from task overfitting and degrades behavioral alignment with the original model, whereas standard knowledge distillation (KD) encounters severe memory bottlenecks when co-locating both teacher and student models on device. E^3-Pruner resolves this by pre-computing and caching the teacher's top-\(K\) logits offline. At a sequence length of 32K, storing top-10 logits incurs only 2.6 MB of GPU memory, bypassing dual-model residency constraints.

To maximize distillation efficiency, the framework recognizes that tokens differ dramatically in informational importance. Tokens with high predictive entropy typically represent critical reasoning junctures, branching logic, or uncertain semantic transitions. The adaptive KD objective weights the token-level KL divergence by the entropy \(H\) of the teacher's predictive distribution:

\[\mathcal{L} = \frac{1}{N} \sum_{i=1}^N H\left(\sigma(z_t^{(i)})\right) \cdot \sum_{c \in S_K^{(i)}} \sigma(z_t^{(i)})_c \log\left( \frac{\sigma(z_t^{(i)})_c}{\sigma(z_s^{(i)})_c} \right)\]

where \(S_K^{(i)}\) denotes the top-\(K\) indices of the teacher distribution for token \(i\). This dynamic re-weighting concentrates model capacity on high-information reasoning steps, enabling the student model to recover general reasoning and math proficiency with as few as 0.5B tokens (roughly 0.5% of standard post-training volume).

Key Experimental Results

Main Results

The framework is evaluated across diverse model families and scales, including Qwen2.5-14B-Instruct, Qwen3-32B, the ultra-large MoE DeepSeek-R1 (671B), LLaMA-2-7B, and openPangu-Ultra-MoE-718B.

The main results on Qwen2.5-14B-Instruct under 50% layer pruning (removing 24 out of 48 layers) are summarized below:

Method Param. SciQ PIQA WG ArcE ArcC(25) HS(10) LogiQA BoolQ MMLU(5) Avg โ†‘ Speedup โ†‘ Token Budget โ†“
Original Model 14.8B 94.0 78.1 69.0 76.3 54.2 78.7 30.4 77.7 45.9 69.8 1.00ร— -
w/o KD fine-tuning
ShortGPT 8.2B 31.6 56.7 50.3 30.5 27.6 34.0 24.9 53.8 24.6 37.1 1.63ร— -
Shortened LLaMA 8.2B 59.8 53.9 51.4 34.4 22.6 26.1 25.5 45.5 25.6 38.3 1.63ร— -
SLEB 8.2B 75.0 64.2 51.5 50.9 24.9 42.1 25.5 58.4 24.6 46.3 1.63ร— -
E^3-Pruner (Stage 1 search) 8.2B 88.3 72.0 54.5 69.2 40.9 57.0 26.9 62.0 28.4 55.5 1.63ร— -
w/ KD fine-tuning
LLM-Streamline 8.2B 89.2 73.5 52.6 68.7 36.0 55.9 25.5 59.9 26.9 54.2 1.63ร— 0.5B
Sheared-LLaMA 8.2B 93.4 74.8 60.4 71.3 44.4 64.1 26.0 65.9 32.1 59.2 1.63ร— 0.5B
Shortened LLaMA 8.2B 91.0 75.6 59.6 74.7 47.5 64.8 27.3 60.8 31.6 59.2 1.63ร— 0.5B
DarwinLM 8.4B 84.3 73.8 59.0 75.8 49.1 53.6 28.9 67.2 42.8 59.4 1.43ร— 1.0B
SLEB 8.2B 93.0 76.8 61.3 75.4 47.8 66.6 28.6 65.1 33.7 60.9 1.63ร— 0.5B
Minitron 8.2B 91.9 73.5 62.0 72.1 44.9 64.5 28.4 70.4 32.8 60.1 1.63ร— 0.5B
E^3-Pruner (Stage 2 full) 8.2B 93.7 76.9 63.0 76.0 47.9 67.2 30.0 66.5 36.5 61.9 1.63ร— 0.5B

The results on reasoning-centric foundation models Qwen3-32B and DeepSeek-R1 (671B) are presented below:

Model & Config Param. Pruning Ratio MATH-500 AIME'24 AIME'25 GPQA-Diam. LiveCode MMLU-Pro ArenaHard Avg โ†‘ Speedup โ†‘
Qwen3-32B Original 32.8B 0% 96.8 79.6 71.3 64.7 66.9 76.5 93.5 78.5 1.00ร—
Qwen3-32B (Minitron) 28.9B 12.5% (1/8 pruned) 95.4 72.1 64.2 61.6 62.9 74.8 92.3 74.8 1.13ร—
Qwen3-32B (E^3-Pruner) 28.9B 12.5% (1/8 pruned) 96.2 76.3 65.8 61.1 65.1 74.5 91.4 75.8 1.13ร—
Qwen3-32B (Minitron) 25.0B 25.0% (1/4 pruned) 95.0 68.3 61.7 59.1 57.7 70.5 85.0 71.0 1.33ร—
Qwen3-32B (E^3-Pruner) 25.0B 25.0% (1/4 pruned) 96.0 71.7 64.2 60.1 60.3 70.1 87.3 72.8 1.33ร—
DeepSeek-R1 Original 671B 0% 97.2 77.9 67.1 73.7 65.4 82.8 96.3 80.1 1.00ร—
DeepSeek-R1 (Minitron) 601B 8 layers (13.1%) 97.0 75.0 62.9 71.2 65.1 81.8 95.0 78.3 1.13ร—
DeepSeek-R1 (E^3-Pruner) 601B 8 layers (13.1%) 97.0 76.3 60.8 73.2 67.3 81.2 96.2 78.9 1.13ร—
DeepSeek-R1 (Minitron) 509B 16 layers (26.2%) 95.6 68.8 55.4 69.2 59.2 80.2 89.8 74.0 1.33ร—
DeepSeek-R1 (E^3-Pruner) 509B 16 layers (26.2%) 95.2 65.8 60.0 69.7 59.9 79.8 90.0 74.3 1.33ร—

Ablation Study

Component-Wise Contribution and Distillation Variants (Qwen2.5-14B-Instruct, 50% pruned):

Configuration / Stage SciQ PIQA WG ArcE ArcC(25) HS(10) LogiQA BoolQ MMLU(5) Avg โ†‘ Note
Original Unpruned Model 94.0 78.1 69.0 76.3 54.2 78.7 30.4 77.7 45.9 69.8 upper bound reference
Heuristic Layer Pruning (ShortGPT) 31.6 56.7 50.3 30.5 27.6 34.0 24.9 53.8 24.6 37.1 unsearched & unrecovered
+ KL Layer Importance Initialization 72.5 63.8 51.0 58.2 25.1 43.5 25.1 58.0 25.2 47.0 provides high-quality search prior
+ Gumbel-TopK Search (Stage 1) 88.3 72.0 54.5 69.2 40.9 57.0 26.9 62.0 28.4 55.5 +18.4% without any fine-tuning
+ Standard Offline KD 94.0 76.2 62.5 75.2 48.0 65.6 29.0 63.8 32.8 60.8 uniform token weight KL
+ Adaptive KD (Stage 2 full) 93.7 76.9 63.0 76.0 47.9 67.2 30.0 66.5 35.8 61.9 token entropy weighting gains +1.1%

Ablation of Mask Searching Budget (Fixed Total Budget of 0.5B Tokens):

Search Budget Ratio SciQ PIQA WG ArcE ArcC(25) HS(10) LogiQA BoolQ MMLU(5) Avg โ†‘ Note
5% (60 steps) 92.0 76.6 62.8 75.6 48.6 67.0 28.0 66.5 29.0 60.7 search is slightly under-converged
10% (120 steps) 93.7 76.9 63.0 76.0 47.9 67.2 30.0 66.5 35.8 61.9 default optimal operating point
20% (240 steps) 91.9 75.8 59.9 75.9 49.7 67.8 28.6 66.4 37.1 61.5 sufficient search but fewer recovery steps
50% (600 steps) 91.0 75.9 59.7 73.7 48.3 65.3 28.3 67.0 34.3 60.4 reduced recovery steps hurts accuracy

Key Findings

  • Pruning Mask Topology Governs Recovery Potential: Without any distillation fine-tuning, E^3-Pruner's Stage 1 mask achieves 55.5% average accuracy on Qwen2.5-14B-Instruct, surpassing fully fine-tuned LLM-Streamline (54.2%). This highlights that locating structurally optimal layers is far more critical than extensive post-hoc retraining.
  • Single-Shot Search Beats 4-Iteration Refinement: In multi-stage iterative comparisons, a single iteration of E^3-Pruner achieves 61.9% average accuracy, outperforming the 4-iteration Minitron baseline (61.4%) while eliminating repetitive training overhead.
  • Distillation Protects Behavioral Consistency: Supervised fine-tuning (SFT) causes behavioral divergence on MMLU/GSM8K/IFeval/HumanEval test samples, with consistency plunging from 72 to below 44 under 1/8 layer pruning due to overfitting. In contrast, adaptive KD maintains steady alignment, steadily improving consistency beyond 70.
  • Resilience on Mathematical Reasoning: When pruning 25% of layers from Qwen3-32B, E^3-Pruner preserves 96.0% accuracy on MATH-500 (against 96.8% unpruned, only a 0.8% degradation), outperforming competing SOTA (95.0%) with a deterministic 1.33ร— inference speedup.

Highlights & Insights

  • Differentiable Relaxation of Combinatorial Architecture Search: By marrying Gumbel-TopK sampling with the straight-through estimator, the framework converts discrete combinatorial layer selection into a tractable gradient-based optimization task, sidestepping costly evolutionary loops.
  • Entropy-Aware Token Prioritization: Recognizing that reasoning uncertainty is non-uniformly distributed across generation sequences, weighting token distillation by predictive entropy effectively directs student model capacity toward critical decision boundaries.
  • Exceptional Hardware and Training Practicality: Pre-caching top-10 teacher logits reduces distillation memory to 2.6 MB per 32K context. Combined with LoRA and Megatron tensor/expert parallelism, a massive 671B MoE model (DeepSeek-R1) can be pruned and recovered within 24 hours on 64 NPUs.

Limitations & Future Work

  • Scope Confined to Depth Pruning: The current formulation focuses on layer-level pruning. While the Gumbel-TopK mechanism is robust, its extension to width pruning, head pruning, or MoE expert pruning requires addressing distinct optimization dynamics and sparsity structures.
  • Reliance on External Pre-training Corpus: Post-pruning recovery currently utilizes general off-the-shelf corpora (Fineweb-Edu, AM-DeepSeek-R1-Distilled). Investigating self-generated synthetic data aligned with the base model's internal representations remains unexplored.
  • Evaluation Limited on Multilingual and Low-Resource Tasks: Benchmarks predominantly cover English reasoning, math, and code; evaluating multilingual safety and low-resource language degradation under deep layer pruning remains an open avenue.
  • vs ShortGPT: ShortGPT relies on training-free activation similarity metrics; however, catastrophic performance drops occur at high sparsity. E^3-Pruner addresses this limitation through differentiable Gumbel-TopK search and adaptive KD, improving 50% pruning accuracy from 37.1% to 61.9%.
  • vs Sheared-LLaMA: Sheared-LLaMA combines soft masks with \(L_0\) regularization and deterministic top-\(k\) truncation, which cannot strictly guarantee exact target layer counts and slows training by up to \(5\times\). E^3-Pruner enforces exact layer budgets via progressive Gumbel-TopK schedules with minimal overhead.
  • vs Minitron & DarwinLM: Minitron requires costly multi-iteration pruning-distillation cycles, while DarwinLM incurs heavy token expenditures (~1.0B tokens) across evolutionary search. E^3-Pruner surpasses 4-iteration Minitron using a single-stage search and half the token budget (0.5B tokens).

Rating

  • Novelty: โญโญโญโญโ˜† [Elegant integration of Gumbel-TopK sampling and entropy-aware distillation for discrete LLM layer pruning]
  • Experimental Thoroughness: โญโญโญโญโญ [Extensive evaluations across 7B to 671B dense and MoE models with downstream reasoning, ablations, behavioral consistency, and real-world vLLM latency benchmarks]
  • Writing Quality: โญโญโญโญโญ [Clear structural formulation, rigorous mathematical definitions, and lucid empirical analyses]
  • Value: โญโญโญโญโญ [High practical significance for cost-effective enterprise deployment and acceleration of frontier LLMs]