SARATHI: Decoupled Data-Free Probing and Adaptive Reconstruction for Structured LLM Pruning¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/snkrbhr/SARATHI
Area: Model Compression
Keywords: Structured Pruning, Data-Free Probing, Non-Negative Matrix Factorization, Optimal Brain Surgeon, Bias Shift Compensation
TL;DR¶
Addressing calibration domain bias and memory wall collapses caused by coupling neuron selection and weight reconstruction in post-training structured pruning, SARATHI completely decouples data-free NMF residual subspace probing from localized \(O(d^2)\) peak-memory Adaptive OBS reconstruction, while introducing closed-form bias shift compensation to eliminate perplexity collapse in post-LayerNorm architectures.
Background & Motivation¶
Large language models deliver remarkable empirical capabilities across diverse natural language tasks, yet their billions of parameters impose formidable memory footprints and latency bottlenecks during deployment. Unlike unstructured pruning methods (e.g., SparseGPT, Wanda) that produce sparse weight tensors requiring specialized sparse kernels or dedicated hardware (such as NVIDIA 2:4 Tensor Cores), structured pruning permanently eliminates entire rows or columns from weight matrices. This yields truly smaller dense matrices that deliver hardware-agnostic, deterministic speedups on standard BLAS hardware without any customized kernel dependencies. In modern Transformer architectures, the feed-forward network (FFN) intermediate sub-layers serve as the primary leverage point for structured compression: FFN blocks contain twice as many parameters as attention sub-layers in Multi-Head Attention (MHA) models, and between 4.2× and 4.7× more parameters in Grouped-Query Attention (GQA) architectures. Moreover, non-linear activations (e.g., ReLU, SwiGLU) reside exclusively within FFN layers, allowing intermediate neuron pruning to simultaneously slash both parameter volume and activation compute overheads.
However, existing state-of-the-art post-training structured pruning frameworks—including SoBP, SliceGPT, and FLAP—suffer from an inherent architectural flaw: they couple neuron selection and weight reconstruction by using a shared calibration dataset \(\mathcal{D}_{\text{calib}}\) for both operations. In computational systems, this coupling induces severe memory bottlenecks. Evaluating and maintaining the layer-wise empirical Hessian matrices \(H = X_{\text{calib}}X_{\text{calib}}^\top \in \mathbb{R}^{d \times d}\), their inverses, Cholesky factorization matrices, intermediate activation tensors, and global OBS compensation buffers across \(L\) layers concurrently incurs \(O(Ld^2)\) peak GPU memory. At aggressive compression rates (such as 40% sparsity), this memory explosion frequently precipitates fatal out-of-memory (OOM) failures on standard enterprise GPUs. Statistically, driving neuron selection via a shared calibration batch imposes severe domain bias, over-prioritizing neurons that fire on calibration tokens while systematically discarding crucial feature pathways essential for open-domain generalization.
Resolving this tension requires disentangling neuron importance discovery from compensatory weight reconstruction, establishing distinct information flows for each stage. The intrinsic low-rank geometry of pre-trained weight matrices intrinsically encapsulates a non-negative consensus manifold, allowing structurally irreplaceable outlier neurons to be identified purely through data-free matrix factorization without consulting external data. Concurrently, compensatory weight updates can be confined to small, layer-wise local batches that preserve functional dynamics in \(O(d^2)\) peak memory, while analytically counteracting systematic mean shifts in post-LayerNorm architectures. Core idea: completely decouple structured LLM pruning into data-free NMF residual subspace probing and localized Adaptive OBS weight reconstruction, utilizing non-negative matrix factorization linked to Robust PCA to score structurally irreplaceable neurons while rectifying post-LayerNorm drift via closed-form bias compensation without retraining.
Method¶
Overall Architecture¶
The SARATHI framework reformulates the coupled combinatorial pruning objective into two sequentially independent subproblems with disjoint information sources: Subproblem P1 performs data-free structural importance scoring based purely on the intrinsic weight matrix geometry, and Subproblem P2 executes localized weight compensation using layer-isolated activation samples. In the first phase, each MLP layer's weight magnitude matrix is decomposed via non-negative matrix factorization (NMF) to isolate the low-rank consensus subspace from sparse geometric outliers, yielding data-free neuron saliency scores aligned across gated projections and thresholded via Median Absolute Deviation (MAD). In the second phase, the binary pruning mask is frozen, and each layer independently constructs a localized Hessian from 128 samples to perform closed-form column-wise Adaptive OBS updates in \(O(d^2)\) peak memory. For post-LayerNorm networks, activation mean drift is analytically absorbed into the layer bias vector before physically slicing the weight tensors into compact dense matrices.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Dense MLP Weight Matrix<br/>W ∈ ℝ^(N×d)"] --> B["NMF Residual Subspace Probing<br/>Data-free outlier extraction S = |W| - VH"]
B --> C["SwiGLU Gate-Value Coupling<br/>Arithmetic mean across projection scores"]
C --> D["Robust MAD Budget Allocation<br/>Global threshold τ with min-keep floor α"]
D --> E["Localized Adaptive OBS Reconstruction<br/>O(d²) layer-isolated closed-form compensation"]
E --> F["Closed-Form Bias Shift Compensation<br/>Analytic correction of cascading mean shift Δμ"]
F --> G["Physical Dimension Slicing<br/>Compact dense hardware-accelerated model"]
Key Designs¶
1. NMF Residual Subspace Probing: Eliminating Calibration Domain Bias via Robust PCA Bounds Addressing the calibration bias and memory overhead of data-dependent scoring, SARATHI leverages the intrinsic low-rank geometry of pre-trained MLP matrices. Well-trained projection matrices satisfy \(\text{rank}_\epsilon(W^{(\ell)}) \ll \min(N, d)\), indicating that their row space is closely approximated by a low-dimensional subspace \(V\) (\(\dim(V) = r \ll d\)). Neurons residing inside this consensus subspace can be eliminated with bounded functional perturbation, whereas neurons outside \(V\) represent structurally irreplaceable features. To strictly respect the non-negative activation regimes of ReLU and SwiGLU FFNs, SARATHI applies Non-Negative Matrix Factorization to the element-wise absolute weight matrix \(|W^{(\ell)}| \in \mathbb{R}_{\ge 0}^{N \times d}\): $\(\min_{V \ge 0, H \ge 0} \| |W^{(\ell)}| - VH \|_F^2\)$ where basis matrix \(V \in \mathbb{R}_{\ge 0}^{N \times r}\) and coefficient matrix \(H \in \mathbb{R}_{\ge 0}^{r \times d}\). The resulting residual matrix \(S = |W^{(\ell)}| - VH\) establishes a direct formal equivalence to the Robust PCA decomposition (\(|W| = VH + S\)), where high-magnitude entries in \(S\) capture the sparse outlier subspace that escapes low-rank consensus. Defining the element-wise indicator mask \(M_{i,j} = \mathbb{I}(S_{i,j} \ge \tau)\), the geometric importance score for neuron \(i\) is formulated as: $\(\eta_i = \sum_{j=1}^d \left( |W_{i,j}^{(\ell)}| \cdot M_{i,j} \right)\)$ Under Proposition 1, for any neuron with \(\eta_i = 0\), all row elements reside within the NMF subspace up to \(\tau\), guaranteeing that its physical removal followed by Adaptive OBS reconstruction incurs an output reconstruction error strictly bounded by \(\mathcal{O}(\tau \sqrt{d} \|X\|_F)\). Conversely, neurons with \(\eta_j > 0\) contain components orthogonal to the span of the NMF basis, rendering their elimination structurally irreversible.
2. Gated Projection Coupling and MAD Budgeting: Mitigating Multiplicative Vanishing and Heavy Tails Modern open-weight LLMs predominantly employ gated FFNs (such as SwiGLU in LLaMA and Mistral architectures), where each intermediate hidden unit is coordinated across three distinct parameter matrices: gate projection \(W_{\text{gate}}\), up projection \(W_{\text{up}}\), and down projection \(W_{\text{down}}\). Conventional multiplicative score coupling suffers from numerical vanishing: if an otherwise vital neuron exhibits near-zero magnitude in just one projection matrix, multiplicative combination prematurely zeroes out the overall score and forces harmful pruning. SARATHI adopts a balanced arithmetic mean across projections: $\(\tilde{\eta}_i = \frac{1}{3} \left( \eta_i^{\text{gate}} + \eta_i^{\text{up}} + \eta_i^{\text{down}} \right)\)$ Furthermore, determining cross-layer pruning thresholds using empirical mean and standard deviation fails due to the heavy-tailed distribution of weight outliers in LLMs. A minute fraction of extreme outlier neurons drastically inflates the global mean, artificially elevating the threshold and causing catastrophic over-pruning in early layers. SARATHI employs Median Absolute Deviation (MAD), achieving the maximal theoretical breakdown point of 0.5: $\(\text{MAD}(\eta) = \text{median}(|\eta - \text{median}(\eta)|)\)$ The global pruning threshold is derived as \(\tau = \text{median}(\eta) - k \cdot \text{MAD}(\eta)\), where scalar \(k\) is determined via binary search to strictly match target global sparsity \(\rho\). An adaptive min-keep floor \(\alpha = 0.5\) enforces \(\|s^{(\ell)}\|_0 \ge \lceil \alpha N_\ell \rceil\) per layer, completely preventing representational bottleneck collapses.
3. Localized Adaptive OBS Reconstruction and Closed-Form Bias Compensation: Resolving the Memory Wall and Post-LN Drift Standard Optimal Brain Surgeon (OBS) formulations jointly utilize inverse Hessian matrices to perform both combinatorial pruning search and parameter compensation, entailing massive concurrent memory buffers. SARATHI decouples search from compensation: by freezing the binary mask \(s^*\) derived from the data-free NMF probe, OBS simplifies into a localized, fixed-constraint quadratic minimization problem. Utilizing a localized batch \(X_{\text{local}}\) of merely 128 samples, the layer-isolated regularized empirical Hessian is formed as \(H_{\text{obs}, j} = X_{\text{local}}X_{\text{local}}^\top + \lambda I \in \mathbb{R}^{d \times d}\). The closed-form optimal compensatory weight adjustment is derived per column: $\(\delta w_{ij} = -\frac{w_{ij}}{\left[ H_{\text{obs}, j}^{-1} \right]_{jj}} \left[ H_{\text{obs}, j}^{-1} \right]_{j, :}\)$ This localized operation restricts peak working memory to \(O(d^2)\) per layer—an \(L\)-fold reduction compared to coupled frameworks that eliminates OOM crashes. Moreover, in post-LayerNorm architectures (such as the OPT family), removing \(K\) neurons induces an empirical mean shift in pre-LayerNorm activations: \(\Delta \mu^{(\ell)} = \mathbb{E}[f(W^{(\ell)}x)] - \mathbb{E}[f(W_{s^*}^{(\ell)}x)]\). This shift cascades through subsequent scale-and-shift operations across all \(L-\ell\) remaining layers, scaling cumulative activation error as \(\mathcal{O}(K \cdot (L-\ell))\) and inducing perplexity explosion. SARATHI computes this shift analytically from \(X_{\text{local}}\) and performs an exact zero-overhead bias compensation: $\(\hat{b}^{(\ell)} = b^{(\ell)} + \Delta \mu^{(\ell)}\)$ This closed-form vector update stabilizes activation distributions across the network without any additional forward or backward passes.
Benchmark & Evaluation Protocol¶
Evaluation encompasses six language models across two primary architectural paradigms: gated FFNs (LLaMA-3-8B, Mistral-7B, LLaMA-2-7B, and LLaMA-2-13B) and non-gated ReLU FFNs (OPT-2.7B, OPT-6.7B), spanning model scales from 2.7B to 13B parameters. Evaluated baselines comprise leading post-training structured pruning methods: SoBP, SliceGPT, FLAP, Dynamic Slicing, and the calibration-free offline RL approach PruneNet. Two internal variants isolate component contributions: SARATHI-NMF (data-free geometric probe) and SARATHI-Wanda (activation-norm probe). Evaluation metrics include WikiText-2 perplexity (PPL) at sequence length 2048, zero-shot accuracy across seven commonsense reasoning benchmarks (PIQA, HellaSwag, ARC-Easy, ARC-Challenge, WinoGrande, BoolQ, OpenBookQA) via lm-eval-harness, inference throughput (tokens/second) on a single NVIDIA A100-40GB GPU at batch sizes 1 and 16, end-to-end pruning latency, and Neural Tangent Kernel (NTK) Cosine Similarity to monitor preservation of local optimization dynamics.
Key Experimental Results¶
Main Results¶
Across all evaluated architectures and sparsity levels (15% to 40%), SARATHI achieves competitive or superior language modeling perplexity while maintaining high zero-shot accuracy. Crucially, under aggressive pruning (40% sparsity), coupled methods fail due to out-of-memory errors or numerical instability, whereas SARATHI completes compression robustly.
Table 1: WikiText-2 Perplexity (PPL ↓) across 15%–40% Sparsity (Sequence Length 2048, Zero Retraining)
| Model Architecture & Size | Pruning Method | 15% Sparsity | 20% Sparsity | 30% Sparsity | 40% Sparsity |
|---|---|---|---|---|---|
| LLaMA-3-8B (Dense: 6.14) | SliceGPT (Ashkboos et al., 2024) | — | — | 13.37 | 20.69 |
| Dyn. Slicing (Dumitru et al., 2024) | — | — | 12.96 | 19.11 | |
| SARATHI-Wanda | 7.78 | 8.18 | 9.04 | 10.10 | |
| SARATHI-NMF (Ours) | 7.28 | 7.70 | 8.70 | 9.92 | |
| Mistral-7B (Dense: 5.32) | SliceGPT (Ashkboos et al., 2024) | — | — | 8.87 | 13.33 |
| Dyn. Slicing (Dumitru et al., 2024) | — | — | 8.76 | 12.62 | |
| SARATHI-Wanda | 6.74 | 7.04 | 7.74 | 8.59 | |
| SARATHI-NMF (Ours) | 6.45 | 6.71 | 7.41 | 8.38 | |
| OPT-2.7B (Dense: 12.47) | FLAP (An et al., 2024) | — | 15.38 | 18.32 | Failed (†) |
| SliceGPT (Ashkboos et al., 2024) | 13.28 | 13.89 | 16.31 | Failed (†) | |
| SoBP (Wei et al., 2024) | 12.55 | 12.72 | 13.89 | Failed (†) | |
| SARATHI-Wanda | 12.74 | 13.48 | 15.89 | 18.72 | |
| SARATHI-NMF (Ours) | 12.52 | 13.25 | 15.61 | 18.98 | |
| OPT-6.7B (Dense: 10.86) | FLAP (An et al., 2024) | — | 12.80 | 15.42 | Failed (†) |
| SliceGPT (Ashkboos et al., 2024) | 11.26 | 11.60 | 12.80 | Failed (†) | |
| SoBP (Wei et al., 2024) | 11.02 | 11.32 | 12.17 | OOM | |
| SARATHI-Wanda | 11.35 | 12.06 | 14.02 | OOM | |
| SARATHI-NMF (Ours) | 11.29 | 11.95 | 13.99 | 16.52 | |
| LLaMA-2-13B (Dense: 4.88) | SoBP (Wei et al., 2024) | 5.38 | 5.62 | 6.27 | 7.39 |
| SARATHI-Wanda | — | 6.36 | 7.12 | 8.25 | |
| SARATHI-NMF (Ours) | — | 5.42 | 5.85 | 6.40 |
On downstream reasoning benchmarks at 30% structured sparsity, SARATHI variants consistently outperform SliceGPT and Dynamic Slicing, matching or exceeding SoBP across non-gated architectures while remaining fully data-free for neuron scoring.
Table 2: Zero-Shot Accuracy (% ↑) across 7 Commonsense Reasoning Tasks and WikiText-2 PPL at 30% Sparsity
| Evaluation Model | Pruning Method | PIQA | HellaSwag | ARC-c | ARC-e | WinoGrande | BoolQ | OBQA | Average | PPL (↓) |
|---|---|---|---|---|---|---|---|---|---|---|
| OPT-2.7B | Dense Baseline | 74.81 | 60.58 | 31.14 | 54.42 | 61.01 | 63.20 | 35.40 | 54.37 | 12.47 |
| FLAP | 72.14 | 54.82 | 29.27 | 48.86 | 57.46 | 61.28 | 33.20 | 51.00 | 18.32 | |
| SliceGPT | 72.85 | 55.43 | 30.12 | 49.75 | 58.09 | 59.42 | 33.60 | 51.32 | 16.31 | |
| SoBP | 72.69 | 56.51 | 30.80 | 50.80 | 58.88 | 56.09 | 34.40 | 51.45 | 13.89 | |
| SARATHI-Wanda | 72.96 | 55.94 | 28.67 | 53.32 | 58.17 | 65.11 | 33.00 | 52.45 | 15.89 | |
| SARATHI-NMF | 73.23 | 56.22 | 28.92 | 52.90 | 57.77 | 65.08 | 34.40 | 52.65 | 15.61 | |
| LLaMA-3-8B | Dense Baseline | 79.82 | 78.79 | 49.66 | 77.86 | 73.16 | 75.81 | 44.40 | 68.50 | 6.14 |
| SliceGPT | 59.30 | 37.20 | — | 42.90 | 56.40 | — | — | — | 13.37 | |
| Dyn. Slicing | 60.40 | 38.40 | — | 42.40 | 58.00 | — | — | — | 12.96 | |
| SARATHI-Wanda | 71.71 | 64.94 | 36.77 | 63.72 | 70.72 | 63.30 | 38.00 | 58.45 | 9.04 | |
| SARATHI-NMF | 70.08 | 58.08 | 30.89 | 53.66 | 66.61 | 62.54 | 38.80 | 54.38 | 8.70 | |
| LLaMA-2-13B | Dense Baseline | 80.52 | 79.37 | 49.23 | 77.48 | 72.30 | 80.58 | 45.20 | 69.24 | 4.88 |
| SoBP | 76.50 | 74.55 | 47.78 | 74.45 | 71.82 | 79.45 | 43.20 | 66.82 | 6.27 | |
| SARATHI-Wanda | 74.92 | 70.21 | 41.04 | 65.82 | 65.04 | 66.21 | 41.80 | 60.72 | 7.12 | |
| SARATHI-NMF | 74.27 | 69.30 | 40.61 | 67.30 | 70.40 | 72.87 | 43.60 | 62.62 | 5.85 |
Ablation Study¶
Ablation experiments systematically isolate the impact of Bias Shift Compensation, the factorization objective (NMF vs. truncated SVD), and layer allocation strategies.
Table 3: Impact of Bias Shift Compensation on OPT-2.7B WikiText-2 Perplexity
| Configuration | 20% Sparsity PPL | 30% Sparsity PPL | 40% Sparsity PPL | Theoretical Insight |
|---|---|---|---|---|
| Without Bias Comp. | 47.06 | 132.00 | 2,082.00 | Mean shift cascades exponentially through deep post-LN layers |
| With Bias Comp. (SARATHI) | 13.25 | 15.61 | 18.98 | Analytic zero-overhead correction eliminates distribution drift |
| PPL Error Reduction | 72.0% | 88.0% | 99.1% | Benefit scales non-linearly with increasing pruning sparsity |
Table 4: Probe-Only Factorization Ablation on OPT-6.7B at 30% Sparsity (Excluding Adaptive OBS and Bias Comp.)
| Structural Diagnostic Metric | NMF Probe | Truncated SVD Probe | Analytical Takeaway |
|---|---|---|---|
| Spearman Rank Correlation (\(\rho\)) | 0.9987 | 0.9987 | Macroscopic ranking across neurons is virtually identical |
| Top-25% Neuron Set Overlap | 98.95% | 98.95% | Discrepancy resides in only ~1% of critical boundary neurons |
| Relative Matrix Reconstruction Error | 0.6103 | 0.6078 | SVD yields marginally lower unconstrained Frobenius norm error |
| Downstream WikiText-2 PPL (Probe-Only) | 19,695 | 28,268 | SVD degrades PPL by >8,500 points, proving non-negativity is crucial |
Key Findings¶
- Elimination of the High-Sparsity Memory Wall: At 40% sparsity on OPT-6.7B, all coupled structured pruning baselines (SoBP, SliceGPT, FLAP) crash due to out-of-memory errors caused by concurrent global Hessian and Cholesky buffer allocations. By decoupling the pipeline and bounding peak memory to \(O(d^2)\) per layer, SARATHI-NMF executes without memory failure, achieving a successful 16.52 PPL.
- Perplexity vs. Zero-Shot Representation Trade-Off: On gated models (LLaMA-3-8B and Mistral-7B), data-free SARATHI-NMF attains lower language modeling perplexity (8.70 vs. 9.04 on LLaMA-3 at 30%), whereas activation-driven SARATHI-Wanda yields higher zero-shot commonsense accuracy (58.45% vs. 54.38%). This demonstrates that data-free geometric probing preserves the broader language manifold, while calibration activations slightly bias subnets toward commonsense patterns.
- Robustness to Factorization Rank: In sensitivity sweeps over NMF rank \(r \in \{7, 16, 32, 64\}\), perplexity varies by at most 0.28 points across all ranks on LLaMA-3-8B and Mistral-7B, establishing \(r=16\) as an exceptionally robust, model-agnostic default.
- Hardware Speedups with Invariant Local Dynamics: On an A100 GPU, physical FFN slicing yields up to 2.05× throughput improvement on OPT-6.7B (BS=16, 40% sparsity) and 1.66× on LLaMA-3-8B. Neural Tangent Kernel (NTK) Cosine Similarity remains strictly 1.0000 across all models and sparsity levels, confirming that Adaptive OBS perfectly preserves functional gradient dynamics.
Highlights & Insights¶
- Theoretically Grounded Data-Free Saliency: Reformulates data-free neuron selection through NMF residual subspace probing with a formal connection to Robust PCA, establishing the first rigorous error bound \(\mathcal{O}(\tau \sqrt{d} \|X\|_F)\) for data-free structured LLM pruning.
- Zero-Overhead Analytic Bias Compensation: Unveils the mechanism of cascading activation mean shift in post-LayerNorm architectures after structural row removal, correcting it via an exact closed-form vector update \(\hat{b}^{(\ell)} = b^{(\ell)} + \Delta \mu^{(\ell)}\) that reduces 40% sparsity perplexity on OPT-2.7B by 99.1%.
- General BLAS Compatibility with Rapid One-Shot Execution: Avoids the custom kernel requirements of unstructured sparsity by producing physical dense weight matrices; compresses models from 2.7B to 13B parameters in 16 to 44 minutes on a single GPU without any fine-tuning.
Limitations & Future Work¶
- Reduced Saliency Discrimination on Non-Gated Architectures: On non-gated ReLU architectures such as OPT, weight rows exhibit near-uniform residual norms, diminishing the probe's discriminative power; on these models, performance is dominated by Bias Compensation and Adaptive OBS rather than NMF ranking.
- Tensor-Core Alignment Overhead at Specific Batch Sizes: Slicing FFN dimensions to non-multiples of 64 can cause hardware alignment penalties. On Mistral-7B at BS=16, structured removal incurs a 0.82–0.92× throughput regression relative to dense baselines, indicating that future work should incorporate hardware alignment constraints into MAD budgeting.
- Scope Confined to Intermediate FFN Neurons: The current framework targets FFN intermediate dimensions; extending the NMF residual subspace formulation to attention heads and GQA key-value groups represents a natural next step.
Related Work & Insights¶
- vs. Unstructured Pruning (SparseGPT / Wanda / FISTAPruner): Unstructured approaches maintain original tensor dimensions with fine-grained element sparsity, delivering practical acceleration only under specialized hardware support (such as 2:4 sparse kernels); SARATHI physically removes entire rows, enabling immediate speedups on standard BLAS engines.
- vs. Coupled Structured Pruning (SoBP / SliceGPT / FLAP): Prior structured methods couple neuron scoring and weight compensation through a shared calibration dataset, creating an \(O(Ld^2)\) global memory bottleneck and calibration domain bias; SARATHI decouples the pipeline into data-free geometric scoring and localized \(O(d^2)\) reconstruction, eliminating OOM failures at 40% sparsity.
- vs. Calibration-Free Policy Pruning (PruneNet): While PruneNet operates calibration-free at inference, it requires pre-training a separate 45M-parameter reinforcement learning policy offline; SARATHI employs closed-form NMF decomposition directly on weights without any prior training phase.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates data-free structured pruning via NMF residual probing with Robust PCA bounds, decoupled localized reconstruction, and analytic bias compensation.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 6 models from two architectural families, 15%–40% sparsity, PPL, 7 zero-shot benchmarks, NTK diagnostics, and hardware throughput.
- Writing Quality: ⭐⭐⭐⭐⭐ Mathematically rigorous, clearly motivated, and structured with clean narrative progression and incisive ablations.
- Value: ⭐⭐⭐⭐⭐ Provides a scalable, OOM-resistant, and data-free blueprint for physical LLM compression, with profound implications for privacy-constrained and memory-limited deployments.