Beyond FLOPs: Benchmarking Real Inference Acceleration of LLM Pruning under a GEMM-Centric Taxonomy¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/EIT-NLP/LLM-Pruning/tree/main/PruningInferSim
Area: LLM Reasoning
Keywords: LLM Pruning, Real Inference Acceleration, GEMM Taxonomy, Pareto Frontier, Empirical Latency Profiling
TL;DR¶
Addressing the disconnect where nominal FLOPs reduction fails to reflect physical GPU speedups in large language model pruning, this paper introduces a unified GEMM-centric taxonomy across M, N, and K logical dimensions and delivers an implementation-consistent benchmarking suite, charting the first empirical throughput-quality Pareto frontier and exposing the critical role of non-GEMM execution overheads.
Background & Motivation¶
Large language models have established transformative milestones across complex reasoning, automated coding, and scientific problem solving, yet their staggering computational requirements and memory consumption continue to impose crippling bottlenecks during practical serving. Pruning has emerged as an indispensable paradigm to compress these architectures, spanning a vast design spectrum that removes computation across depth layers, attention heads, weight matrix channels, semi-structured sub-blocks, or dynamically routed tokens. However, while virtually all proposed algorithms claim substantial reductions in nominal floating-point operations (FLOPs) alongside strong accuracy retention, they induce fundamentally disparate compute access patterns and memory layouts on modern parallel hardware.
Because capturing true end-to-end latency demands intricate, non-trivial low-level system engineering tailored to each distinct sparsity pattern, the literature has historically fallen back on theoretical proxies such as parameter counts, FLOPs, or isolated micro-kernel benchmarks. This evaluation fragmentation obscures the deployable acceleration benefits of different pruning families: fine-grained semi-structured or dynamic skipping patterns that appear efficient on paper frequently suffer from memory fragmentation, kernel launch overheads, or dynamic memory allocations that erase speedups entirely. Furthermore, prior studies evaluate against divergent baselines, heterogeneous hardware, and ad-hoc kernel implementations, leaving practitioners without a fair, unified standard to compare the actual throughput-quality trade-offs among competing pruning methodologies.
Overcoming this pervasive disconnect requires penetrating superficial structural labels and re-evaluating pruning mechanisms through the physical computational substrate of Transformer inference, where general matrix multiplication (GEMM) accounts for over 90% of total execution time. The core idea is to project the entire spectrum of LLM pruning methods onto the logical M, N, and K operational dimensions of GEMM and their inter-operator structural propagation patterns, building a hardware-agnostic benchmarking framework with unified operator-replacement interfaces to chart the true prefill and decode throughput-quality Pareto frontier under identical hardware environments.
Method¶
Overall Architecture¶
In standard Transformer single-step autoregressive inference, per-layer execution is dominated by nine core GEMM operators across multi-head attention (\(Q, K, V, O\) projections, plus \(QK^\top\) and \(PV\)) and feed-forward networks (\(\text{Up}, \text{Gate}, \text{Down}\) projections), collectively consuming approximately 90% of the runtime budget. For any standard GEMM formulated as \(O = AW^\top\) with input activation \(A \in \mathbb{R}^{T \times D}\), parameter matrix \(W \in \mathbb{R}^{I \times D}\), and output \(O \in \mathbb{R}^{T \times I}\), the physical dimensions naturally map to the three logical GEMM axes: token sequence length \(T\) corresponds to logical axis \(M\), output feature dimension \(I\) maps to logical axis \(N\), and input reduction dimension \(D\) corresponds to logical axis \(K\).
Leveraging this operational mapping, the paper establishes a clean four-family taxonomy: Static Depth Pruning (Static M), Static Semi-structured and Low-rank Compression (Static K), Static Width and Cross-layer Pruning (Static NK), and Dynamic Token/Attention Sparsity (Dynamic M / NK). The end-to-end benchmarking framework hooks into target models through a standardized operator-replacement interface, incorporates portable kernel implementations in high-level domain-specific languages (Triton and Tilelang), aligns downstream performance recovery via a standardized LoRA adaptation recipe, and benchmarks execution under CUDA Graph tracing to eliminate CPU dispatch jitter while measuring precise Time-to-First-Token (TTFT) and Time-Per-Output-Token (TPOT).
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Target Dense LLM & Sparsity Budget"] --> B["GEMM Logical Dimension Projection<br/>Classify pruning into logical axes M, N, and K"]
B --> C["Structural Propagation Wiring & Operator Wrapping<br/>Coordinate dependent layers and residual projections"]
C --> D["Hardware-Agnostic High-Performance DSL Pipeline<br/>Incorporate mask reordering and JIT cuSPARSELt"]
D --> E["End-to-End Inference Profiling & Frontier Mapping<br/>Benchmark Prefill & Decode throughput via CUDA Graph"]
E --> F["Deployable Speedup-Quality Pareto Envelope & Overhead Breakdown"]
Key Designs¶
1. GEMM Logical Dimension Projection: Dissecting Structural Propagation and Operational Invariance
To reconcile the ambiguity of macro-level labels like "depth" and "width," this design examines the mathematical manifestation of pruning masks across linear algebra operations. Under structured zeroing, pruning along the \(M\) axis operates on input rows (\(O = (AW^\top) \odot M\)), pruning along \(N\) masks output channels (\(O = (AW^\top) \odot M\) broadcasted across columns), whereas pruning along the \(K\) dimension operates on the inner reduction dimension (\(O = A(W \odot M)^\top\)). Crucially, because \(K\) is the summation reduction axis, it leaves no direct mask equivalent on the output tensor \(O\).
This algebraic distinction governs structural propagation across consecutive pipeline operations: \(M\)-dimension pruning (e.g., token skipping or layer dropping) persists unaltered throughout downstream layers; \(N\)-dimension pruning reduces output channels, which immediately acts as an input feature reduction (\(K\)-dimension pruning) in the subsequent dependent operator, establishing an intrinsic \(N \to K\) propagation pattern (Static NK); conversely, weight-level \(K\)-dimension pruning (such as semi-structured 2:4 sparsity or low-rank SVD decomposition) terminates locally without structural propagation. Grouping algorithms by these operational invariants decouples mathematical compression from hardware-specific implementations.
2. Structural Propagation Wiring & Operator Wrapping: Coarse-to-Fine Dependency Orchestration
To resolve dimension mismatches and residual connection fractures caused by propagating static modifications (Static M and Static NK), the framework introduces a unified structural wiring pipeline (Algorithm 1). The system processes architectural modifications in a top-down hierarchy: layer-wise dropping \(\to\) sublayer (Attention/FFN) removal \(\to\) fine-grained row-column shrinking. For Static M, dropped layers or sublayers are replaced with clean identity forward mappings. For Static NK with intra-layer dependencies, consecutive linear operators are clustered into atomic dependency groups and pruned jointly.
For cross-layer Static NK paradigms like SliceGPT—which alter dimensions from embedding through attention projection and all the way to the language model head—the wrapper automatically injects specialized residual projection layers to restore residual stream dimensionality while bypassing propagation through internal attention score matrices. Concurrently, Grouped-Query Attention (GQA) constraints are strictly respected by enforcing head-group granularity: key and value heads are only dropped when an entire corresponding group of query heads is eliminated, preventing dimensional inconsistencies.
3. Hardware-Agnostic High-Performance DSL Pipeline: Mask Reordering and Portable Kernel Integration
To eliminate severe CPU launch overheads and graph-breaking dynamic allocations in dynamic pruning, the framework develops optimized kernel primitives using Python-based DSLs (Triton and Tilelang). For Dynamic M, naive implementations rely on runtime mask-to-index conversions and explicit gather-scatter memory copies. The proposed pipeline introduces an online mask-reordering scheme: routing masks are sorted in descending order prior to kernel execution, clustering active tokens into contiguous blocks (Tiles). The GPU kernel then conducts a block-level reduction check: blocks containing zero active tokens trigger an immediate early exit, while active tiles undergo fused block-MMA matrix multiplication and epilogue scattering.
For semi-structured Static K, the framework bypasses native PyTorch overheads (which incur 726 \(\mu\)s per call) by utilizing SGLang's TVM-FFI JIT wrapper to directly interface with cuSPARSELt, slashing initialization latency by 94.4% to 40 \(\mu\)s alongside automated tiling configuration caching. For dynamic sparse attention (Dynamic NK), where thread-level shared memory scheduling is paramount, Tilelang is leveraged to overlap QK map estimation with sparse block attention loading, achieving superior throughput over standard portable baselines.
Benchmark & Evaluation Protocol¶
To guarantee an equitable quality comparison and eliminate confounding discrepancies from disparate post-pruning recovery recipes, all retrainable pruning candidates are calibrated and fine-tuned under a standardized LoRA adaptation protocol: - Unified LoRA Fine-Tuning: Standardized on RedPajama-Data-1T-Sample-subset850000, training for 5,000 steps (batch size 16, context length 2,048) with learning rate \(2 \times 10^{-4}\), LoRA rank 16, \(\alpha = 32\), and dropout 0.1. Dynamic M methods are trained for 10,000 steps to ensure full routing convergence. - Calibration and Zero-Shot Evaluation: Unretrainable semi-structured Static K (due to mathematical incompatibility with additive low-rank adapters) is evaluated zero-shot directly following calibration. Matrix factorization methods (Dobi-SVD) split training budgets equally between \(U\) and \(V\) matrices (2,500 steps each). Evaluation spans seven standard zero-shot benchmarks (ARC-c, ARC-e, BoolQ, WinoGrande, PIQA, OpenBookQA, HellaSwag) via lm-evaluation-harness to report normalized average accuracy degradation (Acc. Gap %), alongside WikiText2 perplexity.
Key Experimental Results¶
Main Results¶
Profiling was performed on an NVIDIA RTX Pro 6000 Blackwell workstation GPU (sm120) under CUDA 12.8 and PyTorch 2.9.1, executing within CUDA Graph traces. The table below reports the WikiText2 perplexity, average accuracy gap across the seven reasoning benchmarks, and realized end-to-end throughput speedups on Llama3.1-8B under 25% and 50% nominal sparsity:
| Pruning Taxonomy | Representative Method | Sparsity | WikiText2 (PPL↓) | 7-Task Avg. Acc. Gap (%↓) | Prefill Realized Speedup (Mean [Min, Max]) | Decode Realized Speedup (Mean [Min, Max]) |
|---|---|---|---|---|---|---|
| Dense Baseline | Vanilla Llama3.1-8B | 0.0% | 7.54 | 0.00% | 1.00x [1.00x, 1.00x] | 1.00x [1.00x, 1.00x] |
| Static M | Shortened-taylor | 25.0% | 15.52 | 10.59% | 1.29x [1.28x, 1.34x] | 1.32x [1.31x, 1.34x] |
| Static K (low-rank) | Dobi-SVD | 25.0% | 10.14 | 6.52% | 1.15x [1.11x, 1.19x] | 1.09x [0.97x, 1.23x] |
| Static NK | Týr-the-Pruner | 25.0% | 12.46 | 12.63% | 1.27x [1.24x, 1.35x] | 1.27x [1.23x, 1.39x] |
| Static NK (cross-layer) | SliceGPT | 25.0% | 38.96 | 21.16% | 1.13x [1.12x, 1.16x] | 1.08x [1.02x, 1.17x] |
| Dynamic M | SkipGPT | 25.0% | 9.25 | 3.94% | 1.08x [0.98x, 1.12x] | 0.91x [0.75x, 1.05x] |
| Dynamic NK | SeerAttention (Forced 25%) | 25.0% | 7.72 | 33.40% | 1.02x [1.00x, 1.06x] | 1.02x [1.00x, 1.04x] |
| Static M | CoopPruner | 50.0% | 33.93 | 30.28% | 1.88x [1.85x, 1.92x] | 1.91x [1.89x, 1.94x] |
| Static K (low-rank) | Dobi-SVD | 50.0% | 15.44 | 19.83% | 1.43x [1.31x, 1.53x] | 1.46x [1.38x, 1.57x] |
| Static K (semi-structured) | MaskLLM | 50.0% | 11.45 | 10.67% | 1.08x [1.03x, 1.20x] | 1.14x [1.08x, 1.26x] |
| Static NK | Týr-the-Pruner | 50.0% | 19.80 | 26.10% | 1.77x [1.62x, 1.84x] | 1.70x [1.62x, 1.84x] |
| Static NK (cross-layer) | SliceGPT | 50.0% | 68.71 | 38.36% | 1.59x [1.37x, 1.76x] | 1.46x [1.26x, 1.59x] |
| Dynamic M | SkipGPT | 50.0% | 13.14 | 15.44% | 1.44x [1.13x, 1.56x] | 1.10x [0.76x, 1.44x] |
| Dynamic NK | SeerAttention (Forced 50%) | 50.0% | 7.76 | 34.45% | 1.05x [0.98x, 1.20x] | 1.04x [1.01x, 1.10x] |
Ablation Study: Dimension Alignment & Non-Uniformity Sensitivity¶
In physical hardware execution, width pruning alters matrix shapes away from hardware-native tensor core alignments. The table below quantifies the normalized prefill throughput on Llama3.1-8B (batch size 1, context length 32,768, 50% target sparsity, under \(\pm 20\%\) random perturbation of per-module pruning ratios) across different channel alignment multiples:
| Perturbation Range | Alignment Multiple (Align) | Static K (low-rank) Norm. Speed | Static NK Norm. Speed | Static NK (cross-layer) Norm. Speed |
|---|---|---|---|---|
| \(\pm 0\%\) | Unperturbed Baseline (tokens/s) | 19,259.84 (1.000x) | 28,334.09 (1.000x) | 20,892.26 (1.000x) |
| \(\pm 20\%\) | Nearest Odd Integer (odd) | \(0.7034 \pm 0.0038\) | \(0.7069 \pm 0.0323\) | \(0.6552 \pm 0.0088\) |
| \(\pm 20\%\) | Nearest Even Integer (even) | \(0.9019 \pm 0.0061\) | \(0.9614 \pm 0.0421\) | \(0.8533 \pm 0.0020\) |
| \(\pm 20\%\) | Multiple of 4 (Align=4) | \(0.9279 \pm 0.0057\) | \(0.9961 \pm 0.0238\) | \(0.8762 \pm 0.0057\) |
| \(\pm 20\%\) | Multiple of 8 (Align=8) | \(0.9878 \pm 0.0090\) | \(1.0572 \pm 0.0240\) | \(0.9664 \pm 0.0250\) |
| \(\pm 20\%\) | Multiple of 16 (Align=16) | \(1.0048 \pm 0.0080\) | \(1.0669 \pm 0.0264\) | \(0.9794 \pm 0.0243\) |
| \(\pm 20\%\) | Multiple of 128 (Align=128) | \(0.9946 \pm 0.0117\) | \(1.0654 \pm 0.0277\) | \(0.9852 \pm 0.0106\) |
Micro-benchmarking of standalone GEMM kernels demonstrates an even sharper penalty: in FP8 precision, when inner dimension \(K\) is misaligned below 16 bytes, Triton kernel throughput collapses to merely 11.8% of the baseline, recovering to 70.4% \(\sim\) 104.6% only when aligned to multiples of 16 or greater.
Key Findings¶
- Quality-Dependent Pareto Transitions: Pruning does not follow a monolithic Pareto frontier; rather, the optimal taxonomy shifts dynamically with the permissible quality loss budget. In prefill, Static M establishes the absolute frontier under tight quality constraints (0% \(\sim\) 4% degradation, reaching 1.12x speedup at 12.5% sparsity); Dynamic M commands the moderate loss regime (5% \(\sim\) 16% degradation, delivering 1.24x \(\sim\) 1.44x speedup with superior semantic preservation); while Static NK assumes the frontier at higher degradation regimes (17% \(\sim\) 26% loss), outperforming aggressive depth dropping.
- The Non-GEMM Overhead Penalty: Realized acceleration deviates sharply from theoretical limits due to auxiliary execution overheads. While Static M reaches 1.91x speedup at 50% sparsity in decode (approaching its theoretical ceiling), Dynamic M and semi-structured Static K stall at 1.10x and 1.14x. Profiling breakdowns reveal that Dynamic M and Static K incur substantial non-GEMM overhead increases of 42.4% and 40.8% in prefill, escalating dramatically to +61.5% and +287.2% during autoregressive decode, severely diminishing their nominal computational savings.
- Non-Uniform Task Sensitivity to Sparsity: Complex reasoning and reading comprehension tasks degrade steeply under structural pruning: ARC-C, ARC-E, and HellaSwag exhibit an average relative accuracy degradation approaching or exceeding 10% per 12.5% increase in sparsity. Conversely, physical commonsense QA (PIQA) exhibits robust resilience, retaining over 90% accuracy across multiple pruning taxonomies even at 50% sparsity.
Highlights & Insights¶
- Unified Mathematical Framing for Disparate Hardware Behaviors: Successfully reduces diverse structural pruning methods into logical \(M, N, K\) GEMM operations, clarifying why certain techniques maintain persistent layer-wise acceleration while others suffer from catastrophic dimension-alignment penalties.
- Architectural Guidance for Next-Generation Hybrid Accelerators: Demonstrates that pushing any single pruning taxonomy in isolation encounters steep hardware or representational limits, pointing directly toward hybrid architectures that merge token-adaptive dynamic routing with statically pruned structural backbones.
Limitations & Future Work¶
- Omission of Mixture-of-Experts and Data-Center Blackwell Architectures: The evaluation centers on dense Transformer models evaluated on workstation-class GPUs (RTX Pro 6000 sm120 and A800 sm80). Validating on Mixture-of-Experts (MoE) backbones and data-center platforms (sm100) featuring asynchronous Tensor Memory Accelerator (TMA) instructions represents an essential open frontier.
- DSL Portability vs. Extreme Hand-Tuned Assembly: The unified benchmark utilizes Triton and Tilelang to maintain cross-platform generality, which may yield minor throughput gaps compared to proprietary, hand-crafted CUDA/assembly kernels.
- Single-Step Execution vs. Production Serving Frameworks: Benchmarking isolates single-step prefill and decode latency within CUDA Graph replay, leaving interactions with continuous batching, chunked prefill, and dynamic KV-cache managers in production engines (e.g., SGLang, vLLM) for future system exploration.
Related Work & Insights¶
- vs ELANA (Chiang et al., 2025): While ELANA offers analytical energy and latency estimation tools, this work contributes an operational operator-replacement framework with actual executing kernels, empirical profiling, and unified LoRA recovery.
- vs SliceGPT (Ashkboos et al., 2023) & Týr-the-Pruner (Li et al., 2025a): The original works evaluate compression via theoretical FLOPs and isolated benchmarks; this taxonomy unifies them under the Static NK formulation, identifying how unaligned dimensions and residual projection overheads constrain their realized gains relative to Static M.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Establishes the first unified GEMM-centric operational taxonomy for LLM pruning, permanently shifting the evaluation paradigm beyond theoretical FLOPs.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorously profiles multiple model scales, fine-tuning checkpoints, fine-grained microsecond latency traces, and seven downstream reasoning benchmarks.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptional narrative clarity, mathematically cohesive definitions, and elegant alignment between visual flowcharts and empirical discoveries.
- Value: ⭐⭐⭐⭐⭐ Provides an indispensable, reproducible decision map for both algorithmic researchers and systems engineers deploying compressed LLMs in production.