Skip to content

Shallow to Deep: Aligning Token Pruning with Stage-wise Roles in LVLMs

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Twilight03/STD
Area: Multimodal VLM
Keywords: Visual Token Pruning, Large Vision-Language Models, Spectral Analysis, Attention Smoothing, Stage-Aware Compression

TL;DR

Addressing the severe performance degradation caused by indiscriminate attention-based pruning in shallow layers of LVLMs, this paper proposes STD, a training-free stage-aligned pruning framework that employs high-frequency spectral analysis in shallow layers, Gaussian-smoothed attention in intermediate layers, and stability-adaptive triggering in deep layers, achieving up to 94.4% token reduction and 3.9x prefill speedup while consistently outperforming SOTA baselines.

Background & Motivation

Large Vision-Language Models (LVLMs) incur immense computational and memory overheads during inference due to the proliferation of visual tokens, especially when processing high-resolution inputs and multi-frame video sequences. To alleviate this computational bottleneck during the prefilling stage, training-free multi-layer visual token pruning within the vision encoder has emerged as an attractive paradigm. By progressively removing tokens with low self-attention scores across layers, multi-layer pruning avoids the premature propagation of redundant features into subsequent layers and the LLM backbone. However, empirical studies reveal an overlooked yet pervasive failure mode: executing attention-based token pruning in the shallow layers of the vision encoder consistently harms downstream task accuracy. In fact, simply disabling pruning in early layers while retaining it in middle and deep stages yields substantially better performance than standard full-depth pruning across multiple benchmark evaluations.

An in-depth probe into layer-wise attention distributions uncovers the root cause of this early-stage failure. Existing literature commonly assumes that shallow transformer layers primarily perform local neighborhood feature aggregation. Contrary to this conventional wisdom, empirical attention maps in the very shallow layers exhibit severe chaos and noise: token attention distributions display an abnormal U-shaped distance curve where tokens arbitrarily attend to distant background clutter and padding rather than localized structures. Crucially, this early disorder is not meaningless noise. Rather, the model in its immature semantic phase operates essentially as a high-frequency structural edge detector. As the network deepens, visual representations undergo a pronounced three-stage qualitative transition: shallow layers capture high-frequency contour boundaries; intermediate layers transit into coherent local object-level perception; and deep layers converge onto a small set of hub tokens undergoing non-uniform, volatile abstract semantic aggregation.

Consequently, the core tension lies in the severe misalignment between the uniform attention criteria used in conventional pruning and the evolving functional roles across network depth. Shallow layers lack well-formed semantic attention weights, meaning attention scores act as an unreliable proxy that mistakenly discards vital edge cues, whereas deep layers undergo rapid semantic shifts where uniform-interval pruning easily disrupts delicate concept crystallization. This paper addresses this mismatch by moving away from monolithic attention scoring and explicitly synchronizing pruning operators with the functional characteristics of each network stage. Core idea: propose STD (Shallow to Deep), a training-free hierarchical pruning framework that deterministically preserves structural contours via high-frequency spectral analysis in shallow layers, maintains spatial entity coherence via Gaussian-smoothed attention in intermediate layers, and executes pruning exclusively during stable semantic intervals via an adaptive inter-layer trigger in deep layers.

Method

Overall Architecture

The STD framework aligns visual token compression with the intrinsic stage-wise functional roles of the vision encoder without requiring any gradient updates or parameter fine-tuning. The vision backbone is partitioned into three functional intervals: shallow layers (\(\ell \le L_1\)), intermediate layers (\(L_1 < \ell \le L_2\)), and deep layers (\(\ell > L_2\)), with default boundaries set to \(L_1 = 6\) and \(L_2 = 12\). In shallow layers, STD applies 1D Real-valued Fast Fourier Transform (RFFT) to extract deterministic high-frequency boundary energy, replacing erratic attention scores. In intermediate layers, it applies a 1D Gaussian smoothing filter to attention vectors to prevent object fragmentation. In deep layers, it computes the \(L_2\) norm of inter-layer attention shifts, triggering token elimination only when semantic representations stabilize, before finally projecting the compact visual tokens to the LLM.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Visual Patch Sequence P(โ„“)<br/>(Vision Encoder Input)"] --> B["Shallow Stage (โ„“ โ‰ค L1)<br/>High-Frequency Spectral Analysis"]
    B --> C["Intermediate Stage (L1 < โ„“ โ‰ค L2)<br/>Gaussian-Smoothed Attention"]
    C --> D["Deep Stage (โ„“ > L2)<br/>Stability-Adaptive Trigger"]
    D -->|Semantic shift ฮ”(โ„“) < ฮ”(โ„“-1)| E["Execute Attention Pruning<br/>Enforce Target Token Budget"]
    D -->|Rapid semantic evolution| F["Skip Pruning at Layer โ„“<br/>Safeguard Concept Aggregation"]
    E --> G["Compact Visual Tokens<br/>Forward to Projector & LLM"]
    F --> G

Key Designs

1. High-Frequency Spectral Analysis: Deterministic Structural Contour Preservation in Shallow Layers

To circumvent the chaotic and noisy attention distributions in shallow layers (\(\ell \le L_1\)) that lead to accidental deletion of essential structural outlines, this design replaces attention sorting with frequency-domain spectral analysis. For a sequence of \(N\) patch tokens \(P^{(\ell)} = [p^{(\ell)}_1, \dots, p^{(\ell)}_N]^\top \in \mathbb{R}^{N \times D}\) at layer \(\ell\), 1D Real-valued Fast Fourier Transform (RFFT) is performed along the sequence dimension to yield frequency spectrum \(\widehat{P}^{(\ell)}\). To isolate boundary and edge cues, only the highest frequency band of proportion \(\rho\) (default \(\rho = 1/3\)) is retained:

\[\widehat{P}^{(\ell)}_{\text{high}}[f, :] = \begin{cases} \widehat{P}^{(\ell)}[f, :], & \text{if } f > (1 - \rho) \cdot (\lfloor N/2 \rfloor + 1) \\ 0, & \text{otherwise} \end{cases}\]

The high-frequency spatial representation is reconstructed via Inverse RFFT: \(\widetilde{P}^{(\ell)}_{\text{high}} = \text{IRFFT}(\widehat{P}^{(\ell)}_{\text{high}})\), and the high-frequency energy score for token \(i\) is calculated as \(s^{(\ell)}_i = \|\widetilde{p}^{(\ell)}_{\text{high}, i}\|_2\). To ensure a smooth inductive transition, the final retention score combines normalized spectral energy with CLS attention weights \(a^{(\ell)}_i\):

\[r^{(\ell)}_i = \alpha \cdot s^{(\ell)}_i + (1 - \alpha) \cdot a^{(\ell)}_i\]

where \(\alpha\) is a balancing coefficient. Because high-frequency spectral energy strongly correlates with Canny edge detection ground truth (similarity \(\approx 0.8\)), this formulation provides a deterministic proxy for structural saliency and completely eliminates the vulnerability of early-stage attention collapse.

2. Gaussian-Smoothed Attention: Spatial Coherence and Entity Integrity in Intermediate Layers

As the network progresses into intermediate layers (\(L_1 < \ell \le L_2\)), representations shift toward object-level semantic recognition, where attention maps exhibit localized clustering around salient foreground subjects. However, standard token pruning applies sharp hard-thresholding on individual token scores, frequently discarding transitional boundary patches and fragmenting unified entities into disjoint token islands. To preserve spatial coherence and feature continuity, this design convolves the raw attention vector with a discrete 1D Gaussian smoothing filter:

\[r^{(\ell)}_i = \sum_{k = -\lfloor K/2 \rfloor}^{\lfloor K/2 \rfloor} G_\sigma(k) \cdot a^{(\ell)}_{i+k}\]

where \(G_\sigma\) represents a 1D Gaussian kernel with standard deviation \(\sigma\) and window size \(K\), and \(a^{(\ell)}\) is the CLS attention distribution. By diffusing attention weights across local neighborhoods, peripheral patches belonging to the same object are preserved alongside core tokens, preventing semantic erosion and maintaining unbroken structural contexts as features travel deeper.

3. Stability-Adaptive Trigger: Non-Uniform Pruning Scheduling via Inter-Layer Semantic Divergence

In deep layers (\(\ell > L_2\)), the vision encoder enters an abstract semantic aggregation regime characterized by extreme attention concentration onto a sparse set of hub tokens. Crucially, semantic evolution in this stage does not progress smoothly; rather, it alternates between bursts of rapid conceptual restructuring and plateaus of relative representational stability. Pruning tokens during an unstable transition layer severely degrades the model's ability to synthesize global concepts. To address this, STD introduces a stability-adaptive trigger based on the inter-layer attention shift. Letting \(a^{(\ell)} \in \mathbb{R}^N\) denote the CLS attention vector over all patch tokens at layer \(\ell\), the semantic variation between consecutive layers is measured by:

\[\Delta^{(\ell)} = \|a^{(\ell)} - a^{(\ell-1)}\|_2\]

Instead of pruning at arbitrary fixed intervals (e.g., every 2 layers), pruning is dynamically triggered only when the inter-layer semantic evolution decelerates:

\[\Delta^{(\ell)} < \Delta^{(\ell-1)}\]

When this condition is satisfied, the model has entered a temporarily quiescent state with high informational redundancy, making token removal safe. Tokens are then pruned using pure attention scores \(r^{(\ell)}_i = a^{(\ell)}_i\). At the final pruning layer, the retained token count is strictly aligned with the target budget, preserving critical concept formation steps while fulfilling severe compression targets.

Benchmark & Evaluation Protocol

STD is implemented as a purely training-free, zero-shot plugin without fine-tuning visual or linguistic parameters. Across all tested LVLM families, default partitioning thresholds are standardized to \(L_1 = 6\) and \(L_2 = 12\), with high-frequency spectrum cutoff \(\rho = 1/3\). The weight factor is configured as \(\alpha = 0.8\) for standard-resolution architectures (LLaVA-1.5) and \(\alpha = 0.5\) for high-resolution and dynamic patch models (LLaVA-NeXT, Qwen2-VL). Experiments are conducted on NVIDIA RTX 4090D (24GB) GPUs, evaluating performance across 10 image understanding benchmarks (GQA, MMBench, MMB-CN, MME, POPE, ScienceQA, VQAv2, TextVQA, MIA-Bench, MMStar, RWQA) and 3 video benchmarks (TGIF-QA, MSVD-QA, MSRVTT-QA).

Key Experimental Results

Main Results

STD was comprehensively benchmarked against leading multi-layer and cross-modal pruning baselinesโ€”including PDrop, SparseVLM, Flowcut, FiCoCo-V, and V2Dropโ€”under identical budget constraints.

Table 1: Quantitative Performance on LLaVA-1.5-7B Across Token Retention Budgets (Relative Average to Theoretical Upper Bound)

Method Retained Tokens (Ratio) GQA MME POPE SQA VQAv2 TextVQA Relative Avg.
Vanilla (Upper Bound) 576 (100%) 61.9 1865 85.9 69.5 78.5 58.3 100.0%
PDrop (CVPR25) 192 (โ†“66.7%) 57.3 1766 82.3 68.8 75.1 56.5 96.2%
SparseVLM (ICML25) 192 (โ†“66.7%) 57.6 1721 83.6 68.9 75.6 56.1 95.4%
Flowcut (NeurIPS25) 192 (โ†“66.7%) 59.5 1833 85.7 68.4 76.8 57.3 97.8%
V2Drop (CVPR26) 192 (โ†“66.7%) 58.5 1826 85.1 69.3 75.2 55.6 97.3%
STD (Ours) 192 (โ†“66.7%) 60.1 1812 86.3 68.5 77.5 57.7 98.5%
SparseVLM + STD 192 (โ†“66.7%) 60.6 1818 86.2 68.9 77.7 58.0 99.0%
PDrop (CVPR25) 64 (โ†“88.9%) 47.5 1561 55.9 68.6 69.2 50.6 84.6%
SparseVLM (ICML25) 64 (โ†“88.9%) 52.7 1505 75.1 62.2 68.2 51.8 85.6%
Flowcut (NeurIPS25) 64 (โ†“88.9%) 55.3 1712 79.3 68.9 72.8 55.6 93.2%
FiCoCo-V (AAAI26) 64 (โ†“88.9%) 52.4 1591 76.0 68.1 71.3 53.6 90.5%
V2Drop (CVPR26) 64 (โ†“88.9%) 50.5 1470 75.1 68.9 68.9 51.8 87.5%
STD (Ours) 64 (โ†“88.9%) 56.1 1709 81.2 68.6 73.3 56.0 94.3%
Flowcut + STD 64 (โ†“88.9%) 56.4 1744 81.6 68.8 73.9 55.9 95.0%

Table 2: Inference Acceleration and Accuracy on LLaVA-NeXT-7B (POPE Benchmark, Single RTX 4090D GPU)

Method Tokens POPE TextVQA Prefill Latency Prefill Speedup Total Latency Total Speedup TFLOPs
LLaVA-NeXT Baseline 2880 (100%) 86.4 61.3 323.0ms 1.0ร— 50:24 1.0ร— 45.61
+ SparseVLM 160 (โ†“94.4%) 72.7 49.2 215.3ms 1.0ร— 42:00 1.0ร— 11.20
+ PDrop 160 (โ†“94.4%) 72.3 53.9 124.2ms 1.5ร— 19:23 2.6ร— 9.96
+ Flowcut 160 (โ†“94.4%) 77.1 54.4 100.9ms 2.6ร— 16:48 3.0ร— 6.30
+ V2Drop 160 (โ†“94.4%) 75.2 54.6 114.7ms 2.8ร— 18:10 2.8ร— 8.97
+ STD (Ours) 160 (โ†“94.4%) 77.3 56.6 83.2ms 3.9ร— 13:07 3.8ร— 5.16

Ablation Study

The contribution of each stage-specific module and the validity of layer-specific assignments were evaluated on LLaVA-1.5-7B under the strict 64-token budget setting.

Table 3: Component Ablations and Layer Assignment Permutations (64 Retained Tokens)

Configuration / Layer Placement HSA (Shallow) GSA (Middle) SAT (Deep) TextVQA GQA POPE Observations
Standard Multi-Layer Pruning - - - 54.5 54.7 79.2 Uniform attention baseline
Standard Pruning (w/o Shallow) - - - 55.0 55.1 79.8 Disabling shallow pruning helps
STD(a) (HSA only) โœ“ - - 55.5 55.6 80.3 Addresses shallow edge drift
STD(b) (GSA only) - โœ“ - 55.2 55.4 79.9 Enhances spatial coherence
STD(c) (SAT only) - - โœ“ 55.4 55.6 80.6 Avoids unstable transition layers
Full STD โœ“ โœ“ โœ“ 56.0 56.1 81.2 Optimal synergy across all stages
Misplacement: HSA to Deep - - In Deep 51.6 - 72.7 Destroys semantic aggregation (-8.5% POPE)
Misplacement: GSA to Shallow In Shallow - - 53.9 - 78.6 Diffuses chaotic noise into neighbors

Key Findings

  1. Strictly Synergistic Across Stages: Under heavy compression (64 tokens retained, 88.9% reduction), HSA, GSA, and SAT each independently boost the POPE benchmark by +1.1%, +0.7%, and +1.4% respectively over the uniform pruning baseline. Combining all three yields a cumulative +2.0% gain, confirming that each mechanism targets an orthogonal functional bottleneck.
  2. Layer-Specific Role Irreversibility: Cross-layer permutation tests in Table 8 demonstrate catastrophic failure when modules are deployed outside their designated stages. Forcing High-Frequency Spectral Analysis (HSA) into deep layers plunges POPE score to 72.7% (worse than the 79.2% naive baseline), as deep layers require abstract semantic clustering rather than pixel-level contour preservation. Similarly, applying Gaussian smoothing in shallow layers smears unorganized noise across adjacent patches.
  3. Cross-Stage Complementarity with LLM Pruning: Vision-encoder pruning (STD) and LLM-stage pruning (DART, V2Drop) address perception-level versus cross-modal textual redundancy respectively. Combining STD in the vision encoder with V2Drop in the LLM achieves an impressive 98.3% relative retention score on LLaVA-1.5, significantly outpacing either strategy alone.

Highlights & Insights

  • Reinterpreting Shallow Attention Noise via Frequency Analysis: The paper overturns the prevailing assumption that early transformer layers perform localized neighborhood aggregation, demonstrating instead that chaotic early attention reflects rudimentary edge detection that can be far more reliably captured via deterministic 1D Fourier spectral energy.
  • Dynamic Derivative-Guided Pruning Execution: Departing from the rigid practice of pruning at fixed periodic intervals, the stability-adaptive trigger monitors the second-order acceleration of inter-layer attention shifts, pruning only when representational changes decelerate.
  • Cross-Architectural Empirical Universality: The three-stage functional trajectory (edge contour \(\rightarrow\) local object \(\rightarrow\) abstract hub aggregation) was shown to generalize identically to the newest generation of visual backbones including InternVL3-8B and Qwen3-VL-8B, providing a robust inductive prior for efficient multimodal model design.

Limitations & Future Work

  • Author-Acknowledged Limitations: Stage-wise partition boundaries (\(L_1 = 6, L_2 = 12\)) and balancing hyperparameters (\(\alpha, \rho\)) are empirically fixed; optimal thresholds may vary under radical aspect ratio shifts or entirely novel visual backbones (e.g., pure convnets or state-space models). Additionally, the approach focuses exclusively on visual token length in the prefilling stage, leaving KV-cache footprint and auto-regressive decoding latencies unaddressed.
  • Further Methodological Limitations: Flattening 2D patch grids into 1D sequences prior to RFFT computation disregards 2D spatial grid topology and anisotropic frequency dynamics. Moreover, in dense text scenarios (e.g., Document VQA), rigid spectral filtering risks suppressing fine text strokes.
  • Promising Research Directions: Developing fully parameter-free, adaptive boundary discovery based on layer-wise spectral entropy, and extending 1D RFFT to full 2D spatial-frequency wavelet transforms for fine-grained multimodal document parsing.
  • vs Flowcut (NeurIPS 2025) & FiCoCo-V (AAAI 2026): While Flowcut and FiCoCo-V introduce supplementary criteria like information flow density and local penalty metrics to correct attention biases, they enforce an identical composite scoring function uniformly across all layers. STD demonstrates that early and deep stages require fundamentally distinct mathematical operations (spectral vs. smoothed vs. stability-triggered).
  • vs PDrop (CVPR 2025) & V2Drop (CVPR 2026): Downstream methods prune tokens at the projector or early LLM layers using cross-modal textual queries. However, they force all visual tokens through the entire vision encoder, inducing high memory spikes during initial ingestion. STD prunes early in the visual encoder, cutting latency prior to the projector while offering orthogonal gains when paired with downstream methods.

Rating

  • Novelty: โญโญโญโญโ˜† Unearths the physical rationale behind shallow-layer pruning failures and introduces a well-founded stage-aligned spectral-temporal framework.
  • Experimental Thoroughness: โญโญโญโญโญ Rigorously tested across 13 diverse benchmarks, extreme budgets (32/64 tokens), modern backbones (InternVL3, Qwen3-VL), and real hardware latency profiling.
  • Writing Quality: โญโญโญโญโญ Compelling narrative flow with insightful empirical diagnostic experiments (Canny edge alignment and attention derivative curves).
  • Value: โญโญโญโญโญ Training-free, plug-and-play inference acceleration delivering a 3.9x prefill speedup at 94.4% token pruning with minimal accuracy degradation.