Spectral Evolution-Guided Token Pruning in Multimodal Large Language Models¶

Conference: ECCV 2026
arXiv: 2606.24165
Code: https://github.com/zjubinchen/CLSE
Area: LLM Efficiency / Multimodal VLM
Keywords: Visual Token Pruning, Cross-Layer Spectral Evolution, Frequency-Domain High-Pass Filtering, Training-Free Acceleration, MLLM Inference

TL;DR¶

This paper proposes CLSE (Cross-Layer Spectral Evolution), a training-free visual token pruning framework. Instead of evaluating token importance using single-layer attention or feature magnitudes, CLSE transforms the visual tokens of each layer into the frequency domain, applies high-pass filtering to isolate high-frequency structures, and measures the relative variation of this structural energy between adjacent LLM layers. Tokens with highly dynamic variations are classified as semantically active and are retained. Under aggressive compression (retaining only 11% to 33% of the tokens) across various MLLMs such as LLaVA, Qwen2-VL, and Video-LLaVA, CLSE's accuracy retention significantly outperforms FastV, SparseVLM, PDrop, and other baselines.

Background & Motivation¶

Background: Modern MLLMs concatenate hundreds or thousands of ViT-encoded visual tokens with textual sequences for LLM-based decoding. Due to the quadratic complexity of self-attention with respect to sequence length, processing high-resolution images or videos causes the token count to explode, creating substantial bottlenecks in inference latency and KV cache footprint. Since spatial and temporal representations of visual tokens exhibit massive redundancies (highly correlated adjacent patches or frames), pruning redundant visual tokens during inference has emerged as a mainstream trajectory for training-free acceleration.

Limitations of Prior Work: The vast majority of training-free pruning methods rely on single-layer instantaneous signals, such as attention weights (FastV), feature magnitudes, or token-to-token similarity (DART/ToMe). However, these signals suffer from two major flaws: first, attention displays a distinct position bias (perpetuated by position embeddings), tending to assign higher scores to visual tokens located in the latter part of the concatenated sequence, even when those regions lack semantic significance; second, static magnitude-based methods overlook a fundamental property of MLLM decoding—namely, that visual representations evolve structurally across layers, progressing from low-level spatial details to high-level semantic abstractions while incorporating textual instructions.

Key Challenge: The true utility of a token is reflected in "how it is reshaped by cross-modal interactions along the network depth," rather than its instantaneous strength at an isolated layer. Approximating this inherently "cross-layer dynamic" attribute through a single-layer snapshot is fundamentally a misalignment.

Goal + Key Insight: The authors seek an importance metric that is stable (immune to position bias), effectively captures representation dynamics, and remains computationally cheap (no slower than retrieving attention matrices). The key insight stems from the frequency domain: previous literature (such as FNet and Fourier-VLM) indicates that shallow Transformer layers retain high-frequency spatial details while deep layers consolidate low-frequency semantic structures. This "depth-wise spectral drift" represents the natural progression of multimodal abstraction.

Core Idea: Token importance can be quantified by the intensity of its frequency-domain representation evolving across layers. Semantically meaningful tokens undergo stronger spectral redistribution during the "detail \(\to\) semantics" transition, while redundant background tokens remain practically static. By shifting from "measuring magnitude/attention" to "measuring cross-layer spectral evolution," the framework faithfully identifies which tokens should be preserved.

Method¶

Overall Architecture¶

At its core, CLSE is a token importance scorer. In the early stages of the LLM decoder, it grabs visual token representations from adjacent layers, transforms them into the frequency domain via FFT, and filters them using a high-pass mask to generate a "high-frequency structural energy map." The relative change in this energy map between the two layers represents each token's evolution intensity. Hard pruning is then performed via Top-K selection based on this intensity score, and only the shortened sequence is forwarded through the remaining stages. The entire process requires no training or weight adjustment, serving as a plug-and-play module.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Extract visual token representations<br/>from adjacent layers X^ℓ, X^(ℓ+1)"] --> B["Reshape into spatial map<br/>(H×W×d) and apply 2D FFT + fftshift"]
    B --> C["Gaussian high-pass mask M<br/>suppresses low frequencies, retains local structure"]
    C --> D["Inverse transform to get amplitude,<br/>channel-wise average → high-frequency energy map S_ℓ"]
    D --> E["Cross-layer spectral evolution intensity<br/>I = |S_(ℓ+1) − S_ℓ| / (S_ℓ + mean + ε)"]
    E --> F["Perform Top-K hard pruning based on I<br/>(default 0→1 scoring, pruned before layer 2)"]
    F --> G["Short sequence [text; retained visual]<br/>fed into all subsequent layers"]
    G -.Optional.-> H["CLSE-M: Recall top-scored pruned tokens,<br/>density-peak clustering to merge"]

Key Designs¶

1. High-Frequency Structural Energy instead of Feature Magnitude: Isolating "Local Structure" via Frequency-Domain High-Pass Filtering

Traditional magnitude-based metrics are dominated by the absolute norm of embeddings and fail to distinguish fine-grained structural information from global low-frequency backgrounds. CLSE resolves this by measuring the band-limited high-frequency energy of the tokens. Given the visual tokens at layer \(\ell\), \(\mathbf{X}^{\ell}\in\mathbb{R}^{N\times d}\) (\(N=H\cdot W\)), they are reshaped into a spatial feature map \(\mathbf{X}^{\ell}_{2\mathrm{D}}\in\mathbb{R}^{H\times W\times d}\). After applying a centralized 2D FFT: \(\widehat{\mathbf{X}}^{\ell}=\mathrm{fftshift}(\mathcal{F}(\mathbf{X}^{\ell}_{2\mathrm{D}}))\), they are multiplied by a Gaussian high-pass mask to filter out low-frequency elements:

\[\mathbf{M}_{u,v}=1-\exp\!\left(-\frac{(u-c_u)^2+(v-c_v)^2}{2(r\cdot c_u)^2}\right)\]

where \((c_u,c_v)\) represents the spectral center, and \(r\) is the cutoff ratio (defaulting to \(r{=}0.1\)). Applying the inverse transform, obtaining the amplitude, and averaging across channels yields the high-frequency energy map \(\mathbf{S}_\ell=\mathrm{Avg}_d\!\left(\left|\mathcal{F}^{-1}(\mathrm{ifftshift}(\widehat{\mathbf{X}}^{\ell}\odot\mathbf{M}))\right|\right)\in\mathbb{R}^{H\times W}\), which is flattened to \(\mathbb{R}^N\). Since \(\mathbf{S}_\ell\) captures the "residual energy after low-frequency suppression" (edges and structures) rather than the absolute feature norm, it acts as a precise structural proxy. The ablation study (Fig. 5c) illustrates that directly utilizing the L1 residual norm (SFR) without high-pass filtering degrades performance due to pixel-level noise, vindicating this frequency-domain isolation.

2. Cross-Layer Spectral Evolution Intensity: Measuring Whether a Token is "Being Reshaped" via Relative Changes in Adjacent Layer Energy

Single-layer energy alone cannot reflect dynamic changes. While background tokens yield nearly static residual maps, semantically active tokens undergo significant cross-layer redistribution. For reference layer \(\ell\) and subsequent layer \(\ell{+}1\), CLSE defines the evolution intensity:

\[\mathcal{I}^{(\ell)}=\mathrm{clip}\!\left(\frac{|\mathbf{S}_{\ell+1}-\mathbf{S}_{\ell}|}{\mathbf{S}_{\ell}+\overline{\mathbf{S}}_{\ell}+\epsilon},\ 0,\ \tau_{\max}\right),\quad \overline{\mathbf{S}}_{\ell}=\mathrm{mean}(\mathbf{S}_{\ell})\]

This metric is designed to be scale-invariant: the division by \(\mathbf{S}_\ell\) prevents high-amplitude tokens from skewing the results, while the global average \(\overline{\mathbf{S}}_\ell\) acts as a regularizer to suppress minor spectral noise in inactive background regions. The hyperparameter \(\epsilon\) and threshold \(\tau_{\max}\) ensure numerical stability. Because the numerator measures structural changes between layers—which are highly concentrated in the early decoding phases where visual representations are aggressively updated—tokens with high \(\mathcal{I}^{(\ell)}\) mapped to regions undergoing active semantic reconstruction. A counterfactual study (Fig. 5b)—CLSE-Inertia (which retains only the weakest-evolving, most static tokens)—leads to catastrophic performance collapses, confirming that cross-layer spectral evolution directly indexes semantic engagement.

3. Early-Layer Pruning Scheduling: Pruning at the 1st Decoder Layer to Maximize Prefilling Acceleration

To achieve optimal efficiency gains, CLSE schedules token pruning at the very beginning of the LLM decoder. Under default configurations, \(\mathcal{I}^{(\ell)}\) is computed using the inputs \(\mathbf{X}^0\) and the first layer outputs \(\mathbf{X}^1\) (i.e., \(0\!\to\!1\)). The Top-K selection \(\mathcal{K}=\mathrm{ToK}(\mathcal{I}^{(\ell)},K)\) is then executed prior to the 2nd layer, allowing all subsequent layers to operate exclusively on the pruned visual sequence \(\tilde{\mathbf{X}}^{\ell}=\mathbf{X}^{\ell}[\mathcal{K}]\). Appendix Table 8 demonstrates that early-layer pruning is critical: early visual tokens preserve high spatial diversity, making CLSE signals highly discriminative. Conversely, middle layers (e.g., layer 10) perform poorly because continuous inter-layer mixing flattens spectral diversity, while most computations have already been expended. Deep layers (\(\ge 15\)) show recovered accuracy but offer zero speedup benefits. The optimal range is recommended as \(K\in\{1,3\}\). A mathematical guarantee (Thm. 1) is also introduced: under downstream Lipschitz + saliency alignment assumptions, the output perturbation caused by pruning is bounded by the discarded spectral evolution masses, \(\|g(\mathbf{X}^\ell)-g(P_\Omega(\mathbf{X}^\ell))\|_2\le\alpha\sum_{i\notin\Omega}\mathcal{I}^{(\ell)}_i\), which is minimized by Top-K selection of \(\mathcal{I}^{(\ell)}\). ⚠️ Note that this theoretical bound relies on first-order approximations and Assumption 2, as detailed in the source paper.

4. CLSE-M: Spectral Score-Guided Token Merging for Video-Oriented Information Recall

Under heavy compression in video tasks (e.g., compressing 2048 to 194 tokens, \(>90\%\) reduction), pure pruning inevitably compromises temporal continuity. CLSE-M remedies this by adding a two-stage recall mechanism: it first retrieves the top 30% of pruned tokens based on joint ranking (excluding those with negligible attention), and condenses these \(K\) recalled tokens into \(\lfloor K/10\rfloor+1\) representative tokens via density-peak clustering. Cluster centers are chosen based on local density and distance to higher-density neighbors, and the remaining tokens are averaged and merged into their nearest center. The final sequence is structured as [prefix | r retained visual tokens | ⌊K/10⌋+1 merged representative tokens | text]. Additionally, CLSE-M uses the product of text-to-visual attention and the CLSE factor as the joint score, with the final budget dynamically determined by the numerical rank of the attention matrix (refer to Appendix A.4 for full implementation details).

Loss & Training¶

CLSE is entirely training-free. It does not modify model parameters and requires no post-training or fine-tuning. Pruning boundaries are set on-the-fly during inference. The only active parameters are the cutoff ratio \(r{=}0.1\), pruning step \(K{\in}\{1,3\}\), and temperature \(\tau{=}0.1\) for the CLSE-Hybrid variant (Appendix Table 7).

Key Experimental Results¶

Main Results¶

Performance of LLaVA-1.5-7B (576 tokens pruned to 192/128/64) evaluated as average retention rates relative to the unpruned baseline (Vanilla = 100%) across 9 image benchmarks:

Method	Retain 192 (↓66.7%)	Retain 128 (↓77.8%)	Retain 64 (↓88.9%)
ToMe (ICLR23)	84.8%	84.1%	67.4%
FastV (ECCV24)	92.1%	87.2%	78.0%
PruMerge (ICCV25)	90.8%	88.9%	87.5%
PDrop (CVPR25)	96.9%	95.3%	77.0%
HiRED (AAAI25)	91.0%	89.8%	86.6%
SparseVLM (ICML25)	96.3%	93.7%	84.3%
FiCoCo-V (AAAI26)	96.2%	94.3%	89.8%
CLSE (Ours)	99.4%	98.1%	94.8%

In the most aggressive compression setting (64 tokens, 11% retention), CLSE achieves 94.8% accuracy retention, outperforming the next-best method FiCoCo-V (89.8%) by 5 percentage points. Notably, CLSE beats complex token-recycling/merging schedules using only "spectral evolution scoring + simple hard pruning." This advantage remains consistent across models: Qwen2-VL-7B yields 97.6%/95.8%/90.5%; LLaVA-Next-7B (2880 \(\to\) 320, ↓88.9%) retains 94.7%; LLaVA-1.5-13B yields 99.1%/98.2%/94.1%; InternVL-2.5-8B hits 96.2% under ↓88.9% compression (compared to FastV's 89.4%); Qwen2-VL-72B (4-bit) reaches 96.4%/94.5%/90.0%, where its performance lead over FastV monotonically broadens (from 2.0 \(\to\) 3.7 \(\to\) 7.8 percentage points) as compression intensifies.

Video benchmark performance (Video-LLaVA-7B, 2048 \(\to\) 194, average accuracy evaluated by GPT-4o-mini):

Method	Type	Avg. Acc
Vanilla	—	38.7
FastV	Hard Pruning	26.1
PDrop	Hard Pruning	27.3
FiCoCo-V	Merge	31.9
CLSE	Hard Pruning	35.7
SparseVLM	Merge	38.1
CLSE-M	Merge	38.7 (equal to Vanilla)

Efficiency indicators (LLaVA-1.5-7B, 192 tokens): CLSE drops FLOPs to 43.9%, limits the KV cache to 56 MB, retains 99.4% accuracy, and achieves 1.50\(\times\) prefilling acceleration. FastV is slightly faster (1.69\(\times\)) but its accuracy declines to 92.1%. On videos, CLSE-M matches SparseVLM's efficiency footprint (10.7% FLOPs / 121.1 MB KV cache) while presenting higher accuracy (100% vs 98.4% of Vanilla) and delivering a 2.73\(\times\) speedup.

Ablation Study¶

Analysis of cutoff ratio \(r\) and key components (LLaVA-1.5-7B, 192 tokens, Fig. 5; layer selection details in Appendix Table 8):

Ablation Item	Settings	Conclusion / Data
High-pass cutoff ratio \(r\)	\(r{=}0\) \(\to\) small values	Performance rises sharply from unfiltered baseline to \(r{=}0.1\), validating that mild low-frequency suppression effectively strips structural redundancy; default \(r{=}0.1\)
Spectral evolution direction	CLSE-Inertia (retaining the most static tokens)	Catastrophic collapse, validating that spectral evolution is the primary index of semantic activity
Scoring metric	SFR (L1 residual norm)	Performance collapses; SFR relies too heavily on raw pixel intensities and is highly sensitive to high-frequency artifacts
Scoring metric	MMD (mean difference across channels)	Marginally more stable but remains coarse; scalar values are incapable of representing detailed structural distribution changes within the spectrum
Pruning layer \(K\) (Table 8)	\(K{=}1\) (default)	GQA 61.1 / MME 1814 / POPE 83.7, yielding the lowest total evaluation time (1691s)
Pruning layer \(K\)	\(K{=}10\)	Output severely degrades (POPE 42.7); inter-layer propagation flattens the signal, and computations have already occurred
Pruning layer \(K\)	\(K{\ge}15\)	Accuracy recovers, but acceleration gains dropping to zero
Reference layer selection	\(L{=}K{-}1\) (adjacent layer) vs \(L{=}0\) (global)	Adjacent reference layer is consistently on par with or marginally superior to global reference, marking local step-wise modifications as robust indicators
Video FFT (Table 6)	Frame-wise 2D vs 3D spatiotemporal	Frame-wise 2D comprehensively outperforms 3D (35.7 vs 26.5); sparse frames (8 frames) are too short to fit a reliable temporal 3D spectral analysis

Key Findings¶

Spectral evolution (relative cross-layer changes) is the primary driver of pruning accuracy: Preserving static tokens leads to immediate model failure, proving that visual information value is defined by its "dynamical updating" rather than static magnitude.
High-pass filtering is essential: Discarding the high-pass step (using SFR raw residual norm) degrades performance. Sweeping \(r\) from 0 to 0.1 reveals monotonic accuracy improvements, indicating that global low frequencies must be filtered out to highlight structural changes.
The choice of pruning layer follows a strict U-shaped trajectory: Pruning at very early stages (layers 1, 3) captures sharp signals and maximizes computational savings. Mid-transformer levels (layer 10) suffer from blurred spectral dynamics, and late-level pruning (\(\ge 15\)) yields recovery in accuracy but fails to provide runtime benefits.
Strengthens under severe budgets: CLSE increases its lead under heavy pruning (e.g., under 64-token / ↓88.9% compression, its lead over FastV on the 72B model widens from 2.0 to 7.8 percentage points).

Highlights & Insights¶

Conceptual Shift: Rather than evaluating the static magnitude of a token ("how prominent is this token at this layer"), CLSE tracks representation updates across depth ("how heavily is this token being updated"). Tracking changes across layers aligns closely with multi-modal semantic integration.
Synergy of High-Pass Filters and Scale Invariance: Gaussian high-pass filtering separates fine details from raw background signals, while relative cross-layer normalization protects the scoring system from norm discrepancies between images or tokens.
Negligible Computational Overhead: FFT-based token scoring calls for mere ~108M FLOPs, representing less than 0.12% of a single layer and 0.004% of the complete LLM forward pass. While attention-based methods must fall back to \(O(N^2)\) Eager routines (precluding the use of FlashAttention kernels that avoid materializing the \(N\times N\) matrix), CLSE operates in \(O(N \log N)\) and is highly hardware-friendly.
Orthogonality: CLSE behaves strictly as a scorer, meaning it is complementary to token merging schemas (CLSE-M).
Immunity to Position Embeddings: In standard MLLMs, late-lying visual tokens are often assigned artificial high importance scores purely because of positional bias. As visualised in Fig. 6, CLSE remains completely resilient to positional bias, centering precisely on active objects or humans.

Limitations & Future Work¶

Visual-only Temporal Pruning: When handling video, aggressive hard pruning above 90% requires the CLSE-M recall framework to prevent information loss. In addition, temporal structures are captured indirectly through 2D features moving across layers, rather than through 3D spatiotemporal frequency modeling, which struggles with highly sparse frames (e.g., 8 frames).
Reshapability Constraints: Standard FFT requires tokens to be structured as an \(H \times W\) spatial grid. Handling dynamic tiling (e.g., InternVL) or non-grid visual layouts requires supplementary step-downs or separate localized normalization, which affects generalization.
Theoretical Limitations (Our Observation): Theorem 1 relies on simplified first-order Taylor approximations and a strong Assumption 2 (saliency bounded by the evolution score), which are difficult to verify in practice. The math serves more as post-hoc verification rather than an independent predictive theory.
Discrepancy in Experimental Settings (Our Observation): The main text details "pure spectral evolution hard pruning," but Appendix A.4 reveals that the implemented CLSE/CLSE-M pipelines utilize a compound score computed by multiplying the spectral evolution factor with text-to-visual attention maps, alongside adaptive budgeting. Refer to the public code repository for absolute deployment settings.
Future Directions: Exploring whether hyperparameters \(r\) and \(K\) can be adapted on-the-fly per sample, and whether cross-layer spectral evaluations can be extended toward dynamic, step-wise KV cache eviction.

vs FastV (ECCV24): FastV relies on attention weights at the 3rd layer of the LLM decoder to prune tokens. CLSE argues that single-layer attention is inherently prone to position biases and demands \(O(N^2)\) calculations. Shifting to \(O(N \log N)\) spectral evolution at the 1st layer yields superior accuracy outputs under equal token budgets (94.8% vs. 78.0% at 64 tokens).
vs SparseVLM (ICML25): SparseVLM combines text-visual attention profiling with token recycling. Substituting CLSE-M's spectral evolution scoring into the same recycling pipeline beats SparseVLM on video data (38.7 vs 38.1), demonstrating that the raw importance signal dictates the performance ceiling.
vs ToMe / PruMerge (ICLR23 / ICCV25): These works prune or group tokens based on spatial similarity. CLSE acts as an orthogonal profiling instrument that can be chained alongside similarity merging, arguing that static likeness cannot substitute for cross-layer dynamics.
vs Fourier-VLM / VTC-LFC: These methods utilize frequency low-pass filtering to compress tokens within a single layer. CLSE uses frequency transformation as a cross-layer importance metric (combining high-pass filtering with step-wise differentiation) to trace representation changes across layers.
vs V²Drop (CVPR26, Concurrent Work): V²Drop dynamically prunes "lazy tokens" displaying minimal cross-layer changes. While sharing similar motivations, V²Drop measures variations in the raw feature space, whereas CLSE operates after high-pass filtering in the frequency domain. This filtering isolates local structural changes from static backgrounds, which is essential to avoid performance collapse.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Framing token importance as a cross-layer spectral update is an elegant, well-validated concept. High-pass filtering combined with cross-layer differentiation is a novel approach in MLLM token pruning.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across more than 5 image and video models (including 72B scales and quantized checkpoints) across 9 benchmarks. The ablation studies isolate every design element, layer step, and 2D/3D FFT choice.
Writing Quality: ⭐⭐⭐⭐ Solid clarity and convincing flow of ideas. However, discrepancies in details between the main text and appendix configurations may confuse implementers.
Value: ⭐⭐⭐⭐⭐ Fully training-free, plug-and-play, with negligible profiling cost. It is orthogonal to merging-based approaches and can be adopted into existing MLLM serving frameworks.