SWIFT: Spatial-Window Integrated Frequency-aware Token Pruning for Efficient MLLMs on Edge Devices¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/damo-lgl/SWIFT
Area: Model Compression
Keywords: Multimodal Large Language Models, Token Pruning, Frequency-aware Residual, Spatial Window, Edge Inference
TL;DR¶
Addressing the issues of high-frequency detail loss and spatial collapse caused by conventional attention-based token pruning, SWIFT extracts frequency-domain residuals via matrix factorization and enforces local spatial window regularization, achieving near-lossless multimodal reasoning with up to 75% compression and a 2.66× decoding speedup on edge devices.
Background & Motivation¶
Multimodal Large Language Models (MLLMs) deliver remarkable performance across fine-grained image understanding and multimodal reasoning tasks. However, high-resolution imagery produces thousands of visual tokens, resulting in quadratic (\(O(N^2)\)) computational overhead in self-attention that creates severe memory bandwidth bottlenecks and high Time to First Token (TTFT) on resource-constrained edge devices. To alleviate visual token redundancy, existing compression schemes are largely divided into training-based and training-free paradigms. Training-based techniques introduce parameterized resamplers or fine-tuning pipelines that incur prohibitive retraining overhead and lack adaptability to changing compute budgets; conversely, training-free approaches (such as FastV and SparseVLM) predominantly rely on the heuristic assumption that "attention weight equals token importance", pruning visual tokens based strictly on Softmax attention scores.
Examined from a signal processing perspective, self-attention in deep Transformer layers inherently behaves as a strong low-pass filter. As network depth increases, the attention matrix gradually degenerates into an uninformative uniform averaging matrix, causing smooth low-frequency direct current (DC) components (such as background regions) to dominate while heavily attenuating high-frequency alternating current (AC) components (such as object contours and detailed textures). Compounding this issue, attention mechanisms frequently suffer from "attention sinks", where disproportionate attention weights are allocated to uninformative tokens. Pruning tokens based solely on attention weights inadvertently discards critical high-frequency features, accelerating feature over-smoothing. Furthermore, unconstrained global pruning disrupts the two-dimensional spatial continuity of visual tokens, producing localized perceptual blind spots that trigger catastrophic object hallucinations.
To maintain robust perceptual fidelity under edge-level compute constraints, MLLMs require a compression paradigm that breaks free from attention-score bias by preserving high-frequency features and enforcing spatial continuity. Core idea: reconstruct visual token pruning as a dual-stream mechanism of frequency-domain saliency and local spatial regularization, extracting high-frequency residuals via low-overhead attention matrix factorization without FFT, and performing independent local Top-K selection across 2D spatial windows to simultaneously prevent feature over-smoothing and spatial structure collapse.
Method¶
Overall Architecture¶
SWIFT is an inference-time, training-free, plug-and-play token compression framework that inserts seamlessly into early LLM decoder layers (specifically between layers 2 and 3) during the prefill phase. The framework operates in three primary stages: first, it decomposes the layer's self-attention matrix to strip away the uniform low-frequency base and calculates a Frequency-Aware Indicator (FAI) for each visual token; second, it reshapes the 1D sequence into a 2D spatial grid and blends the frequency indicator with Attention Guidance (AG) via a hybrid ranking formula; third, it applies Spatial-Window Integration (SWI) to execute independent Top-K token selection within each local spatial window, safeguarding both fine-grained texture retention and global field-of-view continuity before passing the pruned visual sequence to subsequent layers.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Visual Tokens X_v and Attention Matrix A"] --> B["Frequency-Aware Indicator (FAI): Factorization Stripping Low Frequencies"]
B --> C["Hybrid Scoring: Blending Frequency Residuals and Attention Ranks"]
C --> D["Spatial-Window Integration (SWI): 2D Grid Partitioning & Local Top-K Retention"]
D --> E["Output Compact Visual Token Sequence"]
Key Designs¶
1. Frequency-Aware Indicator (FAI): Isolating High-Frequency Residuals via Low-Rank Matrix Factorization Standard 2D Fast Fourier Transforms (FFT) impose non-trivial latency unsuitable for millisecond-level inference loops. To capture high-frequency components with negligible overhead, SWIFT adopts an algebraic approximation based on attention matrix factorization. Noting that the extreme low-pass filter corresponds to an entirely uniform matrix, SWIFT decomposes the attention matrix \(A\) into a uniform low-frequency base \(A^{LP}\) and a high-frequency interaction residual \(A^{HP}\): $\(A^{LP} = \frac{1}{n} \mathbf{1}\mathbf{1}^T, \quad A^{HP} = A - A^{LP}\)$ where \(n\) is the sequence length and \(\mathbf{1}\mathbf{1}^T\) is an all-ones matrix. For the \(k\)-th visual token, its global contribution to high-frequency components, \(I_{freq}(k)\), is defined as the mean of the \(k\)-th column of \(A^{HP}\): $\(I_{freq}(k) = \frac{1}{n} \sum_{q=1}^{n} A_{q,k}^{HP}\)$ This score directly captures the token's deviation from a uniform distribution: a positive value (\(I_{freq}(k) > 0\)) signifies an over-attended token carrying salient high-frequency information (such as edges, textures, or foreground boundaries), whereas a negative value (\(I_{freq}(k) < 0\)) indicates an under-attended token belonging to flat backgrounds. Computed using simple column summations and scalar subtractions, FAI requires only \(\sim 0.5\) ms of GPU compute, serving as an ultra-lightweight anchor against deep feature over-smoothing.
2. Hybrid Scoring Mechanism: Complementary Coupling of Semantic Focus and High-Frequency Detail Sole reliance on frequency indicators can over-prioritize complex yet task-irrelevant textured backgrounds, while pure attention scores remain vulnerable to attention sinks and neglect fine-grained geometric anchors. To strike an optimal balance between high-level semantics and low-level spatial detail, SWIFT introduces a rank-normalized hybrid scoring metric: $\(\mathrm{Score}(x) = (1 - \lambda) \cdot \mathrm{Rank}(I_{freq}(x)) + \lambda \cdot \mathrm{Rank}(I_{attn}(x))\)$ where \(I_{attn}(x)\) denotes the self-attention weights extracted from early LLM layers (e.g., layer 2), \(\mathrm{Rank}(\cdot)\) maps raw scores to ordinal ranks to eliminate scaling disparities, and \(\lambda\) is a balancing hyperparameter. Here, the attention score dictates "where to look" (semantic localization), while the frequency score ensures "what to see clearly" (fine-grained detail preservation). Empirical evaluations confirm optimal performance at \(\lambda \approx 0.5\), demonstrating that semantic relevance and spectral saliency provide balanced, mutually beneficial guidance.
3. Spatial-Window Integration (SWI): Local Window Retention Preventing Global Structural Collapse Unconstrained global sorting frequently clusters retained tokens around isolated high-scoring objects, completely purging other visual regions and creating "spatial blind spots" that trigger severe object hallucinations. SWIFT remedies this with a spatial window regularization strategy: it first remaps the 1D token sequence back to a 2D coordinate grid based on original positional indices, partitions the grid into non-overlapping local spatial windows (e.g., \(2 \times 2\) or \(4 \times 4\) sub-regions), and conducts independent Top-K selection within each local window according to the hybrid score. This mandatory local retention guarantees uniform spatial coverage across the entire image canvas, preserving topological relationships and effectively curbing hallucinations induced by missing receptive fields.
Key Experimental Results¶
Main Results¶
SWIFT was comprehensively evaluated across 8 diverse multimodal benchmarks (MMB, MMStar, MME, OKVQA, POPE, MMMU, Nocaps, Flickr30k) on LLaVA-1.5 (7B/13B) and Qwen2.5VL-3B. At a practical 50% compression ratio, it preserves full model fidelity, and under an aggressive 75% pruning ratio, it substantially outperforms existing SOTA baselines.
| Model & Method | Pruning Ratio (Tokens) | MMB | MMStar | MME | POPE | MMMU | Nocaps | Flickr30k |
|---|---|---|---|---|---|---|---|---|
| LLaVA-1.5-7B | ||||||||
| Vanilla (Upper Bound) | 0% (576) | 64.1 | 33.8 | 1610 | 86.4 | 36.3 | 105.6 | 84.4 |
| FastV | 50% (288) | 64.2 | 33.3 | 1599 | 83.7 | 35.3 | 104.4 | 84.3 |
| SWIFT (Ours) | 50% (288) | 64.5 | 34.0 | 1607 | 85.2 | 35.5 | 105.4 | 84.5 |
| FastV | 75% (144) | 57.9 | 32.1 | 1588 | 75.3 | 35.2 | 99.2 | 79.1 |
| SWIFT (Ours) | 75% (144) | 62.6 | 33.0 | 1607 | 81.4 | 34.7 | 102.7 | 83.3 |
| FastV | 90% (58) | 50.6 | 30.6 | 1448 | 66.3 | 35.1 | 74.7 | 55.1 |
| SWIFT (Ours) | 90% (58) | 57.4 | 31.1 | 1540 | 75.3 | 35.4 | 89.1 | 65.0 |
| Qwen2.5VL-3B | ||||||||
| Vanilla (Upper Bound) | 0% (900) | 77.8 | 56.4 | 1918 | 87.0 | 47.6 | 105.8 | 85.7 |
| FastV | 50% (450) | 76.3 | 54.4 | 1848 | 87.3 | 46.5 | 103.8 | 83.4 |
| SWIFT (Ours) | 50% (450) | 77.3 | 55.4 | 1913 | 86.8 | 46.9 | 104.7 | 84.3 |
| FastV | 75% (225) | 71.4 | 48.6 | 1809 | 84.3 | 46.3 | 97.0 | 77.0 |
| SWIFT (Ours) | 75% (225) | 74.0 | 53.4 | 1843 | 84.7 | 45.3 | 100.4 | 80.0 |
On hardware efficiency benchmarks conducted on an NVIDIA RTX 2080 (Qwen2.5VL-3B with \(840 \times 840\) inputs): - 50% Pruning: TTFT decreases by 14.9%, KV cache memory footprint drops by 46.05%, and single-step decoding latency reduces from 7.74 ms to 3.69 ms (2.10× speedup). - 75% Pruning: TTFT drops from 2.61 s to 2.01 s (23.0% improvement), KV cache footprint decreases by 69.07%, and decoding latency drops to 2.91 ms (2.66× speedup). The overhead introduced by the pruning modules constitutes only 1.72% of total TTFT, with +0 MB peak VRAM increase.
Ablation Study¶
Ablations on Qwen2.5VL-3B analyze the individual contributions of Spatial-Window Integration (SWI), the Frequency-Aware Indicator (FAI), and Attention Guidance (AG) across multiple pruning ratios:
| Pruning Ratio | Config | SWI | FAI | AG | MME | POPE | Nocaps | Flickr30k |
|---|---|---|---|---|---|---|---|---|
| 0% | Baseline Full Model | - | - | - | 1918 | 87.0 | 105.8 | 85.7 |
| 50% | w/o SWI (Global Sort) | ✗ | ✓ | ✓ | 1836 | 83.3 | 103.7 | 83.2 |
| w/o FAI (No Frequency) | ✓ | ✗ | ✓ | 1890 | 83.8 | 105.1 | 83.5 | |
| w/o AG (No Attention) | ✓ | ✓ | ✗ | 1900 | 84.2 | 104.5 | 83.6 | |
| SWIFT Full Model | ✓ | ✓ | ✓ | 1913 | 86.8 | 104.7 | 84.3 | |
| 75% | w/o SWI (Global Sort) | ✗ | ✓ | ✓ | 1708 | 70.9 | 68.6 | 63.3 |
| w/o FAI (No Frequency) | ✓ | ✗ | ✓ | 1795 | 77.6 | 94.8 | 74.1 | |
| w/o AG (No Attention) | ✓ | ✓ | ✗ | 1817 | 73.7 | 98.1 | 73.9 | |
| SWIFT Full Model | ✓ | ✓ | ✓ | 1843 | 84.7 | 100.4 | 80.0 |
Key Findings¶
- SWI is essential to prevent collapse under high compression: Without SWI, POPE hallucination performance drops drastically from 84.7 to 70.9 at 75% compression, and Nocaps falls precipitously from 100.4 to 68.6. This confirms that global sorting creates localized blind spots that deprive the model of critical context.
- FAI mitigates deep feature over-smoothing: Spectral energy profiling reveals that FastV experiences severe attenuation in higher frequency bands (\(> 0.25\pi\)), while SWIFT preserves substantial mid-to-high frequency power. FAI provides notable gains on detail-intensive captioning tasks (Nocaps, Flickr30k), showing its importance for preserving slender objects, text edges, and textures.
- Robustness of weighting hyperparameter: Sweeping \(\lambda \in [0, 1]\) demonstrates that performance consistently peaks at \(\lambda \approx 0.5\) across multiple model architectures and input resolutions, highlighting the fundamental complementarity of high-level semantic attention and low-level frequency residual signals.
Highlights & Insights¶
- Reframing Self-Attention as a Low-Pass Filter: Moves beyond heuristic attention assumptions by theoretically showing that self-attention acts as a low-pass filter causing feature over-smoothing, establishing a rigorous signal-processing foundation for token compression.
- FFT-Free Residual Approximation: Elegantly isolates high-frequency deviations by subtracting a normalized all-ones base matrix from the attention matrix, achieving detail-aware pruning with only \(\sim 0.5\) ms runtime overhead and zero learnable parameters.
- Spatial Window Regularization against Hallucination: Directly tackles the spatial blind-spot pathology of token pruning via structured local window Top-K selection, preserving overall topological geometry and effectively suppressing object hallucinations.
Limitations & Future Work¶
- Fixed Grid Partitioning: The spatial window dimensions are currently statically predefined; developing dynamic, content-adaptive window partitioning could better accommodate non-standard aspect ratios and dense micro-object layouts.
- Single-Stage Pruning: SWIFT currently compresses tokens once between layers 2 and 3; progressive multi-stage pruning across intermediate transformer blocks warrants further investigation for even deeper sequence reduction.
- Extension to Multi-Image and Video Domains: The current framework is evaluated on single-image benchmarks; extending spatial windowing to spatiotemporal 3D volumes to balance motion dynamics with spatial textures represents a promising direction.
Related Work & Insights¶
- vs FastV [Chen et al., 2024]: FastV prunes visual tokens using global attention scores; SWIFT exposes the low-pass bias of pure attention and combines FAI high-frequency residual tracking with SWI window constraints, outperforming FastV by 3-6 points under 75% compression.
- vs SparseVLM [Zhang et al., 2024]: SparseVLM relies on text queries to guide token filtering; SWIFT demonstrates that self-attention frequency residuals coupled with geometric windowing achieve superior performance without requiring complex cross-modal scoring.
- vs ToMe [Bolya et al., 2022]: ToMe merges tokens based on cosine similarity, whose iterative averaging further accentuates low-pass filtering; SWIFT adopts hard selection of high-frequency residuals to preserve edge fidelity.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ [Offers an insightful frequency-domain interpretation of self-attention over-smoothing and introduces an ultra-lightweight matrix factorization proxy.]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across 3 model scales, 8 diverse benchmarks, spectral energy analyses, and on-device hardware profiling.]
- Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, rigorous mathematical formulation, and well-structured comparative narrative.]
- Value: ⭐⭐⭐⭐⭐ [Highly practical for edge multimodal deployments, achieving 2.66× speedup and 69% KV cache reduction with negligible computational overhead.]