Sink-Token-Aware Pruning for Fine-Grained Video Understanding in Efficient Video LLMs¶

Conference: ECCV 2026
arXiv: 2604.20937
Code: https://github.com/rlqja1107/SToP
Area: Video Understanding / Model Compression
Keywords: Visual token pruning, sink token, fine-grained video understanding, Video LLM, training-free acceleration

TL;DR¶

This paper identifies that high-attention but semantically void "sink tokens" under attention pruning are the critical obstacle to the collapse of fine-grained video understanding. The authors propose SToP, which defines a cross-frame sink score to quantify the sink tendency of tokens. This score is integrated into spatial pruning (STSP, to lower the priority of retaining sink tokens) and temporal pruning (STTP, to increase the pruning probability of sink tokens). At an extreme retention rate of 10%, SToP significantly shrinks the performance loss on hallucination and compositional reasoning tasks (e.g., reducing the loss of FastVid from 15.69% to 6.32%, and VisionZip from 16.79% to 6.87%).

Background & Motivation¶

Background: Video LLMs demand the processing of multiple frames with hundreds of visual tokens per frame, leading to extremely high inference latency. Training-free visual token pruning (spatial pruning to retain salient tokens based on attention, and temporal pruning to merge redundant tokens based on temporal similarity) has become a mainstream acceleration approach. Existing methods (VisionZip, FastVid, Holitom, etc.) are primarily validated on multiple-choice question-answering (MCQA) benchmarks, maintaining acceptable accuracy when retaining only 10%–20% of tokens.

Limitations of Prior Work: This paper presents the first systematic evaluation of pruning methods on fine-grained video understanding tasks. It reveals that while existing frameworks are robust on MCQA, their performance collapses dramatically on tasks requiring precise visual evidence, such as hallucination detection (EventHallusion), compositional reasoning (VideoComp), and open-ended generation (VCG-Bench). For instance, at a 10% retention rate, VisionZip suffers a 16.79% performance degradation relative to the original unpruned model on these fine-grained tasks, compared to only 7.36% on MCQA. This gap severely restricts the utility of pruning methods in real-world conversational scenarios.

Key Challenge: The core assumption of attention-guided pruning is "higher attention indicates greater semantic importance." However, this work reveals a "sink phenomenon" in visual encoders, analogous to that in LLMs, where a handful of semantically empty patch tokens located in background areas consistently attract extremely high attention. Because of their elevated attention scores, these sink tokens are preferentially retained by spatial pruning. They continuously consume valuable token budgets and crowd out salient tokens that carry actual fine-grained visual details, causing the model to generate responses based on distorted or incomplete visual evidence, thereby exacerbating hallucinations.

Goal: To reveal sink tokens as the bottleneck in fine-grained video understanding and propose an explicit, plug-and-play, training-free pruning method to enhance the robustness of existing pruning frameworks on fine-grained tasks.

Key Insight: Capitalizing on the key characteristic of sink tokens—their attention consistently remains high in specific spatial positions across frames—a "sink score" is defined to quantify the sink tendency of each token. This score is then embedded as a penalty term in both spatial and temporal pruning processes.

Core Idea: Cross-frame attention persistence serves as an effective signal to distinguish sink tokens from genuinely salient tokens. Explicitly suppressing the retention priority of sink tokens via a power-sharpened sink score can substantially enhance fine-grained video understanding without introducing training costs.

Method¶

Overall Architecture¶

SToP is a training-free, plug-and-play token pruning enhancement that preserves the original architecture of pruning frameworks while introducing the sink score as a corrective signal during token selection. The overall pipeline consists of three steps. First, for each frame's attention matrix output by the visual encoder, the raw sink scores are computed by summing values across frames at each patch position, followed by power sharpening and min-max normalization. Second, this score is injected into the spatial pruning module (STSP, which subtracts a sink penalty from attention scores to lower the probability of selecting sink tokens) and the temporal pruning module (STTP, which overlays a sink bonus on inter-frame similarities to make sink tokens more likely to exceed the pruning threshold). Finally, the corrected token selection is fed into the LLM to generate responses.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["T-frame visual tokens + attention matrix A"] --> B["Sink Score Calculation<br/>Cross-frame attention summation → Power sharpening → Normalization"]
    B --> C["STSP: Spatial Pruning<br/>Ã = A − μs·s<br/>Lower attention priority of sink tokens"]
    B --> D["STTP: Temporal Pruning<br/>sim + μt·s > τ<br/>Boost pruning probability of sink tokens"]
    C --> E["Retained semantically-rich token set"]
    D --> E
    E --> F["Feed into LLM for response generation"]

Key Designs¶

1. Sink Score: Quantifying sink tendency via cross-frame attention persistence

Spatial pruning alone cannot distinguish "sink tokens" from "genuinely salient tokens" as both exhibit high attention scores within a single frame. The core insight of this paper is that the high attention of sink tokens persistently appears at the same spatial positions across frames (corresponding to static background areas), whereas the high attention of genuinely salient tokens is only briefly activated within frames where objects actually appear. Based on this, the raw sink score for patch position \(i\) is defined as the sum of attention scores across all \(T\) frames, \(\hat{s}_i = \sum_{t=1}^{T} A_i^t\). The final sink score \(s_i = \text{MinMax-Norm}(\hat{s}_i^w)\) is obtained after a power transformation and min-max normalization. The power exponent \(w\) (default 1.1) acts as a sharpening term: when \(w > 1\), it widens the gap between high-sink and low-sink tokens, concentrating the penalty on true sink tokens to prevent mispenalizing borderline salient tokens.

2. STSP: Sink-aware spatial pruning by subtracting a sink penalty from attention scores

Conventional spatial pruning methods (e.g., VisionZip, FastVid) select tokens based on descending attention scores. The proposed STSP module directly alters the attention scores as: \(\tilde{A}_i^t = A_i^t - \mu_s \cdot s_i\), where \(\mu_s\) controls the penalty strength. Tokens with high sink scores have their priorities systematically lowered and are no longer incorrectly retained due to inflated attention. The key to this formulation is that the sink score is a global metric aggregated across frames. It does not rely on the absolute attention value within a single frame, allowing STSP to distinguish between "normal tokens with high attention in a few frames due to salient objects" and "sink tokens with persistent high attention but no semantic meaning across all frames." The former remains low after cross-frame summation, thus avoiding the penalty. This acts as a persistence prior in attention sorting: brief salience represents true salience.

3. STTP: Sink-aware temporal pruning to facilitate merging of sink tokens

Temporal pruning (e.g., Holitom) identifies redundancy by computing the similarity of tokens at the same location between adjacent frames. Tokens with similarity exceeding a threshold \(\tau\) are merged as redundant. A side effect of this is that sink tokens (corresponding to background regions with minimal inter-frame changes) naturally exhibit high similarity and are easily pruned. This explains why joint temporal-spatial pruning is inherently more robust than spatial-only pruning. However, some sink tokens still survive temporal pruning. STTP explicitly utilizes the sink score to increase the pruning probability of these surviving tokens: \(P_t = \{i \mid \text{sim}(H_v^{i,t}, H_v^{i,t+1}) + \mu_t \cdot s_i > \tau\}\). With the addition of \(\mu_t \cdot s_i\), the scores of sink tokens are shifted upwards, making it easier for them to exceed \(\tau\) and merge.

A complete example: The fate of a background sink token¶

Consider a background region located in the bottom-left corner (patch index 154) of a video—it consists of a static white wall and carries no semantic information. In a standard ViT encoder, its CLS attention score exceeds 0.8 across almost all frames, as it acts as a "visual anchor" that attracts substantial attention. Under purely spatial pruning (e.g., VisionZip), this patch is selected in every frame due to its high attention. At a 10% retention rate, it occupies a budget of 3.2 tokens (among 32 frames competing across 32 patch positions, it wins in almost every frame), pushing out crucial action details. With SToP, the patch's sink score \(s_{154}\) is pushed close to 1.0 due to its persistent cross-frame attention. STSP dampens its single-frame attention from 0.85 to \(0.85 - 0.03 \times 1.0 = 0.82\), placing it below the retention threshold. STTP further adjusts its inter-frame similarity (originally 0.92, which was close to but below \(\tau\)) by adding \(0.07 \times 1.0 = 0.07\), resulting in 0.99, which far exceeds the threshold and triggers temporal merging. Consequently, tokens at this position across the 32 frames are significantly pruned, releasing budget for foreground tokens that capture action changes, thereby visibly reducing hallucinations.

Loss & Training¶

SToP is a training-free method with no loss function. Its three key hyperparameters are determined via a greedy search on a validation set: power exponent \(w=1.1\) (fixed globally), spatial penalty coefficient \(\mu_s = 0.03\) for VisionZip and \(0.02\) for FastVid, and temporal enhancement coefficient \(\mu_t = 0.07\) for Holitom. Ablation studies show that performance is highly insensitive to \(w\) (stable within 1.0–1.15) but more sensitive to \(\mu_s\), where excessively large values may inadvertently penalize borderline normal tokens.

Key Experimental Results¶

Main Results¶

Evaluation is conducted on LLaVA-OneVision-7B at an extreme token retention rate of 10% (90% pruned). The table below lists the core results on fine-grained tasks.

Method	Retention Rate	EventHallusion Binary	EventHallusion Desc	VideoComp Act	VideoComp YC	Performance Loss Rate
Vanilla (No Pruning)	100%	63.33	38.74	70.06	70.95	-
Holitom	10%	60.88	36.75	68.38	68.98	6.22%
Holitom+SToP	10%	62.59	39.74	69.24	69.10	4.80%
FastVid	10%	49.63	31.46	57.37	58.10	15.69%
FastVid+SToP	10%	60.15	36.75	68.38	69.75	6.32%
VisionZip	10%	50.37	28.15	65.39	64.61	16.79%
VisionZip+SToP	10%	60.39	36.09	67.78	68.89	6.87%

Performance Loss Rate = macro-average performance degradation of all metrics relative to Vanilla. SToP provides the most substantial improvements for spatial-only methods (FastVid and VisionZip), narrowing their performance loss by 9.37pp and 9.92pp respectively. It also yields a 1.42pp gain for the temporal-spatial method Holitom, as temporal pruning already implicitly suppresses a subset of sink tokens. Consistent gains are validated across visual backbones (LLaVA-Video-7B) and datasets (Argus). SToP also improves performance on MCQA benchmarks (e.g., boosting VisionZip's MVBench score from 54.16 to 57.34 at 10% retention), though the baseline degradation on MCQA is already quite mild, further confirming that MCQA benchmarks fail to expose the vulnerabilities of token pruning.

Ablation Study¶

Evaluation of the individual contributions of STSP and STTP at a 10% retention rate.

Configuration	STSP	STTP	EventHallusion Binary	EventHallusion Desc	VideoComp Act	VideoComp YC	Performance Loss Rate
Vanilla	-	-	63.33	38.74	70.06	70.95	-
VisionZip (Spatial-only)			50.37	28.15	65.39	64.61	20.46%
VisionZip + STSP	✓		60.39	36.09	67.78	68.89	4.64%
Holitom (Temporal+Spatial)			60.88	36.75	68.38	68.98	3.87%
Holitom + STSP	✓		62.10	37.09	68.02	69.24	1.94%
Holitom + STSP + STTP	✓	✓	62.59	39.74	69.24	69.10	1.17%

Adding STSP to the spatial-only method VisionZip sharply reduces the performance loss from 20.46% to 4.64%, demonstrating the potent corrective capability of the sink penalty on pure attention-based pruning. While Holitom inherently possesses some robustness, adding STSP still yields a 1.93pp improvement, and incorporating STTP achieves optimal performance. This indicates that implicit sink suppression in temporal pruning is insufficient, and explicit sink-aware temporal pruning further purges residual sink tokens.

Key Findings¶

Spatial-only methods benefit the most: Adding SToP to VisionZip and FastVid at 10% retention narrows their performance loss by 9.92pp and 9.37pp, respectively. This demonstrates that sink tokens are the primary bottleneck responsible for the collapse of pure attention-guided pruning on fine-grained tasks, and SToP effectively "fixes" this limitation.
16 frames + SToP outperforms 64 frames without SToP: On EventHallusion, VisionZip+SToP with only 16 frames outperforms the original VisionZip using 64 frames. This highlights that suppressing sink tokens is more effective than simply stacking more frames, as extra frames introduce more sink tokens, offsetting any visual information gains.
Hard pruning baseline is effective but inferior to SToP: Simply discarding the top-K% attention tokens per frame (Hard Pruning) also improves performance. However, SToP leverages the sink score to distinguish "high-attention but semantically void sinks" from "high-attention but genuinely salient tokens," avoiding erroneous pruning.
Feature clustering naturally bypasses sinks but remains imperfect: Feature-clustering-based pruning (e.g., using DPC-KNN) does not rely on attention and outperforms original attention pruning at a 10% retention rate. Nevertheless, SToP-enhanced attention pruning surpasses feature clustering, proving that "directly addressing the sink issue" is superior to "circumventing it."

Highlights & Insights¶

Cross-modal transfer insights of the sink phenomenon: In LLMs, attention sink tokens (e.g., BOS) must be retained to maintain softmax stability. Conversely, visual encoders present the exact opposite case—their sink tokens must be pruned to release token budgets. This deep distinction in behavior represents a highly elegant theoretical contribution of this paper, dictating the design direction.
The "hidden function" of temporal pruning: This work exposes an interesting byproduct: though designed to remove inter-frame redundancy, temporal pruning naturally serves as a sink token suppressor because static background regions exhibit extremely high inter-frame similarity. This explains why Holitom outperforms VisionZip and FastVid, and offers a fresh perspective for future pruner design: spatial and temporal pruning dimensions present strong synergies and should not be designed in isolation.
Power sharpening is a cheap and versatile distribution shaping trick: By adjusting the power exponent \(w\) from 1.0 to 1.1 in \(s_i = \text{MinMax-Norm}(\hat{s}_i^w)\), a slowly decaying score distribution is shaped into a steep distribution where only a few tokens approach 1.0 while most are near 0. This enables the penalty to target only the true sink tokens. This trick can be easily transferred to other tasks requiring the extraction of hard subsets from continuous scores, such as confidence-based pseudo-label filtering, hard negative mining, or attention map thresholding.
Convincing experimental design: From diagnostic experiments (observing anomalous token selection frequency distributions after removing temporal pruning) to causal validation (demonstrating that randomly removing sink tokens drops hallucinations), through to methodology and ablation studies, the paper establishes a thorough "observation \(\rightarrow\) hypothesis \(\rightarrow\) causal validation \(\rightarrow\) method \(\rightarrow\) ablation" chain of evidence. This serves as a model of rigorous experimental design.

Limitations & Future Work¶

Inapplicability to non-attention pruning methods: SToP's sink score depends on attention matrices and cannot be directly applied to feature-clustering-based pruning methods (e.g., DPC-KNN). The authors suggest migrating similar concepts to the feature space in future work—such as identifying "feature-space sink tokens" with high norms and high pairwise similarities.
Sensitivity of \(\mu_s\) and \(\mu_t\) requiring separate tuning: Different baseline methods exhibit varying sensitivity to sinks (e.g., pure attention methods vs. temporal-assisted methods). Hyperparameters must be greedily searched for each combination, which increases engineering deployment costs. A potential enhancement is to make the sink score dynamically adjust the penalty intensity based on the token budget.
Heuristic thresholds for sink token definition: The current approach identifies the sink token set using the "top-K% selection frequency" heuristic, lacking a principled determination criterion. Automated sink detection based on information theory could be explored in future work.
Evaluations restricted to ViT encoders: All experiments were conducted using the SigLIP visual encoder. Whether the sink phenomenon is universal across other encoder architectures (e.g., VideoMAE, InternVideo) remains to be investigated.

vs. VisionZip / FastVid (Spatial-only pruning): These methods select tokens directly based on CLS attention scores, implicitly assuming "high attention = importance." This paper exposes the fundamental flaw of this assumption in fine-grained tasks—sink tokens show high attention but are unimportant—and corrects the attention scores via the sink score. This essentially upgrades "single-frame attention sorting" to "cross-frame persistence-aware attention sorting."
vs. Holitom / PruneVid (Temporal-spatial pruning): These methods perform temporal pruning via inter-frame similarity, which implicitly suppresses sink tokens, though their original intent is redundancy removal rather than sink removal. SToP explicitly elevates this "side effect" into a core design objective, overlaying a sink bonus in temporal pruning to eliminate residual sinks.
vs. Attention Sink related works (LLM side): Research on attention sinks in LLMs (Xiao et al. 2023, Sun et al. 2024) stresses the preservation of sink tokens to maintain softmax stability. Work on the ViT side (Darcet et al. 2023, Jiang et al. 2025) primarily focuses on the impact of sinks on interpretability. This work is the first to connect the sink phenomenon with the fine-grained understanding performance of visual token pruning, taking a completely opposite design direction to LLMs (pruning rather than retaining).

Rating¶

Novelty: ⭐⭐⭐⭐⭐ This paper is the first to prove that sink tokens are the key bottleneck for the collapse of fine-grained understanding in visual token pruning, and its comparative insight ("visual sinks should be pruned, linguistic sinks should be kept") is highly profound.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across 3 fine-grained benchmarks, 5 MCQA benchmarks, 2 backbones, and cross-dataset testing. The chain of diagnostic and ablation experiments is highly complete, comparing multiple alternatives such as Hard Pruning, Attention Redistribution, and feature clustering.
Writing Quality: ⭐⭐⭐⭐ The motivational chain is exceptionally clear (MCQA masks issues \(\rightarrow\) fine-grained tasks expose failures \(\rightarrow\) sink diagnostics \(\rightarrow\) causal validation \(\rightarrow\) method \(\rightarrow\) ablation). The appendix analysis of LLM vs. vision sink tokens is a major plus, though some experimental tables are highly dense, slightly sacrificing readability.
Value: ⭐⭐⭐⭐⭐ A training-free, plug-and-play core method implemented with only three simple formulas that can be seamlessly integrated into almost any attention-based pruning framework. The concept of the sink score is highly likely to inspire follow-up works in fine-grained video understanding.