Perceptual Projection Pruning: Diversity-Aware Video Token Pruning for Multimodal Large Language Models

Conference: ECCV2026
Authors: Zhuangqiu Huang, Minxin Lai, Shuo Liu, Yu Zhang, Jiaqi Wang
Paper: https://eccv.ecva.net/virtual/2026/poster/5817
PDF: https://media.eventhosts.cc/Conferences/ECCV2026/pdfs/13296.pdf
Area: Model Compression
Keywords: Video token pruning, perceptual saliency, spatio-temporal clustering, stratified sampling, training-free inference

TL;DR

P3 compresses the criteria for selecting video tokens into three scores for motion, regional contrast, and local complexity, then clusters within temporal segments and retains original tokens through stratified selection; on Qwen3-VL-8B, it achieves 97.3% average relative performance at 25% retention and a 4.05-fold speedup for pruning plus prefill in a specified dense-video setting.

Background & Motivation

Video inputs produce many visual tokens per frame in a multimodal large language model (MLLM), increasing both attention computation and memory demand during prefill as videos grow longer. Training-free pruning is attractive because it leaves model parameters unchanged and only reduces the visual sequence received by the language model. Yet the pruner itself takes time: repeated matching in the full high-dimensional feature space can consume the computation saved downstream. In the paper's dense-video evaluation, some high-dimensional iterative methods are indeed slower than processing the unpruned input, a problem that token counts alone would not reveal.

An alternative is to retain regions relevant to the current question. This suits focused single-turn question answering, but a selection made for one question may not remain useful for another question about the same video. Ignoring the question creates a different difficulty: the regions with the strongest motion, greatest contrast against the background, and richest textures need not contain the same tokens. A single importance ranking may preserve many examples of one evidence type while losing another. The paper therefore seeks not the most accurate general-purpose semantic similarity measure, but an inexpensive proxy that can distinguish these types of evidence.

Dimensionality reduction here serves only to decide which tokens to select; it does not replace the language model's input with three scalars. Selected tokens retain their original visual representations, while the low-dimensional space organizes candidates and allocates retention budgets. Core idea: replace high-dimensional global matching with three complementary perceptual saliency cues, then select tokens by saliency type within local spatio-temporal clusters to produce a video token subset reusable across questions.

Method

Overall Architecture

P3 sits between the vision projector output and the language model input. Perceptual saliency projection first computes three selection scores per token. Temporal segmentation then restricts downstream decisions to nearby moments. Within-cluster stratified pruning accounts for both spatio-temporal position and saliency type, and global static deduplication finally removes additional zero-displacement tokens from subsequent frames.

The pipeline neither reads the current text question nor updates the vision encoder, projector, or language model parameters. Text tokens follow the original model pathway; the visual branch only changes which original tokens survive. Here, diversity primarily means coverage of perceptual types and local spatio-temporal regions, not lossless preservation of every semantic concept.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Vision projector output"] --> B["Perceptual saliency projection"]
    B --> C["Temporal segmentation"]
    C --> D["Within-cluster stratified pruning"]
    D --> E["Global static deduplication"]
    E --> F["Original token subset<br/>and text enter the language model"]

Key Designs

1. Perceptual saliency projection: replace expensive global semantic comparisons with distinct evidence types

Dynamic saliency \(S_M\) derives from feature correspondences between consecutive frames. For a current token, the method finds the most feature-similar location in the preceding frame, treats the spatial offset as a discrete motion vector, and averages displacement magnitudes over a local spatio-temporal neighborhood. The first frame has no predecessor, so its displacement magnitude is set to zero. This is neither an additionally trained optical-flow network nor merely a feature difference at a fixed position across frames. It attempts to capture content moving across the grid, with neighborhood averaging suppressing noise from individual matches. Motion and scene changes thus provide selection cues absent from static images, although matching errors can also propagate into the scores.

Regional saliency \(S_R\) measures how different a local center is from a larger, disjoint surrounding region: it computes the mean feature of each region and takes their cosine distance. A smooth object that differs from its background can therefore receive a high regional score despite having little texture. Focal saliency \(S_F\) instead asks whether a token can be explained by its neighbors. Spatial neighbors are weighted by feature similarity to reconstruct the token, and the cosine distance between the original and reconstructed features measures unpredictability. Complex textures and details consequently receive a dedicated channel rather than relying indirectly on contrast between regional means. These axes are complementary by design; the paper does not establish their statistical independence or strict orthogonality.

Regional and focal scores also receive boundary credibility correction. Near image edges, fewer valid neighbors make estimates less stable. The method takes the logarithm of the valid neighbor count plus one and applies min–max normalization across the video. This determines the weight on the raw score, with the remaining weight assigned to the global median. Focal saliency uses the same interpolation principle. Rather than declaring boundaries unimportant, this shrinks extreme estimates from undersupported neighborhoods toward a more conservative statistic. Some displayed equations are damaged in the cached extraction, so this explanation follows the explicit prose without inventing neighborhood sizes or similarity-weight details.

2. Temporal segmentation: restrict comparisons before protecting each segment's initial state

Clustering the entire video at once could put visually similar tokens from different events into the same cluster. Subsequent sampling might then treat evidence from different moments as interchangeable redundancy. The default method uses fixed-interval segments, with uniform step 4 in the main experiments, and performs clustering and sampling independently within each segment. Rather than relying on an elaborate event detector to set every boundary, it uses inexpensive temporal locality to prevent comparisons across distant contexts.

The first frame of each segment is an anchor frame. It follows the same clustering and sampling rule but is processed separately from the other frames in that segment. Crucially, the anchor is not retained in full without conditions. Separate processing prevents a large number of similar later tokens from overwhelming it during candidate competition, preserving an interpretable state at the segment boundary. The paper also tests adaptive segmentation driven by dynamic saliency, but the main results do not depend on that variant. Its full definition is deferred to supplementary material absent from the local cache and must not be treated as an established detail of the default pipeline.

3. Within-cluster stratified pruning: group local entities before allocating slots across saliency types

Within each segment's candidate set, the method concatenates the three perceptual scores with time, vertical position, and horizontal position to form a six-dimensional vector for K-Means. Three dimensions describe the perceptual space; six dimensions describe the actual clustering input. These should not be conflated. Position coordinates provide a local-continuity prior, reducing the chance that distant tokens are grouped solely because their saliency scores are similar. The resulting clusters approximate coherent visual regions rather than semantically labeled object segments. They only need to provide more appropriate budget-allocation groups than a global ranking.

The target retention ratio determines each cluster's budget, but selection within the cluster does not simply take the tokens with the largest combined score. Each token is assigned to the type with its highest saliency score. The budget is distributed in proportion to the sizes of these three strata, and each stratum selects its highest-scoring tokens using its corresponding score. A few high-motion regions therefore cannot occupy every slot merely because their scores are large; tokens dominated by regional contrast or local complexity retain their own candidate pools. The following definition reconstructs the sampling rule described in the prose: \(C_k\) is a cluster, \(\mathcal{S}_{k,T}\) is its subset dominated by type \(T\), and \(N_k\) is the cluster budget:

\[ \mathcal{S}_{k,T}=\{i\in C_k:T=\operatorname*{argmax}_{T'\in\{M,R,F\}}S_{T'}(i)\},\qquad N_{k,T}=\left\lfloor N_k\frac{|\mathcal{S}_{k,T}|}{|C_k|}\right\rfloor. \]

The method takes \(N_{k,T}\) tokens from each stratum and combines the selections. Diversity preservation has a clear boundary here: proportional allocation does not guarantee at least one token per type, and a very small stratum may receive no slots after rounding down. The main text does not explain how rounding remainders are redistributed. Selected tokens retain their original features rather than their scores or cluster centers, so the language model still receives the high-dimensional information at those locations. The main text also does not detail scale calibration between the three scores and the coordinates; arbitrary unprocessed scales should not be assumed equivalent.

4. Global static deduplication: remove additional temporal redundancy after local selection

After merging the segment outputs, P3 filters out tokens whose time is nonzero and whose discrete motion-vector magnitude is zero. This step tests the original displacement, not whether neighborhood-averaged dynamic saliency falls below a threshold. The first frame is exempt from this filtering condition. Applying the filter last lets earlier stages determine which regions and evidence types warrant coverage, before reducing the stationary content remaining across frames instead of repeatedly retaining the same background.

Zero displacement, however, only means that the most similar location has not moved on the grid. It does not imply identical features, nor does this condition alone guarantee preservation of the first appearance after a scene cut. The paper offers a stronger interpretation in terms of static repetition; this note preserves the rule without elevating that interpretation into a guarantee. Reproduction also requires checking how global filtering interacts with later anchor frames and how the resulting token count is reconciled with the target budget. Anchors are protected during earlier separate sampling, but the main text does not fully specify these later interactions.

Loss & Training

This is a training-free inference method, with no additional loss or fine-tuning stage. The authors implement it in PyTorch and Triton and run experiments on 4 NVIDIA H100 80GB GPUs. The default segmentation step is 4, and the main experiments examine 50%, 25%, and 10% retention. Adaptive segmentation and its threshold belong to a separate diagnostic setting and should not be mixed into the default configuration. Extended parameters such as neighborhood sizes require the supplementary material for complete reproduction.

Key Experimental Results

Main Results

The following subset of Table 1 uses Qwen3-VL-8B on four video benchmarks through VLMEvalKit. All methods except Vanilla retain 25% of tokens. Higher task scores are better. Average retained performance is an aggregate percentage normalized against the unpruned model, not accuracy itself. The main text does not clearly specify the exact cross-dataset normalization and aggregation formula; the table preserves the authors' reported values.

Method	VideoMME ↑	LongVideoBench ↑	MLVU ↑	MVBench ↑	Average retained performance (%) ↑
Vanilla (100% retention)	67.8	62.5	68.6	69.3	100.0
DivPrune	64.8	60.4	64.8	64.8	95.1
PruneVid†	61.7	59.4	63.7	63.3	92.5
CDPruner	66.0	62.2	66.7	68.1	98.1
PruMerge	65.3	60.6	65.9	67.5	96.6
P3	66.1	61.3	66.2	67.2	97.3

P3 exceeds DivPrune by 2.2 percentage points in average retained performance but trails CDPruner by 0.8 percentage points. Its advantage is not universal superiority over query-guided methods on single-turn metrics. PruneVid† includes only vision-side pruning, not full PruneVid. On Qwen3-VL, PruMerge substitutes the last vision layer's average attention for the missing CLS token. These comparisons therefore use the paper's adapted protocols.

The efficiency table has a separate, explicit setting: Qwen3-VL-8B, 64 frames at 1344×768 each, 32,256 original visual tokens, and 25% retention. Total time means pruner time plus language-model prefill time, not the complete video-encoding and output-decoding pipeline.

Method	Pruner (s) ↓	Prefill (s) ↓	Total (s) ↓	Speedup over Vanilla ↑
Vanilla	Not applicable	2.23	2.23	1.00×
DivPrune	3.43	0.46	3.89	0.57×
CDPruner	0.12	0.45	0.57	3.91×
PruMerge	1.88	0.45	2.33	0.96×
P3	0.10	0.45	0.55	4.05×

Ablation Study

These component ablations from Table 10 use Qwen3-VL-8B at 25% retention with the adaptive segmentation variant, not the default uniform segmentation used above. The last column reports changes in the mean of the four raw task scores, not changes in average retained performance.

Config	VideoMME ↑	LongVideoBench ↑	MLVU ↑	MVBench ↑	Mean score change
All saliency cues	64.8	61.0	64.5	66.9	0.00
Without focal saliency	64.4	60.6	64.1	66.4	-0.43
Without regional saliency	64.1	59.5	64.2	66.4	-0.75
Without dynamic saliency	63.7	58.4	63.7	65.8	-1.40

Segmentation strategies should instead be compared using Table 9: uniform step 4 achieves a mean score of 65.2, adaptive segmentation 64.3, and uniform steps 2 and 8 achieve 64.2 and 64.6, respectively. This supports uniform step 4 as the standard default. Component ablations under adaptive segmentation do not imply that the adaptive variant is better on every task.

Key Findings

• Removing dynamic saliency causes the largest loss, while the other two cues also contribute independently. This is consistent with motion, regional contrast, and local complexity covering different evidence types.
• In Table 1, Qwen3-VL-8B still retains 92.8% relative performance at 10% retention. At 25% retention, InternVL3.5-8B achieves 94.3%, while Qwen3-VL-32B in Table 2 achieves 97.5%. A single statement of approximately 95% should not replace the precise results across models.
• With 1024 frames, P3 achieves 104.3% relative performance. Pruning and prefill take 0.53 s and 1.69 s, totaling 2.22 s against 19.57 s without pruning, or approximately an 8.8-fold speedup. This uses the VLMEvalKit and vllm setup. DivPrune is absent because of an out-of-memory failure, not because it completed evaluation and lost.

Highlights & Insights

• Preserving original representations and reducing selection cost are separate objectives. The low-dimensional proxy only determines indices, while the language model consumes original high-dimensional tokens; the method should not be understood as replacing the entire visual representation with three numbers.
• Stratified sampling protects evidence types, not just high-scoring regions. This operationalizes query-agnostic diversity while exposing the possibility that small strata disappear through rounding under tight budgets.
• Query independence allows the pruned representation of a video to be reused; it does not make subsequent questions free. Default P3 scores 62.5 on MT-Video-Bench versus 61.9 for Vanilla, but new questions still require normal language-model computation.

Limitations & Future Work

• The authors acknowledge that the three-dimensional proxy is lossy: semantically relevant details with low saliency may disappear. P3 scores 44.8 on Charades STA versus 47.9 for CDPruner. Table 8 does not explicitly define this localization metric, so it should not be relabeled as recall at a particular IoU threshold.
• The intended scope is primarily video; static images lack dynamic saliency. Results above the unpruned baseline with 1024 frames or some LLaVA configurations suggest that redundancy can distract the model, not that pruning always improves capability.
• The local cache contains the complete main paper and references but not the cited supplementary material. Exact neighborhoods, scale handling, remainder allocation, and budget accounting after global filtering still require verification. Some extracted equations are damaged, so missing details cannot be presented as verified implementation choices.

Related Work & Insights

• vs CDPruner / FastV: Query-dependent selection can exploit textual cues to locate relevant evidence. P3 does not read the question, trading some targeted retrieval ability for reuse across questions. Single-turn accuracy and amortized multi-turn costs should be reported separately because the objectives differ.
• vs DivPrune / PruMerge: High-dimensional matching retains richer feature relationships, but iterative overhead is high in the paper's dense long-sequence setting. P3's local low-dimensional decisions are cheaper. This does not establish that those baselines are also slow on short sequences or under their original implementation settings.
• vs PruneVid†: The paper compares a vision-only variant to avoid mixing in language-model modifications. Conclusions should remain scoped to that adapted configuration rather than every capability of the complete method.

Rating

• Novelty: 4/5 — The combination of low-dimensional perceptual proxies and within-cluster stratified selection is clear, but mainly heuristic.
• Experimental Thoroughness: 4/5 — Multiple models, long sequences, efficiency measurements, and component ablations are covered; protocol adaptations and some metric definitions require care.
• Writing Quality: 3/5 — The main argument is accessible, but default-versus-variant settings, budget details, and some equations need clearer specification.
• Value: 4/5 — Useful for reusable long-video representations, with appropriate emphasis on the pruner's actual runtime.