AnchorPrune: Relevance-Anchored Contextual Expansion for Visual Token Pruning¶
Conference: ECCV2026
Paper: Official page · PDF
Code: https://github.com/MULTI-cau/AnchorPrune
Area: Vision-Language Model Efficiency
Keywords: Visual token pruning, relevance anchors, adaptive budgets, contextual expansion, training-free inference
TL;DR¶
AnchorPrune separates visual token selection into the ordered stages of protecting query-relevant evidence and adding important, non-redundant context, requiring no training and retaining 97.6% of full-model average relative performance on LLaVA-NeXT-7B with only 160/2880 tokens.
Background & Motivation¶
High-resolution, multi-crop, and video inputs send large numbers of visual tokens to the language model, increasing attention computation and memory consumption during prefilling. Pruning appears to be a matter of selecting a small subset, but the evidence needed to answer a question is not necessarily the most salient, distinctive, or globally attended part of an image. A small object, several adjacent text fragments, or a local spatial relation may look unremarkable while determining whether the answer is correct.
Existing approaches use saliency, query relevance, or feature diversity. Relevance-only selection can spend the budget on correlated local regions, whereas diversity-only selection can fragment text and local details that must survive together. Methods such as CDPruner combine relevance and diversity in a unified objective, but this competition can still allow a more novel candidate to displace indispensable evidence. Moreover, when the query signal is unreliable, reusing that same relevance signal for the remaining tokens may fail to recover missing context.
The paper therefore changes the selection order rather than introducing a more elaborate scorer: evidence directly supporting the answer should not face the same elimination rules as replaceable background information. Core idea: first establish a relevance anchor that later selections cannot replace, then fill the remaining budget using global-importance-weighted novelty, separating what must be protected from what is still missing.
Method¶
Overall Architecture¶
The inputs are visual tokens from an image or video, a text instruction, and a total retention budget \(K\); the output is a sequence of \(K\) tokens restored to their original visual order. Architecture-aware priority scoring is followed by adaptive anchor budgeting based on the novelty profile of the relevance-ranked sequence, and then importance-weighted contextual expansion. The language model's weights and main architecture remain unchanged.
An anchor here is a protected subset of original visual tokens, not a newly generated summary vector. Contextual expansion only adds tokens to the retained set: it does not delete or replace anchor tokens. Consequently, correlated local evidence cannot be removed by a diversity competition in the second stage.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
input["Visual tokens + instruction<br/>Total budget K"] --> priority["Architecture-aware<br/>priority scoring"]
priority --> anchor["Adaptive anchor<br/>budgeting"]
anchor --> context["Importance-weighted<br/>contextual expansion"]
input -->|Encoder attention| context
context --> output["Restore visual order<br/>Unchanged language model"]
Key Designs¶
1. Architecture-aware priority scoring: identify query evidence worth protecting
The method does not assume that every VLM supports image-text similarity in the same feature space. For models with paired CLIP vision and text encoders, scoring happens before the multimodal projector. Visual patches are mapped to comparable visual embeddings and normalized, while the instruction is encoded by the paired text encoder. Instructions exceeding the text encoder's context length are segmented, and patch-text similarities are averaged across segments. An easily missed detail is that the authors negate this average similarity and rank the resulting scores in descending order, rather than retaining patches with the highest positive similarity.
The paper explains that local CLIP similarities can respond more strongly to background regions, and reversing the maps can improve localization. The score is therefore a heuristic anchoring priority, not a calibrated probability of semantic relevance. The comparison between positive and negative similarity directions is deferred to supplementary material absent from the local cache; the main paper does not establish that reversal is better for every input.
For non-CLIP models without a paired pre-projector image-text space, scoring moves after the multimodal projector. Each projected visual token is compared, using normalized similarity, against individual language-model embeddings of the instruction tokens, and the maximum match becomes its priority. A region that strongly matches one semantic component of the instruction is thus not diluted by averaging the whole instruction. This is explicitly different from the CLIP branch's segment averaging followed by negation.
Once the ranking is available, the anchor is a prefix of that ranking, without an additional diversity filter inside the prefix. This matters because similar patches may jointly encode a complete name or essential attributes of one object. Requiring them to be different during the protection stage would reintroduce evidence fragmentation.
2. Adaptive anchor budgeting: inspect novelty events along the relevance ranking
The anchor does not receive a fixed fraction of the budget. The method first retains the top \(K_{\min}\) tokens as an initial anchor and sets the maximum anchor length \(K_{\max}\) to \(K/2\) rounded down, leaving at least half the total budget for context. It then inspects the relevance-ranked candidates after the initial anchor, up to this upper bound, to determine whether they are sufficiently novel relative to the initial anchor.
Two details are essential. First, every candidate in this budgeting step is compared with the fixed initial anchor, not an expanding ranking prefix. Second, the patience parameter \(P\) counts the cumulative number of candidates whose novelty exceeds threshold \(\tau\). When the \(P\)-th novelty event occurs, that candidate's ranking position becomes the final anchor length. If fewer than \(P\) events occur before the upper bound, the method conservatively uses the maximum anchor length. This is not the familiar early-stopping rule of terminating after \(P\) consecutive redundant tokens.
The retained anchor is the entire ranking prefix up to the stopping position, not just the tokens that triggered novelty events. Novelty therefore determines how much budget to allocate to relevant evidence, rather than directly eliminating individual highly ranked tokens. This distinction explains how the method can use novelty information while still protecting correlated local evidence.
Equations (3), (14), and (22) contain mathematical extraction damage in the cache, so their complete novelty-distance expression cannot be recovered unambiguously. This note preserves only the meaning established by the surrounding text: compare a candidate's visual features with selected evidence to assess information not yet covered. It does not supply missing operators or substitute ordinary cosine similarity for the paper's novelty measure. Reproduction requires checking the original formulas; the main text also does not specify numerical values for \(K_{\min}\), \(\tau\), or \(P\).
3. Importance-weighted contextual expansion: add useful new information, not difference alone
The second stage first computes a global importance prior distinct from the query-conditioned anchoring priority. If the vision encoder has a dedicated CLS token, the prior is its attention to each visual token, averaged across attention heads. Without CLS, the method measures the attention mass each token receives from all query positions, averaging over heads and positions. If the encoder subsequently merges several tokens into one candidate, their pre-merge priors are averaged so that importance aligns with the actual selection units.
The retained set starts with the complete protected anchor. At each iteration, the method selects the remaining candidate with the largest product of global importance and novelty relative to the current retained set, adding candidates until the set contains \(K\) tokens. Unlike anchor budgeting, this reference set changes over time and includes both the anchor and previously added context. A candidate that was initially useful may become redundant after similar evidence has been selected.
Multiplication requires both usefulness and missing coverage: a visually distinctive region with negligible global importance cannot easily take budget on distinctiveness alone, and a salient region already represented by the anchor should not be selected repeatedly. Crucially, the weight is the global importance prior, not the first-stage query priority reused in the expansion objective. This distinction is what the controlled ablation examines.
Finally, the selected tokens are restored to their native visual order and passed to the unchanged language model. The procedure selects original tokens rather than learning a compressed representation. Training-free does not mean cost-free: priority scoring, access to encoder attention, and iterative novelty updates still introduce selection overhead.
A Worked Example¶
Figure 2 illustrates the question "What is the name of the runner on the left?" using LLaVA-1.5-7B with 64 of 576 visual tokens retained; the illustrated answer is "Willis." The method establishes an instruction-prioritized anchor before adding context useful for locating the person and interpreting text. Similar patches around the name cannot be exchanged out of the anchor during the second stage merely because they are correlated.
For this total budget, the anchor cap is 32, leaving at least 32 contextual positions. However, the illustration does not report the actual anchor length, threshold, or selected patches at each iteration. The cap should not be presented as the sample's measured stage split. The example illustrates protection followed by expansion, not a fully reproducible selection trace.
Loss & Training¶
There is no additional training objective or retraining of the vision encoder, projector, or language model. Experiments match model checkpoints, input processing, decoding settings, evaluation protocols, and retained-token budgets, using lmms-eval where supported. Architecture-specific implementation, hyperparameters, and hardware details are deferred to supplementary material that is not present in the current cache.
Key Experimental Results¶
Main Results¶
Rel. first divides each benchmark score by the full-token score of the same backbone, then averages over the metrics included in that table and expresses the result as a percentage. It is neither raw accuracy nor a common score for direct cross-backbone comparison. The following table selects the smallest evaluated budget for each backbone from the paper's Tables 1 through 4. "Strongest baseline" means the baseline with the highest Rel. in that setting.
| Backbone | Retained / original tokens | AnchorPrune Rel. | Strongest baseline | Baseline Rel. | Difference |
|---|---|---|---|---|---|
| LLaVA-1.5-7B | 32 / 576 | 93.9% | CDPruner | 92.6% | +1.3 percentage points |
| LLaVA-NeXT-7B | 160 / 2880 | 97.6% | CDPruner | 92.9% | +4.7 percentage points |
| Qwen2.5-VL-7B | 64 / 1296 | 80.8% | DivPrune | 80.0% | +0.8 percentage points |
| LLaVA-Video-7B | 512 / 2704 | 94.1% | DivPrune | 93.4% | +0.7 percentage points |
The two LLaVA image backbones use VQAv2, TextVQA, GQA, ScienceQA-IMG, MME, POPE, MMBench-EN/CN, and MM-Vet. The Qwen table uses MME, TextVQA, DocVQA, AI2D, MMMU, and MMBench-EN/CN. Video evaluation fixes the input to 16 frames and covers Video-MME with and without subtitles, EgoSchema 500-subset, and TempCompass.
Ablation Study¶
The following results are selected from Table 5. All variants share the same first-stage anchor and change only the second-stage selection rule. Here Rel. aggregates only MME, ChartQA, DocVQA, TextVQA, and MMBench-CN, so it must not be mixed with the main-results Rel.
| Budget | Second-stage rule | MME | ChartQA | DocVQA | TextVQA | MMBench-CN | Rel. |
|---|---|---|---|---|---|---|---|
| 64 | Diversity only | 1391.0 | 15.1 | 15.5 | 54.5 | 48.0 | 85.4% |
| 64 | Additive relevance-diversity | 1380.9 | 15.9 | 16.4 | 54.8 | 48.5 | 87.3% |
| 64 | Multiplicative relevance-diversity | 1379.9 | 15.9 | 16.6 | 54.5 | 48.2 | 87.2% |
| 64 | Importance-weighted contextual expansion | 1420.2 | 16.7 | 17.1 | 56.1 | 52.0 | 91.0% |
| 32 | Diversity only | 1317.9 | 14.6 | 12.8 | 52.6 | 44.2 | 79.4% |
| 32 | Additive relevance-diversity | 1344.5 | 14.2 | 14.2 | 52.7 | 45.2 | 81.0% |
| 32 | Multiplicative relevance-diversity | 1329.4 | 14.4 | 14.4 | 52.7 | 45.0 | 81.1% |
| 32 | Importance-weighted contextual expansion | 1394.7 | 15.1 | 14.5 | 54.2 | 49.9 | 85.1% |
Key Findings¶
- With the same anchor, importance-weighted contextual expansion beats the strongest alternative by +3.7 percentage points at 64 tokens and +4.0 percentage points at 32 tokens. This supports using global importance rather than simply reusing query relevance, but the table alone cannot quantify the benefit of adaptive anchoring itself.
- The best average does not imply winning every metric. At 64 tokens on Qwen, AnchorPrune scores 31.3 on DocVQA, below DivPrune's 31.6; both are far below the full model's 94.3. Document detail remains a weakness under severe compression.
- Efficiency analysis includes selection overhead. On LLaVA-1.5-7B at 32 tokens, AnchorPrune's Efficiency Score is 5.24, versus 5.28 for VisionZip, 5.41 for DivPrune, and 4.68 for CDPruner. The score multiplies prefill speedup by the ratio of full-model to pruned-model resident memory; 5.24 must not be described as a measured 5.24-fold latency speedup.
Highlights & Insights¶
- Selection order is an inductive bias. Preventing context selection from replacing the anchor isolates indispensable evidence from subsequent competition. This constraint expresses evidence asymmetry in visual question answering more directly than changing the weights of relevance and diversity.
- Novelty serves two different purposes. It determines the budget relative to a fixed initial anchor, then selects candidates relative to a changing retained set. Keeping the reference sets distinct explains how local coherence is protected while redundant context is avoided.
- Global importance can compensate for incomplete query matching. Information emphasized by the encoder may provide missing support when query priority is unreliable. The transferable lesson is to preserve signals with different roles rather than binding every selection stage to the same relevance estimator.
Limitations & Future Work¶
- Reproduction details are incomplete. Thresholds, minimum anchor size, patience, hardware, and detailed latency measurements are deferred to supplementary material, while the cache damages the novelty formulas. This note cannot provide an unambiguous implementation without checking the original formulas and supplementary settings.
- The protected set follows a heuristic ranking, not ground-truth evidence labels. Incorrect reversed CLIP similarity or post-projector matching may lock irrelevant tokens into the anchor, and the anchor can use at most half the budget. This is a mechanism-based concern from the note author, not a failure rate quantified in the main paper.
- The ablation has a limited scope. It validates second-stage rules but does not separately isolate protection, fixed versus adaptive budgeting, and similarity direction in the same table. Multiple-run variation and confidence intervals are not reported.
- Cross-architecture effectiveness is not lossless performance across tasks. Qwen's document question answering deteriorates substantially, and the video conclusion is based on a fixed 16-frame setting. Future work could protect structured dense-text evidence and evaluate longer videos and end-to-end deployment overhead; these are suggestions, not implemented components of this paper.
Related Work & Insights¶
- Versus VisionZip: Query-agnostic saliency can preserve globally important regions that do not answer the current question. AnchorPrune first protects query-conditioned evidence and then uses global importance to add context, at the cost of additional query-conditioned scoring.
- Versus DivPrune: Diversity promotes coverage but does not guarantee that the covered regions are useful. AnchorPrune weights novelty by importance outside the protected anchor, discouraging weakly informative selections made merely to increase feature dispersion.
- Versus CDPruner: A unified relevance-diversity objective makes the two criteria compete directly. AnchorPrune changes this relationship through ordered selection and a non-replaceable anchor. Main results support the overall strategy, but protection order and adaptive budgeting still merit finer orthogonal ablations.
Rating¶
These are the note author's subjective assessments, not conference review scores.
- Novelty: 4/5. The contribution centers on evidence-protection order and the contextual objective, rather than novel individual scoring signals.
- Experimental Thoroughness: 4/5. Four image or video backbones and multiple budgets are covered, but the main text insufficiently isolates the first stage.
- Writing Quality: 4/5. The motivation and controlled comparison are clear; complete reproduction depends on supplementary material, with separate formula-extraction issues in the cache.
- Value: 4/5. A reusable selection principle for training-free VLM acceleration, though severe-compression task losses and practical system gains require separate validation.