Combating Textual Noise and Redundancy: Entropy-Aware Dense Visual Token Pruning

Conference: ECCV2026
Paper: Official paper page · PDF
Code: https://github.com/SJTU-DeepVisionLab/EADP
Area: VLM Efficiency
Keywords: Visual token pruning, textual noise, information entropy, spatial priors, submodular optimization

TL;DR

EADP filters dispersed text guidance using the spatial entropy of text-to-image responses, combines local and global semantics, smooths the relevance map, and selects visual tokens by coverage gain, allowing LLaVA-NeXT-7B to match the full model's 66.3 average while retaining only 640/2880 tokens.

Background & Motivation

Vision-language models typically turn an image into many patches and feed the resulting visual tokens into a language model alongside the question. High-resolution inputs are particularly expensive, but deleting visual tokens can remove small text, object parts, or contextual evidence needed to answer negative queries. Methods such as CDPruner use the global CLIP EOS text vector to measure image-text relevance. This compressed representation captures the overall topic, but it may not localize the fine-grained entities mentioned in a question.

Comparing every text token with every visual token seems like a natural way to improve precision. Yet the paper observes that directly aggregating these responses does not deliver the expected gains. Entity words often produce peaks over a few patches, whereas function words and punctuation can produce nearly uniform low responses across the image. Because there are many such tokens, their accumulated responses raise the background noise floor. Even manually selecting relevant entity words does not solve everything: independent Top-K selection can repeatedly choose one salient part and omit the rest of the object. Scoring and subset selection are distinct problems.

The paper therefore changes the question from which patches score highest to which patches can represent the image under the question's constraints. Core idea: clean fine-grained guidance using the entropy of spatial text responses, then select a complementary visual subset with a spatially informed, weighted facility location objective.

Method

Overall Architecture

The input is an image and its question; the output is a visual subset passed to the existing LLM. The image encoder first produces N visual tokens. The question follows two text paths: the original LLM tokenizer preserves the language sequence required for generation, while an additional CLIP text encoder supplies individual token features and an EOS feature for pruning. Filtering concerns the CLIP guidance branch, not the words that the LLM actually receives.

Visual features are mapped into the CLIP embedding space through a lightweight projection implemented as in CDPruner. Entropy-guided denoising extracts local dense guidance, and global semantic fusion restores sentence-level context. Spatial smoothing and score polarization refine the relevance map. Facility location selection then combines these weights with visual feature similarities to select K tokens incrementally. The result is a subset of existing visual representations, not merged neighboring patches or a newly trained answering model.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Image and question<br/>Visual and text encoding"] --> B["Entropy-guided denoising"]
    B --> C["Global semantic fusion"]
    A -->|Global EOS responses| C
    C --> D["Spatial smoothing and<br/>score polarization"]
    D --> E["Facility location selection"]
    A -->|Visual feature similarities| E
    E --> F["K visual tokens<br/>Original question and LLM generation"]

Key Designs

1. Entropy-guided denoising: filter text by response concentration, not part of speech

For each non-EOS text feature, EADP computes cosine similarities against all visual tokens, yielding an M×N matrix. A Softmax over visual positions converts each row into a spatial probability distribution, whose information entropy measures dispersion. The following expression reconstructs the meaning of Eqs. (4)–(5), retaining the paper's multiplicative scale τ: larger τ makes the response sharper, unlike the common convention of dividing logits by a temperature.

\[ p_{i,j}=\frac{\exp(\tau c_{i,j})}{\sum_{k=1}^{N}\exp(\tau c_{i,k})},\qquad h_i=-\sum_{j=1}^{N}p_{i,j}\log p_{i,j}. \]

A text token attending uniformly to all patches has high entropy; one concentrating on a few local regions has low entropy. The algorithm uses the lower-tail q-quantile of the entropy distribution as its threshold and retains tokens at or below it. Selection therefore depends on the current image's responses and requires no external NLP parser. Ties at the threshold mean the count need not equal qM exactly. Nor does passing the filter guarantee correct semantics: a concentrated but incorrect match can survive.

The surviving text tokens are not weighted equally. A Softmax over negative entropy scaled by γ assigns greater weight to more concentrated responses. The paper notes CLIP's 77-token text limit and computes dense similarities through a single matrix multiplication, but the available main text does not explain truncation, segmentation, or coverage for longer instructions. Here, dense means token-level text-to-vision guidance, not unlimited instruction length.

2. Global semantic fusion: combine local entity grounding with sentence-level context

Dense guidance emphasizes local entity evidence and may overlook sentence-level constraints. The EOS vector offers coarser localization but retains global semantics. EADP connects the two with a convex combination instead of letting low-entropy entity words control pruning alone. With T′ denoting retained text tokens, G the EOS cosine response, and D the dense guidance, the central computation is:

\[ \alpha_i=\frac{\exp(-\gamma h_i)}{\sum_{r\in T'}\exp(-\gamma h_r)},\qquad s_j^D=\sum_{i\in T'}\alpha_i c_{i,j},\qquad s_j^I=\mu s_j^G+(1-\mu)s_j^D. \]

The fused scores are min-max normalized into [0,1], with ε for numerical stability, before spatial processing. This makes the direction of μ explicit: μ=0 gives dense-only guidance, while μ=1 gives global-only guidance. In the ablation, μ=0.5 reaches a five-task average of 68.5, above the endpoint scores of 67.7 and 68.1. This supports complementarity, but it does not establish that every architecture should use the same mixture.

3. Spatial smoothing and score polarization: inject local structure into weights without requiring contiguous selection

Independent token scores do not explicitly model adjacency, so a peak on one object part does not automatically support its neighbors. The method reshapes normalized scores into an H×W grid, propagates local responses with a 3×3 Gaussian convolution, and flattens the result. Smoothing acts on scores, not visual embeddings. It helps neighboring regions compete for representation, but the final output remains a discrete subset with no requirement to retain a contiguous rectangle.

Smoothing also reduces peaks and raises nearby background scores. A power transform subsequently adjusts their relative importance, implementing the polarization in Eq. (12). This is exponentiation by a power, not the application of an exp function:

\[ \hat{s}_j^I=\left(s_{\mathrm{smooth},j}^I\right)^\beta. \]

For values in [0,1] and β>1, non-endpoint absolute values decrease, but larger scores gain importance relative to smaller ones. That is the precise sense in which salient regions are strengthened. Two controls must also be distinguished: β=1 is the identity transform, whereas β=0 gives uniform weights for positive scores rather than merely disabling sharpening. The paper calls β=0 disabled polarization; this wording is broader than what the equation implies, and the main text does not specify how zero scores are handled.

4. Facility location selection: assess how much uncovered content each additional token represents

Top-K assigns candidates independent scores, so two almost identical high-scoring patches can both be selected. Facility location instead lets every original token find its most similar representative in the selected subset, weighted by instruction relevance. Reconstructing Eq. (13) from its surrounding definitions, the budget-constrained objective is:

\[ \max_{Y\subseteq V,\ |Y|=K}F(Y),\qquad F(Y)=\sum_{j=1}^{N}\hat{s}_j^I\max_{v_i\in Y}\operatorname{Sim}(v_i,v_j). \]

Crucially, the weight belongs to the original token that needs representation, not only to the candidate under consideration. A candidate almost identical to an already selected token improves few best-match similarities. Another candidate representing an uncovered object part may produce a larger gain. Redundancy is thus addressed inside the subset objective rather than through post-hoc deduplication after ranking.

The authors use a greedy approximation: cache each original token's best current coverage similarity as Curr, add the candidate with the highest marginal gain, and update the cache until the budget is filled. The increment is:

\[ \Delta(u\mid Y)=\sum_{j=1}^{N}\hat{s}_j^I\left[\max\{\operatorname{Sim}(u,v_j),\operatorname{Curr}(v_j)\}-\operatorname{Curr}(v_j)\right]. \]

The diminishing-return structure supports submodular optimization. The authors state a 1−1/e approximation guarantee, which should be understood as an objective-value bound under the relevant normalization, nonnegativity, and monotonicity conditions, not a proof that every semantic part survives. The main text uses cosine similarities but delegates empty-set initialization, negative-similarity handling, and proofs to supplementary material. These details cannot be verified from the available main-text cache. Precomputing the N×N similarity matrix also requires O(N²) storage, so pruning itself has a cost for high-resolution images and video.

A Worked Example

Consider the Fig. 2 instruction asking whether an image contains an oven. The CLIP guidance branch examines each word's response over the visual grid. If oven concentrates on the corresponding object parts, its low-entropy response contributes more than dispersed formatting instructions. Meanwhile, the LLM question remains intact, so the request to answer with a word or phrase is not removed from the generation input.

Under one actual LLaVA-1.5 experimental budget, 576 visual tokens are reduced to 128. Fused high-response regions first propagate scores to neighboring structure; the power transform then adjusts their weights, and greedy selection compares incremental coverage to avoid retaining only a cluster of similar patches. This is a procedural illustration: the cache does not provide this image's selected indices at every iteration, complete entropy values, or a quantitative validation of its final answer. It is not an additional experiment.

Loss & Training

The paper presents EADP as a plug-and-play inference pruning module and introduces no new training loss, additional fine-tuning data, or reproducible training schedule. Experiments use existing LLaVA implementations, lmms-eval for video, and VLMEvalKit for the Qwen family, on NVIDIA RTX 3090 GPUs.

Reproduction requires distinguishing ablation settings from a universal default configuration. Table 7 explicitly uses LLaVA-1.5-7B with 128 visual tokens and varies μ, q, β, smoothing kernels, and aggregation rules. The available main text does not fully specify τ, γ, default β across architectures, or cross-architecture projection details. Mathematical extraction is damaged in the cache; the equations above are reconstructed from nearby definitions. Unspecified parameters and implementation conditions are not invented; exact implementation remains subject to the original paper and code.

Key Experimental Results

Main Results

The following table selects reported Avg. values from Tables 1–5. Comparisons are matched by architecture and budget within each row. LLaVA image averages cover nine tasks and exclude VizWiz; Qwen2.5, Qwen3, and video averages cover eight, ten, and three tasks, respectively. These are the paper's aggregate scores, not one comparable accuracy metric across architectures. The main text also does not fully explain aggregation across different raw metric scales.

Model	Retained / original tokens	Full-model Avg.	Baseline Avg.	EADP Avg.	Difference from baseline
LLaVA-1.5-7B	128 / 576	64.9	CDPruner 63.2	63.5	+0.3
LLaVA-1.5-7B	32 / 576	64.9	CDPruner 60.4	60.9	+0.5
LLaVA-NeXT-7B	640 / 2880	66.3	CDPruner 66.0	66.3	+0.3
LLaVA-NeXT-7B	320 / 2880	66.3	CDPruner 64.5	65.2	+0.7
Qwen2.5-VL-7B	512 / 1296	84.0	DivPrune 78.1	78.0	−0.1
Qwen2.5-VL-7B	128 / 1296	84.0	DivPrune 66.4	68.4	+2.0
Qwen3-VL-8B	128 / 1024	84.3	DivPrune / CDPruner 59.2	62.7	+3.5
LLaVA-Video-7B	64×32 / 64×169	61.2	DivPrune 56.2	57.2	+1.0

At 256 tokens, Qwen3-VL achieves 62.8 on DocVQA versus 55.8 / 53.0 for DivPrune / CDPruner, gains of 7.0 / 9.8 points, suggesting a benefit for fine-grained document evidence. The video discussion states 57.0 at 81.1% pruning, a drop of 4.2, but Table 5 reports 57.2, which is 4.0 below 61.2. This note uses the table value and flags the conflict. VizWiz also mixes official-test and reproduced-validation reporting elsewhere in the source; those results are not folded into the image averages above.

Ablation Study

These values come from Table 7: LLaVA-1.5-7B retains 128 visual tokens, and Avg. covers VizWiz, SQA, TextVQA, POPE, and MME. Rows belong to separate parameter sweeps, not a sequential experiment that adds components cumulatively.

Sweep	Configuration	Avg.	Supported interpretation
Global-dense fusion	μ=0.0	67.7	Dense guidance only
Global-dense fusion	μ=0.5	68.5	Combining both streams works better
Global-dense fusion	μ=1.0	68.1	Global guidance only
Text retention	q=0.3	68.6	Removing more high-entropy text remains effective
Text retention	q=0.5	68.5	Intermediate retention
Text retention	q=0.9	67.6	Retaining more text preserves more noise
Spatial smoothing	None	67.9	No neighborhood score propagation
Spatial smoothing	3×3	68.5	Gain of 0.6 over no smoothing
Score polarization	β=0.0	67.8	Uniform weights for positive scores
Score polarization	β=5.0	68.3	Power setting in this sweep

The 1.0-point gain of q=0.3 over q=0.9 is relatively clear evidence for denoising in these sweeps. However, the table does not provide an isolated replacement of facility location with Top-K, so its contribution cannot be claimed to have been independently quantified. Entropy, central mass ratio, and variance score 68.5, 68.4, and 68.5, respectively, suggesting that the benefit is not exclusive to entropy as the dispersion statistic.

Key Findings

Efficiency measurements come from Table 6: 2,000 instances sampled equally from VizWiz, TextVQA, POPE, and MME, evaluated with LLaVA-1.5-7B. The table below preserves raw times and FLOPs rather than treating a FLOPs ratio as an end-to-end speedup.

Configuration	Visual tokens	Prefill / ms	End-to-end latency / ms	FLOPs / G
Full model	576	207.4	256.8	4489.8
EADP	128	118.8	169.3	1538.4
EADP	32	81.9	129.63	904.1
DivPrune	128	109.6	158.0	1529.9

At 128 tokens, EADP reduces FLOPs by approximately 65.7%, but end-to-end speedup is only about 1.52×. It is also 11.3 ms slower than DivPrune at the same budget. The conclusion is an improved quality-efficiency trade-off, not simultaneous leadership in quality and latency. More aggressive pruning is not lossless either: Qwen3-VL's 62.7 remains 21.6 points below the full model's 84.3.

Highlights & Insights

• Textual noise can be defined through cross-modal responses rather than a word list. This could transfer to text-conditioned region retrieval, provided spatial concentration correlates with useful evidence.
• Relevance and representativeness are different optimization dimensions. Building reliable query weights before evaluating subset marginal gains explains why better scoring alone does not make Top-K sufficient.
• The spatial prior modifies weights while preserving original visual features. This makes the method easier to insert into existing inference pipelines and leaves selected representations open to inspection.

Limitations & Future Work

• Compute and reproducibility boundaries: The authors explicitly acknowledge O(N²) storage for precomputed similarities and delegate further implementation details and timings to supplementary material. The available cache does not establish peak memory on long videos, component-wise latency, or every default parameter.
• Concentration is not correctness: This is an analytical limitation identified in this note. Low-entropy responses can be incorrectly localized, while valid evidence for negation, relations, or multiple objects can be spatially dispersed. Global guidance can compensate only partially when filtering is wrong.
• Theory is not a semantic guarantee: Facility location optimizes weighted feature coverage; it does not ensure retention of every part, rare object, or negative-query cue. Object-level coverage metrics and difficult negative-query breakdowns would be needed for stronger claims.
• Experimental attribution remains limited: The available results lack a readable Top-K replacement ablation, repeated-run variance, and statistical significance. EADP also loses to a baseline under the larger Qwen2.5-VL budget, so universal superiority is not supported.
• Text and parameter boundaries: Handling instructions beyond CLIP's limit is not explained in the main text. Adapting q, μ, and the visual budget to instruction length or response uncertainty is a plausible research direction, not a capability already validated by this paper.

Related Work & Insights

• Compared with CDPruner: EADP retains EOS global guidance and the projection implementation, but adds low-entropy dense guidance and replaces a DPP objective primarily modeling diversity with facility location coverage. Neither method should be indiscriminately described as Top-K.
• Compared with DivPrune: EADP augments diversity-oriented visual compression with query-dependent weights. It scores higher under most strict budgets listed here, but DivPrune can be faster and has a slightly higher Qwen2.5-VL Avg. at 512 tokens.
• Compared with FastV and VisionZip: The paper includes importance-based and structure-aware pruning baselines. The transferable lesson is not simply to add another score to every method, but to test separately whether the guidance is contaminated and whether the subset objective covers complementary evidence.

Rating

• Novelty: 4/5. Combining spatial text-entropy denoising with query-weighted facility location addresses a clear failure mode, although the statistical and optimization tools are established.
• Experimental Thoroughness: 4/5. Multiple architectures, images, video, and measured latency are covered; isolated selector ablations and uncertainty reporting remain limited.
• Writing Quality: 3/5. The methodological narrative is clear, but video numbers, polarization terminology, and default-configuration reporting contain inconsistencies or gaps.
• Value: 4/5. Useful for studying visual evidence retention under strict budgets, with deployment requiring reevaluation for the target architecture and latency constraints.