ToDRE: Effective Visual Token Pruning via Token Diversity and Task Relevance

Conference: ECCV 2026
Paper: ECCV Official Page
Area: Multimodal VLM / VLM Efficiency
Keywords: visual token pruning, token diversity, task relevance, information migration, training-free inference

TL;DR

ToDRE greedily selects diverse visual tokens before the large language model (LLM), then removes all remaining visual tokens at a deeper layer where cross-modal interaction has subsided, preserving 95.0% mean relative performance across eight image tasks on LLaVA-NeXT-7B with only 10% of input visual tokens retained and achieving about 2.6x speedup in the four-task ablation timing without training.

Background & Motivation

High-resolution images produce many similar visual tokens when split into a global thumbnail and local crops; video adds temporal redundancy. Once these tokens enter an LLM, they increase prefilling attention and feed-forward computation and affect subsequent decoding. For LLaVA-NeXT-7B with an input length of 3000 and an output length of 20, the paper estimates a vision-encoding:prefilling:decoding FLOPs ratio of 1:63.6:0.4. This motivates pruning early instead of concentrating only on the end of generation.

However, identifying duplicate tokens is different from identifying tokens that are no longer needed for a question. Attention ranking can retain many similar regions, while visual similarity alone does not establish whether textual representations have absorbed a detail. The authors observe that cross-modal attention concentrates in earlier layers and fades in deeper layers. This suggests a temporal source of redundancy: a visual token that was initially indispensable might no longer need to persist separately after its information has migrated into text representations.

The paper motivates separate treatment through an orthogonal-subspace model and decorrelation assumptions; this is a modeling argument, not an unconditional independence proof for arbitrary real vision-language models (VLMs). Core Idea: first preserve nonduplicate visual evidence through diversity, then decide when to withdraw the entire visual sequence based on whether cross-modal interaction has subsided, instead of asking one importance score to determine both content selection and token lifetime.

Method

Overall Architecture

The inputs are an image or video and a textual question; the original VLM still generates the output. The vision encoder and projector extract features as usual. ToDRE selects representative visual tokens in the projected LLM embedding space, then identifies a deeper point during LLM prefilling at which visual participation can end.

The pipeline consists of Pivot Initialization, Greedy Diversity Selection, and Relevance-Driven Withdrawal. The first two form Stage 1, while the last is Stage 2. The retention ratio describes the visual token count after Stage 1, not a constant proportion maintained throughout every model layer.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Image or video"] --> B["Vision encoder<br/>and projector"]
    B --> C["Pivot Initialization"]
    C --> D["Greedy Diversity Selection"]
    D --> E["Early LLM prefilling"]
    Q["System prompt and question"] --> E
    E --> F["Relevance-Driven Withdrawal"]
    F -->|Both ratios below threshold| G["Text-only subsequent layers<br/>and answer generation"]

The diagram shows inference data flow, with no new training supervision. Stage 2 withdraws visual tokens after interaction at the selected layer. This does not mean that earlier layers never receive visual information or that visual keys and values can be unconditionally deleted from every layer.

Key Designs

1. Pivot Initialization: connect a salient starting point with broader coverage

Greedy selection needs an initial retained token. The authors choose the visual token with the highest attention from the [CLS] token in the vision encoder's final layer. This determines only the starting point, rather than ranking the entire retained set by the same score. It uses visual salience without allowing similar high-attention regions to consume all subsequent slots.

For image models supporting AnyRes, the pivot comes only from the global thumbnail because it covers the whole image; local-crop tokens still participate in later diversity selection. Without AnyRes, selection operates on the original image tokens. For video, the highest-attention token from each frame becomes a candidate, and the strongest candidate supplies the pivot for cross-frame selection. Encoders without [CLS] can use a random pivot. The authors report nearly comparable performance, but the detailed comparison is in unavailable supplementary material, so this does not establish equivalence among all initialization strategies.

2. Greedy Diversity Selection: add the token least similar to the retained set in aggregate

After initialization, selection asks which candidate adds information not yet covered rather than which position is most salient. Cosine similarity is computed on projected visual embeddings. Each unselected token maintains its cumulative similarity to the entire retained set, and the next token is the one with the smallest sum. This is a sum-based criterion, not a nearest-neighbor-only comparison or an averaging operation that merges similar tokens.

The following notation restates the textual definitions around Eqs. (8)-(10). The cached equations have damaged formatting, so this is an explanatory transcription, not a verbatim reconstruction:

\[ \operatorname{sim}(u,v)=\frac{u^\top v}{\|u\|\|v\|},\qquad \operatorname{next}(C)=\arg\min_{u\in U\setminus C}\sum_{v\in C}\operatorname{sim}(u,v). \]

Here \(U\) contains all visual embeddings and \(C\) is the current retained set. After adding a token, the algorithm adds its similarity contribution to each remaining candidate's accumulator and masks the selected index. This continues until \(k\) tokens are retained. With distance defined as \(1-\operatorname{sim}\), the rule greedily maximizes the new token's summed distance to the set; it does not guarantee a globally optimal subset.

Unselected tokens are discarded before entering the LLM, while all text tokens remain. This selection stage does not explicitly use the current question and does not create mixed visual vectors. It prioritizes coverage of visual content, leaving the decision about when vision is no longer needed to Stage 2. The paper's example budget reduces at most 2880 visual tokens to at most 288, approximately 10%.

3. Relevance-Driven Withdrawal: let text absorb visual information before ending visual participation

Stage 2 no longer ranks individual visual tokens. Instead, it measures global cross-modal interaction at a few candidate layers in the latter half of the LLM. The authors argue that attention ratios are stable across neighboring deeper layers, making inspection at every layer unnecessary. When both text-to-visual and visual-to-text ratios fall below a threshold \(\tau\), all remaining visual tokens are removed after that layer, reducing subsequent prefilling and decoding overhead.

Each metric is the cross-modal attention mass divided by the total attention mass for the corresponding query class. The definitions below follow the surrounding explanation and legible summation domains of Eq. (12). System-prompt tokens contribute to the denominators but not to the cross-modal numerators:

\[ \alpha_{T\to V}^{(\ell)}= \frac{\sum_{i\in T}\sum_{j\in V}A_{ij}^{(\ell)}}{\sum_{i\in T}\sum_{j\in S\cup V\cup T}A_{ij}^{(\ell)}},\qquad \alpha_{V\to T}^{(\ell)}= \frac{\sum_{i\in V}\sum_{j\in T}A_{ij}^{(\ell)}}{\sum_{i\in V}\sum_{j\in S\cup V\cup T}A_{ij}^{(\ell)}}. \]

Here \(A\) denotes softmax-normalized attention averaged across heads, with queries as rows and keys as columns. \(S\), \(V\), and \(T\) denote system-prompt, visual, and textual token sets. Requiring both ratios to be small is intended to avoid withdrawal while information exchange continues. This is a proxy for completed information migration, not a direct verification that text representations contain every correct visual fact; token order, causal masking, and prompt format also affect the signal.

Task relevance enters through the question's participation in LLM interactions and the resulting withdrawal-layer decision, not through an additional trained token-level relevance classifier. The main text does not provide a reproducible numerical value for \(\tau\), the full candidate-layer list, or explicit behavior when no candidate triggers withdrawal. These details are not invented here.

A Worked Example

Consider the towel-color question in Fig. 3. Suppose the image produces 2880 visual tokens and the retention budget is 10%. The algorithm first selects the highest-[CLS]-attention pivot from the global thumbnail, then expands the set for diversity using candidates from both the thumbnail and local crops until 288 remain. This is not equivalent to retaining the top 288 attention-ranked tokens.

The 288 visual tokens and the question enter early LLM layers, allowing textual representations to absorb cues about towel color and material. At an inspected deeper layer, if both ratios fall below \(\tau\), subsequent layers stop processing visual tokens and continue answer generation through text states. Otherwise, withdrawal does not occur at that checkpoint. These token counts illustrate the specified budget, not a measured trigger layer or attention trace for a particular image.

Loss & Training

ToDRE introduces no new loss and requires neither encoder/LLM fine-tuning nor distillation. Its inference settings include the visual retention budget, candidate inspection layers, and attention threshold. Existing features and attention determine both retained content and withdrawal timing.

The authors claim compatibility with efficient attention operators, but the available main text does not fully specify low-overhead extraction of the required ratios. A reproduction must include metric computation, greedy selection, and cache handling in latency measurements rather than estimating end-to-end speedup from token counts alone.

Key Experimental Results

Main Results

Experiments cover LLaVA-NeXT-7B/13B, Qwen2.5-VL-7B-Instruct, and InternVL2-8B across eight image and four video benchmarks. The following selection from Table 1 uses LLaVA-NeXT-7B throughout. The eight-task mean averages each task's score relative to the original model; it is neither a mean of raw scores nor an accuracy measure.

Method	Stage 1 retention	MME	GQA	POPE	VQAv2	Eight-task mean relative performance
Original	100%	1519.6	64.2	87.7	80.1	100.0%
DivPrune	25%	1486.5	61.8	87.4	76.4	96.6%
ToDRE	25%	1504.3	63.3	87.5	77.5	98.2%
FasterVLM	10%	1359.2	56.9	83.6	72.3	91.4%
DivPrune	10%	1396.2	59.2	84.3	73.2	93.5%
ToDRE	10%	1464.4	59.4	85.0	74.4	95.0%

FLOPs alone do not establish efficiency. Table 4 reports the following measurements on one NVIDIA RTX 3090. Throughput concerns POPE only, whereas performance covers the eight image tasks:

Method	FLOPs (T)	Peak memory (GB)	POPE throughput (samples/s)	Eight-task mean relative performance
Original	31.4	15.9	1.5	100.0%
DivPrune, 10%	6.0	13.6	2.8	93.5%
ToDRE, 10%	6.0	13.6	2.9	95.0%

ToDRE reduces FLOPs by 80.9% and memory by 14.5%, with approximately 1.9x the original POPE throughput. This is distinct from the abstract's approximately 2.6x total-time speedup, which corresponds to the different timing scope in the ablation below.

Ablation Study

Table 5 reports only MME, ScienceQA, GQA, and POPE, so its mean cannot be directly compared with Table 1's eight-task mean. Total time is in minutes:seconds.

Config	Stage 1 retention	Total time	MME	ScienceQA	GQA	POPE	Four-task mean relative performance
Original	100%	77:04	1519.6	72.0	64.2	87.7	100.0%
Stage 2 only	100%	70:15	1522.7	71.9	64.3	87.6	100.0%
Stage 1 only	25%	48:10	1503.8	70.6	63.1	87.5	98.8%
ToDRE	25%	44:18	1504.3	70.7	63.3	87.5	98.9%
Stage 1 only	10%	31:18	1458.6	70.4	59.4	85.0	95.8%
ToDRE	10%	29:43	1469.3	70.5	59.4	85.0	96.0%

The source contains unexplained differences: at 10% retention, ToDRE's MME/ScienceQA scores are 1464.4/70.2 in Table 1 but 1469.3/70.5 in Table 5. Both are preserved as reported rather than silently reconciled. The difference between four-task and eight-task averages should not be attributed entirely to this inconsistency.

Key Findings

• Stage 1 supplies most of the speedup: the 10% budget reduces time from 77:04 to 31:18. Adding Stage 2 brings it to 29:43, saving a further 1 minute 35 seconds relative to Stage 1 alone.
• The change from 77:04 to 29:43 corresponds to about 2.59x, consistent with the paper's rounded 2.6x claim. Stage 2 alone retains 100.0% mean relative performance, not exactly unchanged scores on every task.
• Table 2 reports video mean relative performance of 103.1%/100.9% at 25%/10% retention. However, at 10%, MLVU drops from 20.1 to 18.2; an above-baseline mean does not imply universal task improvements.
• In Table 3, Qwen2.5-VL and InternVL2 preserve 92.0% and 91.5% mean relative performance across twelve tasks at 10% retention. LLaVA's 95.0% should therefore not be treated as a model-independent result.

Highlights & Insights

• Separating what to retain from how long to retain it creates two distinct intervention points. The first protects input coverage; the second exploits the progression of visual information into textual states.
• Attention serves pivot initialization and deep-layer withdrawal detection, while set diversity drives the main selection process. This can preserve distinctive but less salient regions better than ranking every visual position by one attention score.
• Directly retaining original tokens avoids altering embeddings through artificial averaging. The trade-off is that deleted regions cannot be recovered from a newly aggregated token, so the retention budget remains consequential.

Limitations & Future Work

• The authors note that most benchmarks require short answers, limiting decoding-stage gains. Long answers might benefit more, but the available main text lacks sufficient long-generation latency curves to treat this expectation as an established result.
• The theory relies on orthogonal subspaces and decorrelation assumptions. Approximate orthogonality in real embeddings, statistical relationships under nonlinear kernels, and practical error bounds need stronger validation; this argument cannot guarantee lossless task performance.
• A small global attention ratio does not prove that every detail has been encoded correctly. Fine-grained OCR, repeated visual checks, or evidence revisiting during long reasoning might require a small revisitable visual reserve, offering a testable extension.
• The supplied cache does not include supplementary material, preventing further verification of random-pivot results, layer-selection ablations, thresholds, and implementation details. Unexplained differences between Tables 1 and 5 also complicate precise reproduction.

Related Work & Insights

• vs FasterVLM: FasterVLM uses [CLS] attention to assess visual-token importance. ToDRE uses it only for initialization, expands coverage through cumulative similarity, and additionally decides when deeper layers can withdraw visual tokens.
• vs DivPrune: Both emphasize diversity rather than attention alone. ToDRE uses greedy max-sum selection and adds a cross-modal-interaction-driven second stage; at the same 10% budget, eight-task relative performance is 95.0% versus 93.5%, but FLOPs and memory do not improve further.
• vs FastV / SparseVLM: These approaches use attention to guide pruning inside the LLM, while ToDRE first reduces visual sequence length before LLM input. The paper excludes FastV and SparseVLM from the 13B main comparison because transferred 7B settings degrade, so the reported results do not establish universal superiority across every scale and tuning budget.
• Transferable direction: Combine the information-absorption trigger with a minimum visual-token reserve to test whether long answers and fine-grained tasks benefit from revisiting visual evidence. This is a research direction suggested by this note, not an existing component of the paper.

Rating

• Novelty: 4/5. The two-stage division is clear, although greedy diversification and attention-guided pruning build on established ideas.
• Experimental Thoroughness: 4/5. Four model configurations and twelve benchmarks provide efficiency comparisons and stage ablations; long generation and trigger parameters need more evidence.
• Writing Quality: 3/5. The mechanism is understandable, but theoretical assumptions require careful interpretation and unexplained numerical differences remain across tables.
• Value: 4/5. Training-free operation and measurable gains offer a composable engineering approach to heavily compressed VLM inference.