DIVA: Instruction-Aware Vision Token Pruning via Dual-Probe Attention Discrepancy¶

Conference: ECCV 2026
Paper: ECCV 2026
Area: VLM Efficiency
Keywords: Vision Token Pruning, Instruction-Aware, Dual-Probe Attention Discrepancy, Training-Free Compression, KV Cache

TL;DR¶

Instead of directly treating absolute attention values as vision token importance, DIVA compares the attention distribution discrepancy of two sets of text probes ("can see instructions" vs. "cannot see instructions") on vision tokens, and then calibrates tokens that bypass self-attention. On LLaVA-NeXT-7B, DIVA retains 92.6% of baseline performance while keeping only 80/2880 vision tokens and reducing the KV cache size by 10.8\(\times\).

Background & Motivation¶

Large Vision-Language Models (LVLMs) typically flatten images into hundreds or even thousands of vision tokens before feeding them into the language model alongside text. This long visual sequence leads to quadratic growth in self-attention computation with sequence length, and continuously occupies KV cache during autoregressive generation. Training-free vision token pruning is therefore highly attractive: it requires no model retraining and only preserves a small number of "important" tokens during inference. However, existing methods like FastV, SparseVLM, and PDrop mostly rank importance directly based on absolute text-to-vision attention scores. This value does not only reflect the visual evidence required by the query; it is also heavily confounded by positional biases and visual attention sinks. Consequently, a token that is irrelevant to the question but naturally attracts high attention might be retained indefinitely, whereas local regions crucial for answering the question may be discarded.

Layer-wise pruning can reassess token importance at each language model block, allowing unselected tokens to re-enter competition in subsequent layers and thus preventing irreversible information loss from one-off deletion. However, this introduces a second issue: selected tokens undergo self-attention updates, whereas bypassed tokens do not. The representation discrepancy between these two calculation paths accumulates with depth, making subsequent layer-wise importance ranking increasingly unstable.

This paper decouples "instruction relevance" from "high attention values": if a certain attention pattern appears in both with-instruction and without-instruction probes, it is more likely a common bias; only the discrepancy between the two is closer to the true visual requirement driven by the instruction. Core Idea: Define the Context Shift (CS) score as the token-wise discrepancy of dual-probe attention distributions. Only compute self-attention for high-scoring vision tokens layer-by-layer, while utilizing a lightweight VF-Calibrator to simulate the missing self-attention aggregation for bypassed tokens, enabling their reliable reuse in subsequent layers.

Method¶

Overall Architecture¶

The input sequence consists of system tokens, vision tokens, and text tokens. The text tokens are further divided into pre-instruction delimiters, the instruction itself, and suffixes such as ASSISTANT:. DIVA retains full computation for the first two language model blocks to facilitate early cross-modal alignment. Starting from the third block, it reconstructs dual probes and computes CS scores at each layer, dividing the vision tokens into important and non-important sets. Important tokens are processed through normal self-attention, while non-important tokens bypass this step and are passed to the VF-Calibrator. Finally, both sets are merged back in their original order, passing through the residual path and FFN before entering the next block for reassessment.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
        A["Image + Instruction<br/>Fully pass through the first two blocks"] --> B["Dual-Probe CS Scoring<br/>Layer-wise reassessment of instruction relevance"]
        B -->|Top-K| C["Important Tokens<br/>Perform self-attention"]
        B -->|Other| D["VF-Calibrator<br/>Calibrate bypassed tokens"]
        C --> E["Merge in original order<br/>Residual + FFN"]
        D --> E
        E -->|Next language model block| B
        E --> F["Generate answer"]

Key Designs¶

1. Dual-Probe CS Scoring: Identifying Visual Evidence through "Context-Induced Shifts" Rather than Absolute Attention Values

The non-contextual probe is chosen as the delimiter token before the instruction. Since causal masking prevents it from attending to subsequent instructions, its attention over vision tokens serves as a baseline pattern free of question semantics. The contextual probe contains the post-instruction suffix tokens, which have already aggregated the instruction's semantics. The paper also filters the instruction tokens themselves, selecting those whose representations are closer to the suffix's average representation than the prefix delimiter's average representation, thus avoiding signal dilution from abundant weak-context tokens.

For each attention head and each contextual probe, DIVA computes the token-wise forward KLD contribution of its vision attention distribution relative to the non-contextual probe's distribution. If the attention probability of the \(i\)-th vision token under the two probes are \(A^h_{t,i}\) and \(A^h_{0,i}\) respectively, the discrepancy is defined as:

\[ D^h_{t,i}=A^h_{t,i}\log\frac{A^h_{t,i}}{A^h_{0,i}}. \]

The maximum value across all contextual probes is taken within the same head to retain the strongest instruction-induced shift; then, scores are averaged across all attention heads to obtain \(CS_i=\frac{1}{H}\sum_h\max_{t\in\mathcal{T}_C}D^h_{t,i}\). This "canceling out shared patterns, highlighting contextual shifts" ranking strategy is less sensitive to positional biases and attention sinks. Ablation studies also show that forward KLD yields a 98.8% relative performance, outperforming JSD (98.1%) and reverse KLD (95.9%), confirming that the direction of discrepancy matters.

2. Layer-wise Top-K Bypassing: Saving Computation per Layer Without Permanently Discarding Visual Information

Each layer selects the Top-K important vision tokens according to their CS scores, and only these tokens participate in self-attention and are written to the corresponding KV cache. The remaining tokens are retained in the sequence state rather than permanently deleted. During merging, they are restored to their original positions, allowing the subsequent layers to re-evaluate them based on the updated context. Regions deemed unimportant in the current layer can still be re-selected for the Top-K if they become relevant to the reasoning step in deeper layers.

This design reformulates "pruning" as layer-wise sparse computation instead of one-off input compression. This avoids irreversible information loss from early misjudgments at extreme budgets, at the cost of passing all bypassed tokens through subsequent FFNs and computing probe-related scores in each layer. Emphasizing this, DIVA is enabled starting from the third block to preserve early-stage full vision-language interactions.

3. VF-Calibrator: Patching the Representation Gap for Bypassed Tokens with Shared Global Anchors

Bypassed tokens do not undergo weighted value aggregation through self-attention, and directly passing them to the next layer causes them to drift into different representation trajectories compared to normally updated tokens. Without introducing any training, VF-Calibrator reuses the existing visual values and output projection in the current attention layer. It mixes the individual value features of each bypassed token with the global reference value of <BOS>, followed by a residual update through the output projection. For a bypassed token \(v_j\), the calibrated feature is:

\[ \hat{\mathbf v}_{j}=\gamma\mathbf v^{\mathrm{self}}_{j}+(1-\gamma)\mathbf v^{\mathrm{ref}},\qquad \mathbf h_j\leftarrow\mathbf h_j+\mathrm{Proj}_O(\hat{\mathbf v}_j). \]

Since <BOS> often acts as a high-attention anchor aggregating global context, this fixed mixing approximates the missing contextual aggregation for bypassed tokens with negligible overhead. On LLaVA-1.5-7B, \(\gamma=0.1\) performs best; increasing it to 0.5 drops relative performance from 98.8% to 92.7%, suggesting the global reference is indispensable. The cosine distance in representation at the final layer is also reduced from 0.77 (uncalibrated) to 0.28, demonstrating that the module successfully narrows the trajectory gap.

A Complete Example¶

Consider LLaVA-NeXT encoding an image into 2880 vision tokens for the question "Is the boat moving or stationary?". The first two blocks fully process all tokens. In a subsequent block, the post-instruction suffix ASSISTANT: and selected instruction tokens like "boat" and "moving" form the contextual probe, whereas the pre-instruction newline delimiter forms the non-contextual probe. If attention near the boat and the water surface changes significantly under the contextual probe, while certain sequence-end tokens maintain high attention on both probes, the CS score will scale up the former and suppress the latter.

Under an 80-token budget, only the top 80 tokens perform self-attention in this layer, while the remaining 2800 tokens pass through the VF-Calibrator and merge in their original order. The next layer re-evaluates the CS scores so that new local evidence can still be selected. This pipeline reduces the KV cache from 1496.0 MB to 138.9 MB, while raising the POPE F1 score from 86.5 (unpruned baseline) to 88.3.

Loss & Training¶

DIVA is a completely training-free inference-time method that introduces no additional loss functions or parameter updates to the original LVLM. The primary hyperparameters are the number of preserved vision tokens \(K\) per layer, and the VF-Calibrator mixing coefficient \(\gamma\). Experiments on LLaVA-1.5-7B use \(\gamma=0.1\); LLaVA-NeXT-7B and LLaVA-Video-7B use 0.2; Qwen3-VL-8B and InternVL3.5-8B use 0.3. All efficiency measurements are conducted on a single NVIDIA RTX A6000 GPU.

Key Experimental Results¶

Main Results¶

Model / Scenario	Token Budget	Metric	DIVA	Baseline / Comparison	Relative Baseline Performance
LLaVA-1.5-7B, General Visual Reasoning	128 / 576	GQA	61.4	Vanilla 62.0	100.9%
LLaVA-1.5-7B, General Visual Reasoning	64 / 576	MME	1833	Vanilla 1862	99.8%
LLaVA-1.5-7B, Extreme Compression	8 / 576	Avg. Rel. Perf.	89.0%	FastV@64 75.4%	89.0%
LLaVA-NeXT-7B, High Resolution	80 / 2880	POPE F1	88.3	Vanilla 86.5	92.6%
Qwen3-VL-8B, Fine-grained Reasoning	Retain 10%	DocVQA	81.3	PDrop 56.4	82.6%
LLaVA-Video-7B, Video Understanding	16 Tokens/Frame	Video-MME Short	73.1	DivPrune@32 67.7	94.5%

Ablation Study¶

Configuration	Relative Performance	Extra Evidence / Efficiency	Description
Neither: FastV-style Score + Direct Recovery of Bypassed Tokens	87.9%	-	Both types of issues coexist
VF-Calibrator Only	95.6%	-	Calibration significantly improves reuse
DIVA Pruner Only	97.8%	-	CS sorting is the main source of contribution
Full DIVA	98.8%	Cosine distance of final layer 0.28	The two modules complement each other
Full DIVA, 80 Tokens (LLaVA-NeXT)	POPE 88.3	15.3 GFLOPs / 0.58 ms / 138.9 MB	Reduced by 34.6\(\times\) / 7.2\(\times\) / 10.8\(\times\) compared to unpruned
Full DIVA, 40 Tokens (LLaVA-NeXT)	POPE 86.7	9.7 GFLOPs / 0.51 ms / 120.4 MB	Reduced by 54.6\(\times\) / 8.2\(\times\) / 12.4\(\times\) compared to unpruned

Key Findings¶

CS ranking contributes the most, but not entirely: utilizing only the DIVA-Pruner yields 97.8%, which rises to 98.8% with the VF-Calibrator. Using only calibration yields 95.6%, showing that the representation gap in layer-wise token reuse is a genuine bottleneck.
The advantage is more pronounced under extreme budgets. When retaining only 32 tokens, DIVA's relative performance on LLaVA-1.5-7B is 96.1%, outperforming CDPruner (95.2%) and VisionDrop (91.6%). With only 8 tokens, it still maintains 89.0% relative performance.
The method is generalized beyond a single model or static images: it covers LLaVA, Qwen3-VL, InternVL3.5, and LLaVA-Video, demonstrating robust performance on document QA and video tasks. However, different models utilize varying \(\gamma\) values, meaning across-model numbers should not be directly cross-compared under a unified hyperparameter config.

Highlights & Insights¶

The most elegant aspect of DIVA is treating the attention bias as a shared "background noise" between dual probes. Instead of manually identifying and correcting sources like positional bias or sinks, it directly extracts instruction-induced signals through context discrepancies. This mechanism can be extended to dynamic filtering of text tokens, video frames, or retrieval candidates.
DIVA does not equate bypassing with permanent deletion. Layer-wise reassessment allows pruned tokens to re-enter calculations while being calibrated with low overhead, making the compression process akin to reversible computational scheduling. This is much better suited to multi-step reasoning where attention pivots change dynamically than one-off vision token dropping.
VF-Calibrator approximates global aggregation by mixing pre-trained projections with <BOS>, without requiring any training of new adapters. This showcases a practical trick: representation drifts in sparse paths can be aligned at low cost using existing model-internal aggregation anchors.

Limitations & Future Work¶

The method still needs to store the states of all bypassed tokens, re-evaluate them layer-by-layer, and pass them through FFNs. The evaluated gains are primarily concentrated in self-attention computing and KV cache rather than the overall calculation of the entire model. Thus, a 54.6\(\times\) self-attention GFLOPs reduction does not equate to a 54.6\(\times\) end-to-end speedup.
The probe construction relies on the prompt template structure under causal masking, specifically where "pre-instruction delimiters cannot see the instruction, while the suffix can." While the paper claims this principle generalizes across templates, it lacks dedicated validation for non-standard conversational templates, bidirectional attention architectures, or interleaved multi-turn inputs.
The variable choices of \(\gamma\) (0.1, 0.2, or 0.3 across model families) suggest that calibration strength still needs model-specific tuning. Future work could adaptively determine the mixing scale based on the actual attention weights or representation distances of <BOS>.
Although the experiments span 15 image/video benchmarks over multiple model families, qualitative failure case analysis is limited. Future studies should analyze which tokens are consistently missed by CS scores in OCR-dense images, counting, tiny objects, and long videos requiring reviews of early evidence.

vs FastV / PDrop: These methods rank tokens primarily on raw attention values from a single text probe, which is simple but easily conflates positional bias and attention sinks with true semantic relevance. DIVA calculates differences between dual-probe distributions with minimal extra computation, remaining robust under strict budgets.
vs SparseVLM / ZipVL: While they also feature layer-wise sparsity and token reuse, DIVA explicitly addresses the representation gap between self-attention-active and self-attention-bypassed paths, quantifying the alignment distance before and after calibration.
vs D2Pruner: D2Pruner mitigates positional bias and incorporates structural diversity via normalization. DIVA instead bypasses categorizing bias types, unifying the mitigation of diverse distractors under dual-probe discrepancy, achieving a POPE F1 of 88.3 and a KV cache of 138.9 MB in the 80-token setup on LLaVA-NeXT-7B.

Rating¶

Novelty: ⭐⭐⭐⭐ Re-interpreting "large absolute attention" as "instruction-induced attention shifting" via dual-probe forward KLD is conceptually clear and targeted.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Covers multiple models, image/document/video tasks, extreme token budgets, and includes ablation on components, probes, metrics, hyperparameters, bias diagnostics, and efficiency.
Writing Quality: ⭐⭐⭐⭐ Good alignment between motivations, modules, and experiments. However, some equations in the PDF are tightly packed and require consulting the framework diagram during the first read.
Value: ⭐⭐⭐⭐⭐ Training-free, plug-and-play, and simultaneously improves accuracy preservation, self-attention computation, and KV cache footprint, providing direct utility for LVLM deployment.