DIVA: Instruction-Aware Vision Token Pruning via Dual-Probe Attention Discrepancy

Conference: ECCV 2026
Paper: https://eccv.ecva.net/virtual/2026/poster/4758
Area: VLM Efficiency
Keywords: Vision Token Pruning, Instruction-Aware, Dual-Probe Attention Discrepancy, Training-Free Compression, KV Cache

TL;DR

Instead of directly using absolute attention values as vision token importance, DIVA compares the discrepancy in attention distributions over vision tokens between two sets of text probes—one that "sees the instruction" and one that "cannot see the instruction"—and then calibrates the bypassed tokens that skip self-attention. On LLaVA-NeXT-7B, retaining only 80 out of 2880 vision tokens preserves 92.6% of the baseline performance, while reducing the KV cache size by a factor of 10.8.

Background & Motivation

Large vision-language models (LVLMs) typically flatten images into hundreds or thousands of vision tokens, which are then fed into the language model along with text. These long vision sequences cause self-attention computation to grow quadratically with sequence length, while continuously occupying KV cache during autoregressive generation. Consequently, training-free vision token pruning is highly appealing: it requires no model retraining and selectively retains a small number of "important" tokens during inference. However, existing methods such as FastV, SparseVLM, and PDrop mostly rank token importance based on the absolute text-to-vision attention scores. These absolute values do not solely reflect the visual evidence required by the prompt, as they are often confounded by positional biases and visual attention sinks. A token irrelevant to the question but naturally attracting high attention may be retained indefinitely, while the actual local regions crucial for answering the question are discarded.

Layer-wise pruning allows re-evaluating token importance at each language model block, enabling previously unselected tokens to compete again in subsequent layers, thereby avoiding irreversible information loss caused by one-time pruning. However, this introduces a second issue: selected tokens undergo self-attention updates while bypassed tokens do not. The representation discrepancy between these two computational paths accumulates with depth, destabilizing the importance ranking in deeper layers.

This study disentangles "instruction relevance" from "high attention values": if a certain attention pattern appears under both instruction-aware and instruction-unaware contexts, it is likely a shared bias; only the discrepancy between the two reflects the genuine visual demands introduced by the instruction. Core Idea: Define a Context Shift (CS) score as the token-wise discrepancy between the attention distributions of dual probes, perform self-attention only on high-scoring vision tokens layer-by-layer, and employ a lightweight VF-Calibrator to simulate the missing self-attention aggregation for bypassed tokens, enabling their reliable reuse in subsequent layers.

Method

Overall Architecture

The input sequence consists of system tokens, vision tokens, and text tokens. The text is further divided into pre-instruction separators, the instruction itself, and suffixes like ASSISTANT:. DIVA retains full computation in the first two language model blocks to allow early cross-modal alignment. Starting from the third block, each layer dynamically constructs the dual probes, computes CS scores, and partitions the vision tokens into important and unimportant sets. Important tokens undergo normal self-attention, while unimportant tokens bypass this step and are processed by the VF-Calibrator. The two sets are then merged back in their original order, jointly passing through the residual path and FFN before entering the next block for re-evaluation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
        A["Image + Instruction<br/>fully pass through first two blocks"] --> B["Dual-Probe CS Scoring<br/>layer-wise re-evaluation of instruction relevance"]
        B -->|Top-K| C["Important Tokens<br/>perform self-attention"]
        B -->|Remaining| D["VF-Calibrator<br/>calibrate bypassed tokens"]
        C --> E["Merge back in original order<br/>Residual + FFN"]
        D --> E
        E -->|Next LM Block| B
        E --> F["Generate response"]

Key Designs

1. Dual-Probe CS Scoring: Finding visual evidence via "context-induced shifts" instead of absolute attention values

The non-contextual probe consists of separator tokens preceding the instruction. Due to the causal mask, it cannot perceive the subsequent instruction, making its attention distribution over vision tokens a reference baseline free of query semantics. The contextual probe initially includes the suffix tokens following the instruction, as they have already aggregated the preceding instruction context. Additionally, the paper filters the instruction tokens to identify those whose representations are closer to the average representation of the suffix rather than the prefix separator tokens, adding them to the contextual probe to prevent weak-context tokens from diluting the signal.

For each attention head and contextual probe, DIVA calculates the token-wise forward KLD contribution of its visual attention distribution relative to the non-contextual probe distribution. Let \(A^h_{t,i}\) and \(A^h_{0,i}\) be the attention probabilities of the \(i\)-th vision token under the two probes, respectively. The discrepancy is formulated as:

\[ D^h_{t,i}=A^h_{t,i}\log\frac{A^h_{t,i}}{A^h_{0,i}}. \]

To retain the strongest instruction-induced changes, DIVA takes the element-wise maximum over all contextual probes within the same head, and then averages across all attention heads to obtain \(CS_i=\frac{1}{H}\sum_h\max_{t\in\mathcal{T}_C}D^h_{t,i}\). This "shared-pattern cancellation, context-variation emphasis" ranking strategy is significantly less sensitive to positional biases and attention sinks. Experiments demonstrate that the relative performance using forward KLD is 98.8%, which is superior to JSD (98.1%) and backward KLD (95.9%), confirming that the direction of the divergence indeed matters.

2. Layer-wise Top-K Bypassing: Saving current-layer computation without permanently discarding visual information

In each layer, the Top-K important vision tokens are selected based on the CS scores. Only these tokens undergo self-attention and are written to the KV cache, while the remaining tokens are preserved in the sequence state rather than being permanently deleted. Upon merging, their original positions are restored, allowing the subsequent layer to re-score them based on the updated context. Regions deemed unimportant in the current layer can still be re-selected into the Top-K if they become relevant to the reasoning steps in deeper layers.

This design shifts "pruning" into a layer-wise sparse computation rather than a one-time input compression. The cost is that bypassed tokens must still pass through subsequent FFNs, and probe-related scores must be computed in each layer. However, the benefit is that the model is unlikely to suffer from irreversible loss of evidence under extremely constrained budgets due to early misjudgments. The paper enables DIVA starting from the third block to maintain full vision-language interaction in shallow layers.

3. VF-Calibrator: Bridging the representation gap for bypassed tokens via a shared global anchor

Bypassed tokens do not undergo self-attention value aggregation, so directly propagating them to the next layer causes their representations to deviate from the trajectory of normally updated tokens. VF-Calibrator reuses the existing values and output projections of the current attention layer without introducing any training. It blends the self-value feature of each bypassed token with the global reference value of <BOS>, and then performs a residual update through the output projection. For a bypassed token \(v_j\), the calibrated feature is formulated as:

\[ \hat{\mathbf v}_{j}=\gamma\mathbf v^{\mathrm{self}}_{j}+(1-\gamma)\mathbf v^{\mathrm{ref}},\qquad \mathbf h_j\leftarrow\mathbf h_j+\mathrm{Proj}_O(\hat{\mathbf v}_j). \]

Since <BOS> often acts as a high-attention anchor aggregating global context, this static mixture approximates the missing contextual aggregation of bypassed tokens with neglectable overhead. On LLaVA-1.5-7B, setting \(\gamma=0.1\) yields the best result; increasing it to 0.5 degrades relative performance from 98.8% to 92.7%, indicating that the global reference is indispensable. The cosine distance of representations in the last layer falls from 0.77 (uncalibrated) to 0.28, supporting that the module indeed bridges the representation gap between the pathways.

A Complete Example

Suppose LLaVA-NeXT encodes an image into 2880 vision tokens, and the query is "Is the boat moving or stationary?". The first two language model blocks process all tokens fully. In a subsequent block, the contextual probe is composed of the ASSISTANT: suffix and filtered instruction tokens (such as "boat" and "moving"), while the non-contextual probe consists of the newline separator preceding the instruction. If attention on the boat and surrounding water changes significantly under the contextual probe compared to the non-contextual probe, whereas certain sequence-end tokens maintain high attention under both probes, the CS score will upgrade the former and downgrade the latter.

Under an 80-token budget, this layer allows only the top-80 ranked tokens to execute self-attention, while the remaining 2800 tokens are processed by the VF-Calibrator before being merged back to their original positions. Since the CS score is recalculated in the next layer, new local evidence can still be dynamically selected. This entire process reduces the KV cache from 1496.0 MB to 138.9 MB, while simultaneously improving POPE F1 from 86.5 (unpruned) to 88.3.

Loss & Training

DIVA is a fully training-free, inference-time approach that introduces no additional loss functions and keeps the original LVLM parameters fixed. The primary hyperparameters are the number of retained vision tokens per layer, \(K\), and the VF-Calibrator mixing coefficient, \(\gamma\). In the experiments, \(\gamma=0.1\) is uniformly used for LLaVA-1.5-7B, 0.2 for LLaVA-NeXT-7B and LLaVA-Video-7B, and 0.3 for Qwen3-VL-8B and InternVL3.5-8B. All efficiency metrics are evaluated on a single NVIDIA RTX A6000 GPU.

Key Experimental Results

Main Results

Model / Scenario	Token Budget	Representative Metric	DIVA	Key Comparison	Relative Baseline Performance
LLaVA-1.5-7B, General Visual Reasoning	128 / 576	GQA	61.4	Vanilla 62.0	100.9%
LLaVA-1.5-7B, General Visual Reasoning	64 / 576	MME	1833	Vanilla 1862	99.8%
LLaVA-1.5-7B, Extreme Compression	8 / 576	Overall Relative Performance	89.0%	FastV@64 75.4%	89.0%
LLaVA-NeXT-7B, High Resolution	80 / 2880	POPE F1	88.3	Vanilla 86.5	92.6%
Qwen3-VL-8B, Fine-Grained Reasoning	Retain 10%	DocVQA	81.3	PDrop 56.4	82.6%
LLaVA-Video-7B, Video Understanding	16 Tokens/Frame	Video-MME Short	73.1	DivPrune@32 67.7	94.5%

Ablation Study

Configuration	Relative Performance	Additional Evidence / Efficiency	Description
Neither: FastV-style scoring + direct restoration of bypassed tokens	87.9%	-	Both issues co-exist
Only VF-Calibrator	95.6%	-	Calibration alone significantly improves reuse
Only DIVA-Pruner	97.8%	-	CS ranking is the primary contributor
Full DIVA	98.8%	Last-layer representation distance: 0.28	The two modules complement each other
Full DIVA, 80 Tokens (LLaVA-NeXT)	POPE 88.3	15.3 GFLOPs / 0.58 ms / 138.9 MB	Reduced by 34.6× / 7.2× / 10.8× compared to unpruned
Full DIVA, 40 Tokens (LLaVA-NeXT)	POPE 86.7	9.7 GFLOPs / 0.51 ms / 120.4 MB	Reduced by 54.6× / 8.2× / 12.4× compared to unpruned

Key Findings

• CS ranking makes the primary contribution, but is not the sole factor: enabling only DIVA-Pruner yields 97.8% relative performance, which increases to 98.8% when the VF-Calibrator is added; dynamic calibration alone delivers 95.6%, proving that representation gaps in layer-wise token reuse constitute a practical bottleneck.
• The advantages become more pronounced under extremely constrained budgets. When LLaVA-1.5-7B retains only 32 tokens, DIVA maintains 96.1% relative performance, compared to 95.2% for CDPruner and 91.6% for VisionDrop; with only 8 tokens retained, it still achieves 89.0%.
• The method is not restricted to a single model or static images; it covers LLaVA, Qwen3-VL, InternVL3.5, and LLaVA-Video, demonstrating consistent advantages across document QA and video tasks. However, since different models utilize different values of \(\gamma\), these figures should not be interpreted as direct cross-comparisons under a single set of hyperparameters.

Highlights & Insights

• The most design ingenuity lies in treating attention biases as shared "background terms" across dual probes. Instead of diagnosing and manually correcting each bias source (such as positional biases or attention sinks) individually, the method isolates instruction-induced signals directly via context discrepancies. This formulation is highly transferable to the dynamic filtering of textual tokens, video frames, or retrieval candidates.
• DIVA does not equate bypassing with permanent deletion. Allowing tokens to be dynamically re-selected at each layer, combined with lightweight calibration, makes compression behave more like a reversible computational scheduling. This is far more resilient than one-time pruning for scenarios where visual focus shifts dynamically during multi-step reasoning.
• VF-Calibrator reuses pre-trained projections and leverages <BOS> to approximate global context aggregation without training new adapters. This demonstrates a general strategy: when representation drift occurs between sparse and dense computational paths, alignment can be achieved cost-effectively using existing aggregation anchors inherent to the model.

Limitations & Future Work

• The method still needs to store the states of all bypassed tokens, perform layer-wise re-scoring, and execute FFNs. The primary reported benefits lie in self-attention calculation and KV cache capacity rather than the entire model's computational load. Consequently, the 54.6× reduction in self-attention GFLOPs does not translate to a 54.6× speedup in end-to-end latency.
• Probe construction relies on the prompt template structure under causal masking, where "prefix separators cannot view instructions, but suffixes can." While the paper posits that this principle generalizes across templates, dedicated validation is still lacking for non-standard dialog templates, bidirectional attention architectures, or interleaved multi-turn text-image inputs.
• The hyperparameter \(\gamma\) takes different values (0.1, 0.2, or 0.3) across model families, indicating that the calibration intensity needs model-specific tuning. Future work could adaptively determine the mixing ratio based on the actual attention weight of <BOS> or representation distance in the current layer, instead of manual thresholding.
• Although the experiments cover 15 image/video benchmarks across various model families, qualitative failure cases are under-analyzed. Future research should investigate which tokens are continuously overlooked by CS scoring in dense OCR images, counting tasks, tiny objects, or long video understanding that requires retrieving early evidence.

Related Work & Insights

• vs FastV / PDrop: These methods rank token importance using the absolute attention values of a single text probe, which is simple but tends to confuse positional biases and attention sinks with query relevance. In contrast, DIVA ranks tokens by the distribution discrepancy between instruction-aware and instruction-unaware contexts, which delivers far greater stability under tight token budgets with minimal extra computational overhead.
• vs SparseVLM / ZipVL: These methods also consider layer-wise sparsity and token reuse. However, DIVA explicitly addresses the representation gap between tokens that undergo self-attention and those that bypass it, quantifying the distances before and after calibration through detailed experiments.
• vs D2Pruner: D2Pruner mitigates positional biases through normalization and integrates structural diversity, whereas DIVA does not enumerate bias types but uniformly suppresses mutual interference via dual-probe discrepancies. Under the 80-token configuration on LLaVA-NeXT-7B, DIVA achieves a POPE F1 score of 88.3 alongside a 138.9 MB KV cache.

Rating

• Novelty: ⭐⭐⭐⭐ Formulating token importance as "instruction-induced attention shifting" rather than "absolute high attention" via dual-probe forward KLD is a clear and target-oriented approach.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive coverage across multiple models, image/document/video tasks, and extreme token budgets, accompanied by analyses of components, probes, metrics, hyperparameters, bias diagnostics, and efficiency.
• Writing Quality: ⭐⭐⭐⭐ The motivations, modules, and experiments are tightly coupled, although some mathematical formulas in the PDF are tightly spaced, requiring reference to the architecture diagram for a clear first-time reading.
• Value: ⭐⭐⭐⭐⭐ Training-free and plug-and-play, simultaneously improving accuracy retention, self-attention computation, and KV cache consumption, presenting direct value for LVLM deployment.

Related Papers

• [ECCV 2026] Paying More Attention to Visual Tokens in Self-Evolving Large Multimodal Models
• [ECCV 2026] Spectral Evolution-Guided Token Pruning in Multimodal Large Language Models
• [ECCV 2026] Accelerating Multimodal Large Language Models with Prior-Corrected Token Reduction
• [ECCV 2026] SMART: When is it Actually Worth Expanding a Speculative Tree?
• [ECCV 2026] ViQ: Text-Aligned Visual Quantized Representations at Any Resolution