Skip to content

VisCache: Visual KV Cache Pruning for Efficient Vision Large Language Models

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Wlklk/VisCache
Area: Model Compression
Keywords: Vision Large Language Models, KV Cache Pruning, Visual Attention Redundancy, Parabolic Budget Allocation, Asymmetric Update

TL;DR

VisCache is a training-free and plug-and-play coarse-to-fine visual KV cache pruning framework that filters temporal redundancy with a lightweight scout VLM and surgically compresses visual KV states via PruneKV—featuring a parabolic layer-wise budget schedule and asymmetric key pruning with value fusion—achieving up to 2.35× end-to-end speedup while retaining only 19–28% of the visual KV cache without degrading multimodal reasoning accuracy.

Background & Motivation

Vision Large Language Models (VLLMs, such as LLaVA and the Qwen-VL series) have demonstrated remarkable proficiency across complex multimodal reasoning and long-video comprehension benchmarks. However, scaling VLLM inference to long video streams and high-resolution frames produces a massive volume of visual tokens that severely burdens memory and computation. The inflated visual Key-Value (KV) cache dominates GPU memory capacity and triggers intensive memory bandwidth contention, while the quadratic scaling of self-attention mechanisms substantially escalates both Time to First Token (TTFT) and Time per Output Token (TPOT), posing prohibitive bottlenecks for real-world deployment on resource-constrained platforms.

To alleviate these pressures, recent plug-and-play acceleration approaches have explored KV cache compression, predominantly applying rigid and uniform reduction strategies such as layer-by-layer linear arithmetic decay (e.g., PyramidKV) or stage-wise geometric truncation (e.g., PDrop). Nevertheless, these uniform pruning paradigms overlook the structured, heterogeneous dynamics intrinsic to vision-language processing. In long-video understanding, temporal relevance is highly sparse, meaning only a compact fraction of video frames contains query-relevant cues. Furthermore, transformer representations exhibit distinct hierarchical abstraction across depth: shallow layers encode fine-grained spatial and geometric textures that require wider token capacity, whereas deeper layers encapsulate abstract semantics where token redundancy is pronounced. Finally, keys and values play fundamentally asymmetric roles in attention computation: keys act as relevance selectors governing the attention distribution, whereas values serve as the actual information carriers encoding contextual representations. Discarding keys and values uniformly inevitably induces irreversible loss of salient context.

Motivated by these structural characteristics, the authors rethink visual KV cache compression from a joint temporal, hierarchical, and attention-functional perspective. Core idea: establish a training-free, coarse-to-fine pruning framework, VisCache, that synergistically deploys a lightweight scout VLM with Maximal Marginal Relevance (MMR) to eliminate temporal frame redundancy before inference, and executes PruneKV during inference to dynamically allocate parabolic layer-wise budgets and asymmetrically prune keys while fusing dropped values into retained tokens.

Method

Overall Architecture

VisCache operates via two synergistic stages: pre-inference temporal redundancy filtering (Stage 1: Prompt-Aware Scout) and in-inference fine-grained KV cache compression (Stage 2: PruneKV). Given an input video and a user prompt, a lightweight cross-modal scout model (e.g., pretrained CLIP ViT-B/32) computes multimodal embeddings and selects a compact, query-relevant, and non-redundant keyframe subset using the Maximal Marginal Relevance (MMR) criterion. During the subsequent VLLM prefilling phase, attention weights across all transformer layers are aggregated to establish a shared global visual token ranking. During generation, PruneKV enforces a truncation threshold (\(h = \frac{3}{4}L\)) that completely evicts visual KV states in deeper layers, while distributing retention budgets across layers \(1 \le l \le h\) according to a smooth parabolic schedule. For tokens designated for eviction, PruneKV prunes their key vectors entirely to streamline attention addressing, but softly redistributes and fuses their value vectors into the retained tokens via a temperature-scaled affinity matrix, constructing a compact and semantically rich KV cache for autoregressive decoding.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Long Video Frames & Text Prompt"] --> B["1. Prompt-Aware Scout for Temporal Redundancy Filtering<br/>Cross-modal cosine similarity with MMR diversity penalty"]
    B --> C["2. Shared Global Attention Scoring & Ranking<br/>Decouple token saliency identification from layer budget"]
    C --> D["3. Parabolic Layer-Wise Budget Allocation<br/>h=3/4L truncation and parabolic decay preserving shallow details"]
    D --> E["4. Asymmetric Key Pruning with Value Fusion<br/>Prune discarded keys and fuse discarded values via affinity matrix"]
    E --> F["Compressed Visual KV Cache for Fast Autoregressive Decoding"]

Key Designs

1. Prompt-Aware Scout for Temporal Redundancy Filtering: eliminating input-level temporal bloat at negligible cost Raw long-video inputs contain substantial static and irrelevant background frames that unnecessarily inflate the prefilling sequence length. VisCache deploys an off-the-shelf, lightweight vision-language model (defaulting to CLIP ViT-B/32) as an external scout before activating the heavy VLLM backbone. The textual query \(T\) and candidate frame \(f\) are projected into aligned embeddings \(h_t = \mathrm{Enc}_{\mathrm{text}}(T)\) and \(h_f = \mathrm{Enc}_{\mathrm{vis}}(f)\). To jointly balance semantic relevance to the query against intra-frame visual redundancy, keyframes are iteratively chosen into the active set \(\Omega\) using the Maximal Marginal Relevance (MMR) criterion: $\(\mathrm{MMR}(f) = \lambda \cdot \mathrm{sim}(h_f, h_t) - (1 - \lambda) \cdot \max_{f' \in \Omega} \mathrm{sim}(h_f, h_{f'})\)$ where \(\mathrm{sim}(\cdot, \cdot)\) denotes cosine similarity and \(\lambda \in [0, 1]\) controls the relevance-diversity trade-off (set to 0.7). Frames with the highest MMR scores are iteratively selected until reaching the target frame retention ratio \(p\). Empirical analysis shows high Jaccard similarity (0.70–0.85) between the visual tokens of scout-selected keyframes and the top attention tokens identified by the full VLLM backbone, confirming that temporal filtering safely discards non-informative frames prior to heavy computation.

2. Shared Global Attention Scoring: decoupling token saliency from layer-wise capacity Standard layer-specific token pruning evaluates importance independently per layer, which frequently breaks the representational continuity of visual entities as token identities fluctuate across depth. PruneKV accumulates attention weights received by each visual token \(v\) across all queries \(i\) and all \(L\) layers during prefilling to compute a unified importance score: $\(s_v = \frac{1}{L} \sum_{l=1}^L \sum_{i=1}^N A_{i, v}^l\)$ where \(A_{i, v}^l\) denotes the attention weight directed from token \(i\) to visual token \(v\) at layer \(l\). This shared global ranking decouples which tokens are universally salient from how many tokens each specific layer should retain. By smoothing out localized attention fluctuations of individual layers, the global ranking preserves consistent semantic features across the network hierarchy.

3. Parabolic Layer-Wise Budget Allocation: aligning compression budgets with hierarchical feature abstraction Vision Transformers process representations hierarchically: early layers capture fine-grained spatial and localized geometric textures requiring dense visual tokens, while deeper layers compress tokens into abstract semantic concepts where redundant tokens can be aggressively pruned. PruneKV establishes a truncation depth \(h \in (1, L]\) (defaulting to \(h = \frac{3}{4}L\)), beyond which visual KV states in layers \(l > h\) are completely evicted. Within the retained layers \(l \in [1, h]\), layer-wise token retention budgets \(b_l\) follow a parabolic decay schedule: $\(b_l = 1 - \frac{(l - 1)^2}{2(h - 1)^2}\)$ This schedule naturally anchors \(b_1 = 1.0\) (complete preservation at layer 1) and \(b_h = 0.5\), decaying gently across shallow layers and steepening across intermediate and deeper layers. Compared to linear arithmetic schedules (PyramidKV) or geometric step-downs (PDrop), the parabolic curve maintains a generous buffer in early layers to safeguard fine-grained visual details while ramping up compression at deeper stages, mirroring the natural information funnel of multimodal transformers.

4. Asymmetric Key-Value Pruning and Fusion: preserving attention routing while salvaging dropped semantic context In scaled dot-product attention \(\mathrm{Softmax}(\frac{QK^\top}{\sqrt{d}})V\), keys and values serve distinct functional roles: keys determine the routing distribution through dot-product matching with queries, whereas values carry the actual content aggregated into the output. Conventional methods prune keys and values in lockstep, discarding all content carried by pruned values. PruneKV exploits this asymmetry: - Key Pruning: For the drop set \(\mathcal{C}_d\) (the lowest-scoring tokens beyond budget \(b_l\)), their key vectors are completely removed (\(K_{\mathrm{new}}^l \leftarrow K_k\)), as their contribution to the attention distribution is negligible. - Value Fusion: Instead of discarding the corresponding value vectors \(V_d \in \mathbb{R}^{(n - b_l) \times d}\), PruneKV computes a redistribution affinity matrix \(\Phi\) against the retained value cache \(V_k \in \mathbb{R}^{b_l \times d}\): $\(\Phi = \mathrm{Softmax}\left(\frac{V_k V_d^\top}{\tau}\right) \in \mathbb{R}^{b_l \times (n - b_l)}\)$ where \(\tau\) is a temperature hyperparameter (default \(\tau = 1.0\)). Dropped values are aggregated into retained positions via \(\Phi V_d\) and merged through a convex combination: $\(V_{\mathrm{new}}^l = \mu V_k + (1 - \mu)(\Phi V_d)\)$ where \(\mu \in [0, 1]\) is the fusion strength factor (empirically optimized at \(\mu = 0.7\)). This mechanism eliminates unneeded key addressing overhead while recycling residual contextual information back into the active KV cache.

Benchmark & Evaluation Protocol

  • Benchmarks & Metrics: Evaluated across Video Summarization (VS) on ActivityNet Captions (ActCap, 20K videos / 100K captions) and DREAM1K (1,000 long-form clips), reporting ROUGE-L; and Video Question Answering (VQA) on NExTQA (causal and temporal reasoning), ActivityNet-QA (ActQA, 5.8K open-ended videos), EgoSchema (long-form egocentric video comprehension), and MVBench (20 fine-grained multi-modal video tasks), reporting Accuracy (%).
  • Implementation & Hardware: Implemented on Qwen2.5-VL-3B-Instruct and Qwen2.5-VL-32B-Instruct using PyTorch on 4 NVIDIA A100 (80GB) GPUs; cross-architecture generalization evaluated on Qwen3-VL-4B-Instruct and LLaVA-OneVision-Qwen2-7b-ov-hf. Hyperparameters are fixed to: MMR weight \(\lambda = 0.7\), truncation layer \(h = \frac{3}{4}L\) (layer 48 on 64-layer 32B model), average layer budget \(m = 0.75\), temperature \(\tau = 1.0\), and fusion factor \(\mu = 0.7\). Global visual KV retention ratio follows: \(\mathrm{RR} = p \times q \times \frac{h}{L} \times m\).

Key Experimental Results

Main Results

Under matched or more aggressive retention ratios (RR), VisCache consistently achieves superior efficiency-performance trade-offs compared to plug-and-play compression baselines (PyramidKV, FastV, PDrop, Q-Frame). Below are the core comparative results from Table 1, Table 11, and Table 12:

Model / Method RR FLOPs (T) FLOPs Ratio ActCap (ROUGE-L) DREAM1K (ROUGE-L) NExTQA (Acc) ActQA (Acc) EgoSchema (Acc) Avg. Acc (%)
Qwen2.5-VL-3B
Full Cache 100% 14.80 100% 2.63 9.19 34.69 40.58 57.20 44.16
Q-Frame 40% 6.26 42% 2.42 5.86 39.52 39.42 46.40 41.78
PyramidKV 40% 1.99 13% 2.43 8.48 39.53 39.56 54.80 44.63
FastV 40% 3.11 21% 2.43 8.63 37.67 40.65 50.40 42.91
PDrop 40% 2.99 20% 2.46 8.63 34.24 40.21 50.00 41.48
VisCache (\(p=0.75, q=0.95\)) 40% 1.32 9% 2.35 9.79 37.62 37.32 52.00 42.31
VisCache (\(p=0.75, q=0.67\)) 28% 1.05 7% 2.45 8.70 41.25 40.66 55.00 45.64
VisCache (\(p=0.50, q=0.67\)) 19% 0.88 6% 2.39 8.26 41.44 38.51 54.60 44.85
Qwen2.5-VL-32B
Full Cache 100% 93.08 100% 2.82 7.87 60.71 46.47 65.20 57.46
Q-Frame 40% 40.80 44% 2.81 7.84 50.08 43.86 54.60 49.51
PyramidKV 40% 13.88 15% 2.78 7.55 61.58 42.96 65.20 56.58
FastV 40% 51.63 55% 2.81 7.47 60.29 38.41 61.60 53.43
PDrop 40% 31.03 33% 2.75 7.38 61.00 41.59 63.20 55.26
VisCache (\(p=0.75, q=0.67\)) 28% 10.75 12% 2.72 7.29 62.08 42.69 65.80 56.86
VisCache (\(p=0.50, q=0.67\)) 19% 9.08 10% 2.81 7.35 56.61 41.86 65.00 54.16

In terms of real-system deployment efficiency on ActCap with Qwen2.5-VL-3B-Instruct, Full Cache requires 5.57 GB total GPU memory, 0.06 GB KV cache memory, achieving 10.4 TPS. VisCache at 28% RR reduces total GPU memory to 4.10 GB and boosts throughput to 18.6 TPS. At 19% RR, total memory drops to 3.68 GB (a 33.9% reduction). On DREAM1K, VisCache (19% RR) achieves a 2.35× end-to-end acceleration, reducing latency from 12.24s to 5.20s and slashing TPOT from 118.81ms to 45.60ms.

Ablation Study

On Qwen2.5-VL-32B-Instruct with a fixed global visual KV cache retention ratio of \(\mathrm{RR} = 28\%\), the budget allocation schedule and asymmetric value fusion mechanism were systematically isolated (Table 4 and Table 5):

Budget Strategy Value Cache Fusion ActCap (ROUGE-L) DREAM1K (ROUGE-L) NExTQA (Acc) ActQA (Acc) EgoSchema (Acc) Avg. Acc (%)
Full Cache (Upper bound) — 2.82 7.87 60.71 46.47 65.20 57.46
Without Value Fusion
Fixed (Uniform) ✗ 2.43 7.24 59.00 43.44 63.60 55.35
Arithmetic (PyramidKV) ✗ 2.67 7.17 58.07 42.31 57.00 52.46
Geometric (PDrop) ✗ 2.69 7.25 55.56 42.68 57.60 51.95
Parabola (Ours) ✗ 2.70 7.05 61.61 42.64 65.00 56.42
With Value Fusion
Fixed (Uniform) ✓ 2.69 7.36 61.58 42.42 64.80 56.27
Arithmetic (PyramidKV) ✓ 2.72 7.23 61.32 42.33 65.00 56.22
Geometric (PDrop) ✓ 2.69 7.27 56.07 40.49 56.00 50.85
Parabola (Full VisCache) ✓ 2.72 7.29 62.08 42.69 65.80 56.86

Two-Stage Synergy Ablation (Scout vs. PruneKV): On EgoSchema at 28% RR, using Scout filtering alone yields 46.00% accuracy; using PruneKV alone yields 48.80%; combining both in VisCache reaches 55.00% (a 6.2–9.0 point leap). At 19% RR, VisCache achieves 54.60% versus 48.20% (Scout only) and 48.60% (PruneKV only), corroborating the necessity of joint coarse-to-fine compression.

Key Findings

  • Counter-Intuitive Accuracy Gain under 28% Retention: On Qwen2.5-VL-3B, VisCache at 28% RR achieves an average VQA accuracy of 45.64%, surpassing both baseline methods and the unpruned Full Cache upper bound (44.16%). This highlights that dense, unpruned visual tokens in long videos introduce visual clutter and noise; surgical pruning sharpens multimodal cross-attention onto salient action events.
  • Early-Layer Protection is Critical under Tight Budgets: Geometric decay (PDrop) collapses under 28% RR (50.85% average accuracy) due to premature token elimination in early layers that damages low-level geometric cues. In contrast, the parabolic schedule's flat initial slope preserves essential spatial primitives before accelerating compression deeper down.
  • Universal Benefit of Asymmetric Value Fusion: Across Parabola, Fixed, and Arithmetic schedules, enabling value fusion consistently improves average accuracy by 0.44 to 3.76 points. This proves that discarded values contain reusable contextual information that can be effectively restored into retained tokens via softmax affinity weighting.

Highlights & Insights

  • Dual-Stage Funnel Architecture: Synergizing an external, lightweight Scout VLM for temporal frame filtering with internal attention-guided PruneKV establishes a coarse-to-fine reduction pipeline that expands the efficiency-accuracy Pareto frontier beyond what single-stage algorithms can reach.
  • Attention Asymmetry Exploitation ("Prune Key, Fuse Value"): Recognizing that keys dictate routing whereas values embody content, VisCache eliminates key tokens to trim attention complexity while softly aggregating discarded values into retained tokens, preventing irreversible semantic degradation under aggressive compression.
  • Parabolic Scheduling Matched to Multimodal Abstraction: Departing from heuristic linear or geometric token drops, the parabolic decay function \(b_l = 1 - \frac{(l-1)^2}{2(h-1)^2}\) accurately accommodates the transition from dense spatial textures in shallow layers to sparse abstract semantics in deeper layers.

Limitations & Future Work

  • Cross-Modal Embedding Discrepancy between Scout and Backbone: The lightweight scout (CLIP ViT-B/32) operates with a distinct visual encoder and pretraining objective from the main autoregressive VLLM backbone. In fine-grained or subtle transitional scenes, the scout may occasionally drop frames that the backbone could have leveraged. Future directions include lightweight cross-modal distillation or bidirectional feedback loops.
  • Memory Footprint of Attention Accumulation during Prefilling: PruneKV requires storing and summing attention weights across all transformer layers during the prefilling phase to construct the global score \(s_v\). For ultra-long video inputs, this creates a transient peak in activation memory. Developing streaming or sampled estimation algorithms would facilitate further efficiency gains.
  • vs PyramidKV / PyramidInfer: PyramidKV relies on an arithmetic progression and uniformly drops both keys and values. VisCache introduces a smoother parabolic schedule combined with asymmetric value fusion, achieving far superior stability under aggressive retention rates (19%–28%).
  • vs PDrop / FastV: PDrop partitions the network into rigid blocks with geometric token drops, risking early-layer feature collapse; FastV employs a naive single-layer cutoff. VisCache decouples importance ranking from layer-wise budgets via shared global scoring and filters redundant frames upstream.
  • vs Post-Training Quantization (KIVI / FlatQuant): VisCache provides structural sparsity and is fully orthogonal to weight/KV quantization. Integrating VisCache (28% RR) with 4-bit KV quantization methods maintains strong generation and reasoning performance, demonstrating great potential for extreme memory-constrained edge deployment.

Rating

  • Novelty: ⭐⭐⭐⭐☆ Highly innovative formulation leveraging attention functional asymmetry and parabolic decay in a training-free KV cache compression framework.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation spanning 6 video benchmarks, 20 fine-grained MVBench tasks, multiple VLLM families (Qwen2.5-VL, Qwen3-VL, LLaVA-OneVision), real-system memory profiling, and extensive ablations.
  • Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical modeling, lucid prose, clean figures, and well-structured empirical discussions.
  • Value: ⭐⭐⭐⭐⭐ Plug-and-play, training-free, and delivers up to 2.35× latency speedups and >30% memory savings, offering substantial utility for long-video multimodal applications.