FlowMM: Cross-Modal Information Flow Guided KV Cache Merging for Efficient Multimodal Context Inference¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Multimodal Large Language Models, KV Cache Compression, Information Flow, Token Merging, Sensitivity-Adaptive Matching
TL;DR¶
FlowMM reveals that multimodal large language models exhibit distinct layer-wise divergence in cross-modal attention flows, and proposes a tuning-free KV cache merging framework combining layer-adaptive merging domains with sensitivity-protected token matching, cutting cache memory by 80% to 95% and boosting decoding throughput by 1.3-1.8x while suffering only a 0.12% accuracy drop on InternVL2.5-8B.
Background & Motivation¶
Multimodal Large Language Models (MLLMs), such as Qwen2.5-VL, InternVL, and MobileVLM, have substantially expanded the horizons of artificial intelligence by binding high-capacity visual encoders with generative language model backbones. These architectures achieve remarkable empirical success across multi-image semantic reasoning, long-context narrative tracking, and dense visual dialogue. However, unlike purely textual sequences where token density is relatively modest, visual inputs encoded via Vision Transformers (ViT) generate voluminous token sequences per image patch. In autoregressive generation, storing Key-Value (KV) cache tensors for these extended sequences leads to severe memory inflation that rapidly exhausts high-bandwidth GPU memory (HBM) and chokes inference throughput in production.
To mitigate this bottleneck, previous research has explored two predominant non-quantization paradigms: KV eviction and KV merging. Eviction strategies, exemplified by H2O and SnapKV, compute cumulative or instant attention scores to aggressively prune KV pairs deemed non-critical. Unfortunately, permanently discarding states incurs irreversible context loss, frequently triggering hallucinations and breaking multi-step reasoning chains in long-context scenarios. Conversely, KV merging approaches, such as CaM and KVMerge, attempt to preserve contextual information by clustering and consolidating eviction candidates into retained states. Nevertheless, existing merging strategies are predominantly designed for unimodal text LLMs and falter in multimodal contexts due to severe distributional shifts between vision and text embeddings, where naive merging causes cross-modal interference and semantic distortion.
Crucially, mechanistic exploration into MLLM architectures unveils that cross-modal interaction intensity is highly non-uniform across Transformer layers. Shallow layers primarily engage in intra-modal feature extraction with minimal cross-modal attention transfer, whereas deep layers undergo a pronounced transition toward dense cross-modal fusion and high-level reasoning. Violating this natural computation trajectory—such as prematurely forcing cross-modal merging in shallow layers—destroys original unimodal representations and causes catastrophic performance collapse. Core idea: FlowMM introduces an adaptive multimodal KV cache merging framework that dynamically tailors merging strategies to layer-wise cross-modal attention flows, combined with a sensitivity-adaptive token matching strategy that protects task-critical states from dilution while safely merging redundant low-sensitivity representations without requiring any parameter fine-tuning.
Method¶
Overall Architecture¶
FlowMM functions as a modular, tuning-free inference acceleration pipeline integrated directly into the autoregressive decoding loop of MLLMs. The framework operates across three key stages: First, at each Transformer layer, it computes the ratio of cross-modal attention between vision and text tokens to measure interaction intensity and dynamically determine whether to execute conservative intra-modal merging or cross-modal consolidation; second, it identifies task-specific pivotal KV pairs using prompt proxy tokens to focus compression on contextually relevant states; third, it applies a sensitivity threshold to shelter high-influence pivotal tokens and merges remaining non-pivotal keys into low-sensitivity eligible targets based on cosine similarity.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Interleaved Multimodal Input<br/>XT, XI"] --> B["Cross-Modal Information Flow Guided Merging<br/>Layer-wise attention ratio guides intra vs cross modal merging"]
B --> C["Proxy-Guided Pivotal Set Selection<br/>Prompt-tail proxy tokens assess importance to split Top-B pivotal states"]
C --> D["Sensitivity-Adaptive Token Matching<br/>Cosine similarity matching with high-sensitivity target shielding"]
D --> E["Compressed KV Cache Output<br/>O(Lcomp) state tensors feed autoregressive decoding"]
Key Designs¶
1. Cross-Modal Information Flow Guided Merging: Heterogeneous distribution divergence and static merging -> Quantifying layer-specific cross-modal attention ratios to dynamically govern merging domains -> Preserving shallow unimodal representations while enabling deep cross-modal abstraction
Multimodal tokens occupy distinct representational subspaces, and applying uniform merging across all layers forces premature semantic mixing in shallow stages. To harmonize compression with internal model dynamics, FlowMM formulates the layer-specific cross-modal interaction ratio \(\rho_l\). For layer \(l\) and attention head \(h\), given visual token set \(V\) and text token set \(T\), the directional cross-modal attention sums are defined as: $\(A_{v \to t}^{l,h} = \sum_{v \in V} \sum_{t \in T} \alpha_{v \to t}^{l,h}, \quad A_{t \to v}^{l,h} = \sum_{t \in T} \sum_{v \in V} \alpha_{t \to v}^{l,h}\)$ Averaging over all \(H\) attention heads yields the layer ratio: $\(\rho_l = \frac{1}{H} \sum_{h=1}^H \frac{A_{v \to t}^{l,h} + A_{t \to v}^{l,h}}{A^{l,h}}\)$ where \(A^{l,h}\) denotes the total attention score across all tokens. FlowMM introduces a predefined threshold \(\theta\): when \(\rho_l > \theta\), the layer exhibits strong cross-modal synergy, and cross-modal merging is enabled; when \(\rho_l \le \theta\), the layer is restricted strictly to intra-modal merging (merging visual with visual, text with text). This constraint prevents early representation distortion while fully unlocking cross-modal compression in deep reasoning layers.
2. Proxy-Guided Pivotal Set Selection: Quadratic global clustering complexity and attention sinks -> Utilizing prompt-tail proxy tokens to assess task-focused importance and isolate Top-B pivotal states -> Filtering task-critical context while sharply bounding search overhead
Clustering the entire KV cache across long multimodal contexts incurs prohibitive quadratic computational overhead \(O(L_p^2)\), while cumulative attention metrics are often distorted by attention sinks. FlowMM overcomes these issues by designating a compact set of tokens at the very end of the prompt as proxy tokens \(P\), which inherently encapsulate the user instructions and current task queries. For any token \(i\) at layer \(l\) and head \(h\), its task importance score is computed as: $\(I^{l,h}(i) = \sum_{j \in P} \alpha_{j \to i}^{l,h}\)$ FlowMM isolates the top-\(B\) scoring tokens to form the pivotal set \(K_p\), delegating all remaining tokens to the non-pivotal candidate set \(K_n\). This design anchors the retention budget to task-salient context and confines subsequent matching exclusively between \(K_n\) and \(K_p\).
3. Sensitivity-Adaptive Token Matching: Dilution of task-critical representations by incoming merges -> Jointly filtering candidate targets via cosine similarity and sensitivity thresholds -> Shielding core reasoning pathways while cleanly consolidating redundant states
Certain pivotal tokens carry sharp, indispensable factual context whose representation would be corrupted or diffused if forced to absorb disparate non-pivotal keys. FlowMM adopts the proxy attention score \(I(i)\) as an efficient surrogate for token sensitivity and enforces a protection filter on eligible merge recipients. The semantic similarity between non-pivotal key \(k_i \in K_n\) and pivotal key \(k_j \in K_p\) is computed via normalized cosine similarity: $\(u_{i,j} = \frac{k_i^\top k_j}{\|k_i\|_2 \|k_j\|_2}\)$ To safeguard vital anchor representations, FlowMM defines the admissible target index set \(\mathcal{C}\) bounded by sensitivity threshold \(\tau\): $\(\mathcal{C} = \{j \mid k_j \in K_p, I(j) \le \tau\}\)$ Each non-pivotal key \(k_i \in K_n\) is merged strictly into its closest eligible peer: $\(j^*(i) = \arg\max_{j \in \mathcal{C}} u_{i,j}\)$ Pivotal keys exceeding threshold \(\tau\) are frozen as read-only anchors, preventing task-critical states from being diluted by background context merges.
Benchmark & Evaluation Protocol¶
The framework is rigorously validated on MileBench, the premier long-context multimodal benchmark spanning temporal multi-image reasoning, semantic multi-image QA, needle-in-a-haystack retrieval, and complex image search. Evaluated models comprise Qwen2.5-VL-7B, InternVL2.5-8B, and MobileVLM-V2-3B. FlowMM is benchmarked against leading eviction approaches (StreamingLLM, H2O), text-based merging methods (D2O, KVMerge), and the multimodal merging baseline LOOK-M under an identical 20% cache budget. Decoding latency and peak GPU memory are measured on a single NVIDIA A100 Tensor Core GPU.
Key Experimental Results¶
Main Results¶
The primary comparative evaluation across seven MileBench tasks under a strict 20% KV cache budget (80% memory reduction) is summarized below (directly quoted from Table 1 of the paper):
| Backbone & Method | ALFRED | IEdit | STD | MMCoQA | CLEVR-C | TextNeedle | ImageNeedle | Average | \(\Delta\) (vs Full) |
|---|---|---|---|---|---|---|---|---|---|
| Qwen2.5-VL-7B | |||||||||
| Full Cache (100%) | 36.92 | 30.16 | 28.13 | 50.50 | 45.45 | 11.56 | 24.38 | 32.44 | - |
| StreamingLLM | 27.61 | 30.43 | 26.85 | 46.00 | 37.00 | 4.38 | 1.88 | 24.88 | -7.57 |
| H2O | 34.31 | 30.91 | 26.63 | 45.50 | 42.49 | 4.69 | 5.00 | 27.08 | -5.36 |
| D2O | 33.59 | 30.56 | 26.16 | 39.50 | 41.58 | 4.69 | 8.75 | 26.40 | -6.04 |
| KVMerge | 27.94 | 31.16 | 27.83 | 44.50 | 37.95 | 9.69 | 15.00 | 27.72 | -4.72 |
| LOOK-M | 34.76 | 30.58 | 27.53 | 43.50 | 40.41 | 2.50 | 3.13 | 26.06 | -6.38 |
| FlowMM (Ours) | 35.43 | 31.67 | 28.08 | 48.50 | 41.79 | 10.00 | 17.13 | 30.37 | -2.07 |
| InternVL2.5-8B | |||||||||
| Full Cache (100%) | 35.34 | 9.12 | 26.37 | 52.50 | 22.76 | 25.00 | 24.69 | 27.97 | - |
| StreamingLLM | 23.36 | 10.71 | 25.33 | 51.50 | 19.39 | 10.00 | 2.50 | 20.40 | -7.57 |
| H2O | 33.05 | 11.31 | 26.21 | 51.50 | 21.01 | 10.93 | 21.25 | 25.04 | -2.93 |
| D2O | 32.63 | 11.08 | 26.64 | 50.00 | 20.82 | 11.25 | 21.25 | 24.81 | -3.16 |
| KVMerge | 33.61 | 11.42 | 26.33 | 51.50 | 20.47 | 17.63 | 21.45 | 26.06 | -1.91 |
| LOOK-M | 31.81 | 11.18 | 26.82 | 51.00 | 21.21 | 8.25 | 17.15 | 23.92 | -4.05 |
| FlowMM (Ours) | 34.77 | 11.93 | 26.59 | 52.00 | 22.58 | 23.12 | 23.93 | 27.85 | -0.12 |
| MobileVLM-V2-3B | |||||||||
| Full Cache (100%) | 25.18 | 6.55 | 13.57 | 7.00 | 15.97 | 9.68 | - | 12.99 | - |
| StreamingLLM | 13.73 | 5.82 | 7.65 | 2.50 | 6.55 | 3.12 | - | 6.56 | -6.43 |
| H2O | 24.86 | 6.22 | 10.27 | 3.00 | 14.06 | 2.50 | - | 10.15 | -2.84 |
| D2O | 23.96 | 6.48 | 11.59 | 4.50 | 13.85 | 4.34 | - | 10.79 | -2.20 |
| KVMerge | 24.47 | 6.39 | 11.51 | 4.00 | 14.67 | 5.38 | - | 11.07 | -1.92 |
| LOOK-M | 24.40 | 6.12 | 10.86 | 4.00 | 13.04 | 2.87 | - | 10.22 | -2.78 |
| FlowMM (Ours) | 25.06 | 6.57 | 12.73 | 5.50 | 15.39 | 8.67 | - | 12.32 | -0.67 |
Efficiency Analysis & Ablation Study¶
Latency, memory savings, and ablation results on Qwen2.5-VL-7B (from Tables 2 and 3 of the paper):
Table 2: Inference Speed and KV Cache GPU Memory Footprint (Qwen2.5-VL-7B on single A100) | Method | Cache Budget | Decoding Latency | KV Cache Memory | Speedup / Memory Reduction | |---|---|---|---|---| | Full Cache | 100% | 29.08 ms/token | 2.06 GiB | 1.00× (Baseline) / 0% | | FlowMM | 50% | 23.04 ms/token | 1.05 GiB | 1.26× / -49.0% | | FlowMM | 35% | 19.18 ms/token | 0.74 GiB | 1.52× / -64.1% | | FlowMM | 20% | 17.35 ms/token | 0.44 GiB | 1.68× / -78.6% | | FlowMM | 5% | 15.81 ms/token | 0.13 GiB | 1.84× / -93.7% |
Table 3: Ablation Study of Individual Components (Qwen2.5-VL-7B at 20% Cache Budget) | Configuration | TextNeedle (Accuracy) | STD (Accuracy) | ALFRED (Rouge-L) | Note | |---|---|---|---|---| | Full Cache | 11.56 | 28.13 | 36.92 | Full context retention | | FlowMM (Full framework) | 10.00 | 28.08 | 35.43 | Intact information flow + protection | | w/o Information Flow Guidance | 5.67 | 26.32 | 33.58 | Drops 4.33 on TextNeedle | | w/o Sensitivity-Adaptive Matching | 6.32 | 27.14 | 33.75 | Drops 3.68 on TextNeedle | | w/o Both | 3.61 | 25.24 | 31.01 | Collapses to naive merging |
Key Findings¶
- Near-lossless long-context compression: On InternVL2.5-8B under an aggressive 20% cache budget, FlowMM delivers an average score of 27.85%, trailing the uncompressed Full Cache baseline (27.97%) by merely 0.12%, while significantly outperforming H2O (25.04%) and LOOK-M (23.92%).
- Superiority in needle retrieval: In the challenging TextNeedle task, eviction baselines suffer massive accuracy collapse due to irreversible state pruning (StreamingLLM: 4.38%, H2O: 4.69%), whereas FlowMM preserves 10.00% accuracy on Qwen2.5-VL-7B. FlowMM at a 20% budget even outperforms eviction methods operating under a 60% budget.
- Catastrophic cost of inverted information flow: In preliminary experiments (Figure 2b), reversing the merging strategy (forcing inter-modal merging at shallow layers and intra-modal at deep layers) cut ALFRED task performance in half (from 36.92% to 18.5%), affirming that adhering to natural layer-wise cross-modal dynamics is indispensable.
Highlights & Insights¶
- Discovery of layer-wise multimodal attention divergence: The paper rigorously documents that early Transformer layers are predominantly unimodal while deeper layers dominate cross-modal fusion. This finding explains why text-centric KV compression collapses on MLLMs and provides a clear theoretical blueprint for multi-modal cache management.
- Zero fine-tuning plug-and-play efficiency: FlowMM requires no parameter retraining, auxiliary adapters, or offline calibration. On a single NVIDIA A100 GPU, it achieves up to 1.84x decoding speedup and 93.7% memory reduction, dramatically lowering the operational hardware bar for long-context vision-language serving.
- Proxy-based sensitivity protection principle: Utilizing prompt-tail proxy tokens to measure token sensitivity without perturbation overhead provides an elegant balance between task awareness and execution speed.
Limitations & Future Work¶
- Scale constraints in empirical validation: Due to compute constraints, the evaluation is primarily focused on 3B to 8B parameter models. Scaling behavior across ultra-large models (such as 34B or 70B+ architectures) and extreme context windows (e.g., millions of tokens) remains unverified.
- Scope limited to image-text modalities: FlowMM was designed around interleaved image and text inputs, leaving continuous audio streams and dense spatio-temporal video tokens unexplored.
- Future directions: Integrating FlowMM with ultra-low-bit KV cache quantization (such as 2-bit/4-bit mixed-precision schemes) and extending information flow tracking to streaming video reasoning.
Related Work & Insights¶
- vs H2O / SnapKV (Eviction methods): Eviction methods permanently drop non-salient KV states based on local attention scores, inducing severe context fragmentation; FlowMM merges rather than discards, retaining critical multi-hop context while maintaining equal sparsity.
- vs CaM / KVMerge (Unimodal merging methods): Unimodal merging assumes homogeneous representation distributions across tokens. FlowMM adaptively regulates cross-modal vs intra-modal merging per layer, eliminating semantic distortion.
- vs LOOK-M (Multimodal merging baseline): LOOK-M applies static modality separation without tracking cross-modal flow across network depth. FlowMM outperforms LOOK-M by over 4 percentage points on MileBench average accuracy.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Pioneering layer-wise information flow guided merging and sensitivity-adaptive token protection for multimodal KV cache]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive validation across 3 MLLM families, 7 long-context benchmarks, ablations, and physical GPU latency/memory benchmarks]
- Writing Quality: ⭐⭐⭐⭐⭐ [Exceptional conceptual clarity, rigorous mathematical formulations, and compelling narrative flow]
- Value: ⭐⭐⭐⭐⭐ [Plug-and-play framework slashing MLLM inference memory by up to 95%, highly impactful for edge and cloud deployment]