Can Retrieval Heads See Images? Multimodal Retrieval Heads in Long-Context Vision-Language Models¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/HKUST-KnowComp/MMRetHead
Area: Large Language Model Reasoning
Keywords: Multimodal Large Language Models, Long-Context Retrieval, Retrieval Heads, Mechanistic Interpretability, Document Re-Ranking
TL;DR¶
This work formalizes and detects multimodal retrieval heads (MMRetHeads) in long-context vision-language models, demonstrating that a sparse set of 4.4%–10.2% of attention heads causally drives multimodal evidence retrieval and reasoning, while their aggregated attention directly yields state-of-the-art zero-shot document re-ranking on MMDocIR.
Background & Motivation¶
Long-context vision-language models (LCVLMs) such as Qwen2.5-VL, Qwen3-VL, and Gemma3 have expanded their operational context windows to hundreds of thousands of tokens, enabling seamless reasoning over lengthy multi-page documents, hour-level video streams, and extended embodied agent trajectories. Performing robust multi-step reasoning across such vast multimodal contexts fundamentally hinges on the model's ability to precisely locate task-relevant evidence items amidst extensive distractor text passages and interleaved images. In text-only large language models, this evidence-retrieval capability has been mechanistically explained via "retrieval heads" identified by observing token-by-token copy-paste behavior from needles during Needle-in-a-Haystack (NIAH) tasks. However, when evidence is visual—embedded within diagrams, tables, charts, or photographs—the model frequently synthesizes an answer through cross-modal semantic understanding without copying any prompt token verbatim, causing copy-based detection criteria to break down completely.
Furthermore, our understanding of the internal causal mechanisms governing how multimodal attention routes cross-modal evidence remains remarkably limited. While raw attention distributions have been examined for multimodal grounding, hallucination tracking, and cross-modal alignment, raw attention scores are heavily contaminated by query-independent positional sinks, token-type priors, and modality-dependent aggregation biases. Without a rigorous calibration mechanism, merely observing high attention weights does not provide a faithful causal explanation of evidence extraction. Crucially, open fundamental questions persist: do vision-language models reuse pre-existing language retrieval circuits for visual evidence, or do they establish dedicated, modality-specific visual retrieval heads? Moreover, how do varying text-to-image ratios in the haystack and context length scaling from 8K to 128K dynamically perturb the topology and layer distribution of these critical attention circuits?
To address these foundational tensions, this paper steps beyond superficial string-matching criteria to formulate evidence retrieval around post-softmax attention mass concentrated from question tokens into annotated evidence spans. By introducing null-question calibration to strip away background query-independent attention sinks across text, image, rendered-text (OCR), and identical-image verification tasks, the authors provide the first systematic detection and mechanistic characterization of multimodal retrieval heads. Core idea: detect multimodal retrieval heads (MMRetHeads) by scoring calibrated question-to-evidence attention mass, revealing their extreme sparsity, intrinsic pretraining origin, and dynamic modality adaptation, while directly repurposing their attention into a superior zero-shot multimodal document re-ranker.
Method¶
Overall Architecture¶
The framework establishes a comprehensive pipeline spanning detection, causal validation, and downstream application of multimodal retrieval heads. Given an interleaved multimodal context (the haystack) and an evidence-seeking query, the system first conducts dual forward passes with task queries and uninformative null queries to derive bias-calibrated retrieval scores, ranking and isolating the top-5% MMRetHeads. Next, it executes rigorous causal interventions—including zero-masking during both prefill and decoding as well as path-specific information-flow blocking—to evaluate performance degradation across long-context NIAH, document visual question answering, and complex multimodal reasoning benchmarks. Finally, without any additional fine-tuning or external supervision, the post-softmax attention from these identified retrieval heads is aggregated across candidate pages and layout elements to execute zero-shot multimodal document re-ranking.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Interleaved Multimodal Context & Needle Spans<br/>Standardized MM-NIAH Benchmark"] --> B["Null-Question Calibrated Scoring<br/>Post-Softmax Attention Mass Accumulation"]
B --> C["Mechanistic Topology Characterization<br/>Sparsity, Intrinsicity & Modality Dynamics"]
C --> D["Two-Stage & Path-Specific Causal Masking<br/>Prefill + Decode Zero & Information Flow Interventions"]
D --> E["Zero-Shot Document Attention Re-Ranking<br/>Unsupervised MMDocIR Page & Layout Scoring"]
Key Designs¶
1. Null-Question Calibrated Scoring: resolving visual non-copying evidence and query-independent attention sinks
Addressing the reality that visual evidence cannot be detected via token-copy matching and that raw attention suffers from positional sink biases, the authors introduce a post-softmax attention scoring formulation calibrated against an uninformative null question. For each example \(x = (C, q, G_x)\) containing interleaved context \(C\), query \(q\), and annotated evidence token intervals \(G_x = \{g_i\}_{i=1}^{m_x}\) (encompassing text token spans or vision-encoder image tokens), the retrieval score for attention head \(h\) sums the post-softmax attention \(A^h_{t_q \to t}\) from all question tokens \(t_q\) to all tokens \(t\) within the evidence spans:
To eliminate background attention bias where certain tokens or structural positions draw disproportionate attention irrespective of question semantics, a parallel score \(S_h^{\text{null}}(x)\) is evaluated on \(x_{\emptyset} = (C, q_{\emptyset}, G_x)\) using a fixed null prompt \(q_{\emptyset}\) ("N/A"). The calibrated retrieval score is then computed as:
Heads are ranked by their calibrated excess question-induced attention, selecting the top 5% of heads. Stability tests confirm that this ranking is exceptionally invariant to alternative null-prompt wordings (\(\rho > 0.998\)).
2. Mechanistic Topology Characterization: uncovering extreme sparsity, pretraining intrinsicity, and dynamic modality division
Across six major vision-language foundation models (Qwen2.5-VL 7B/32B, Qwen3-VL 8B/32B, Gemma3 12B/27B) evaluated from 8K to 128K context lengths, the detection pipeline establishes three core mechanistic properties of MMRetHeads: - Sparsity: As few as 4.4%–10.2% of attention heads account for over 50% of the total positive calibrated retrieval score mass, indicating that evidence localization is concentrated in a tight functional circuit; - Intrinsicity: Comparing the base language model Qwen3-8B against its multimodal counterpart Qwen3-VL-8B shows that 46 of the top 58 text retrieval heads (79.3%) remain identical in layer and head index; similarly, 34 of 39 top heads (87.2%) are preserved between Gemma3-12B-PT and Gemma3-12B-IT. This proves that retrieval mechanisms are intrinsic circuits inherited directly from language pretraining rather than de novo constructs induced by vision-language fine-tuning; - Modality and Length Dynamics: Text and image retrieval share on average only 51% of their top-5% heads, reflecting a clear division of labor between general-purpose and modality-specific retrieval circuits. Strikingly, rendered-text retrieval (OCR images) overlaps with pure text retrieval by 79%, demonstrating that the routing circuit responds to the underlying nature of the retrieved content rather than the raw perceptual input channel. Furthermore, scaling from 8K to 128K triggers a pronounced upward layer shift for image retrieval heads (mean layer shifting from 33.1 to 52.4), showing that long-range visual retrieval demands higher-level semantic abstractions.
3. Two-Stage & Path-Specific Causal Masking: isolating causal necessity from correlational artifacts
To verify whether the identified heads actively govern retrieval rather than merely correlating with evidence locations, the authors implement comprehensive causal interventions. Applying zero-masking to the post-softmax attention weights of the top-5% retrieval heads simultaneously during prompt prefill and answer decoding produces dramatically steeper performance drops than decode-only masking (e.g., crashing MM-NIAH image retrieval accuracy from 42.8% to 0.0%), proving that evidence extraction takes place heavily during initial prompt processing. Rigorous controls—including random-head masking and layer-matched random masking—exhibit far milder degradation, ruling out general architectural damage. Furthermore, path-specific masking that exclusively severs attention from evidence tokens toward question and generation tokens (while leaving the heads' remaining internal computations intact) completely annihilates retrieval performance, providing causal proof that MMRetHeads act as dedicated information-routing pathways.
4. Zero-Shot Document Attention Re-Ranking: converting internal mechanistic signals into state-of-the-art retrievers
Moving beyond interpretability analysis, the authors directly transform the identified multimodal retrieval heads into an unsupervised re-ranking engine for visually rich document retrieval. Given a question \(q\), candidate document units \(D = \{d_i\}_{i=1}^n\) (such as full PDF pages or fine-grained bounding-box layout elements), and the selected head union \(H_{\text{sel}}\) across the four detection tasks, the relevance score is computed as:
Calibrated with the null-question baseline, this score directly ranks candidate pages and layout units without requiring contrastive pretraining, multi-vector late-interaction indexers, or domain fine-tuning, successfully leveraging the model's native attention mass for realistic information retrieval.
Benchmark & Evaluation Protocol¶
The empirical evaluation spans three distinct dimensions: 1. Controlled MM-NIAH Evaluation: Standardized 8K to 128K contexts with 6 needle-depth intervals across text retrieval, image retrieval, rendered-text retrieval, and identical-image verification; 2. Downstream Long-Context and Reasoning Benchmarks: 128K Long Document VQA (MMLongBench-Doc, SlideVQA) and multi-discipline expert reasoning benchmarks (MMMU, MMMU-Pro, MathVision, MathVista), evaluating direct-answer and Chain-of-Thought (CoT) prompting under head masking; 3. Multimodal Document Retrieval: The MMDocIR benchmark covering 10 real-world domains (academic papers, financial reports, technical manuals, government documents) evaluated at both Page-level and Layout-level against 13 leading document retrieval baselines.
Key Experimental Results¶
Main Results¶
| Task / Benchmark | Model / Method | Core Metric (Macro / Micro) | Strongest Baseline / Comparison | Gain / Delta |
|---|---|---|---|---|
| MMDocIR Page Retrieval | Qwen3-VL-8B MMRetHeads (Ours) | 64.7 / 64.5 (Recall@1) | 57.0 / 57.1 (Col-Phi3) | +7.7 / +7.4 pts |
| MMDocIR Page Retrieval | Qwen3-VL-8B All-Head Attention | 61.8 / 61.2 (Recall@1) | 57.0 / 57.1 (Col-Phi3) | +4.8 / +4.1 pts |
| MMDocIR Layout Retrieval | Qwen3-VL-8B MMRetHeads (Ours) | 39.0 / 39.3 (Recall@1) | 32.7 / 32.5 (ColPali) | +6.3 / +6.8 pts |
| MMDocIR Layout Retrieval | Qwen3-VL-8B All-Head Attention | 36.3 / 37.2 (Recall@1) | 32.7 / 32.5 (ColPali) | +3.6 / +4.7 pts |
| MMLongBench-Doc (128K VQA) | Qwen3-VL-8B Top-5% Head Masked | 5.7% (Accuracy) | 48.2% (No masking) | -42.5 pts (Collapse) |
| SlideVQA (128K VQA) | Qwen3-VL-8B Top-5% Head Masked | 8.9% (Accuracy) | 71.2% (No masking) | -62.3 pts (Collapse) |
Ablation Study¶
| Configuration / Intervention | Evaluation Setting & Metric | Observed Score | Analysis & Mechanistic Attribution |
|---|---|---|---|
| Baseline (No masking) | MM-NIAH Image Needle (Accuracy) | 55.00% | Unperturbed model baseline performance |
| Random Head Masking (58 heads) | MM-NIAH Image Needle (Accuracy) | 35.70% | Non-specific capacity reduction across layers |
| Layer-Matched Random Masking (58 heads) | MM-NIAH Image Needle (Accuracy) | 22.84% | Controls for depth distribution; retains substantial accuracy |
| Detected Retrieval Heads Masking (58 heads) | MM-NIAH Image Needle (Accuracy) | 0.00% | Complete catastrophic collapse of visual evidence retrieval |
| Path-Specific Masking (58 heads) | MM-NIAH Image Needle (Accuracy) | 0.00% | Selectively severing evidence-to-query paths destroys retrieval |
| Decode-Only Masking (Top-58 heads) | MM-NIAH Image Needle (Accuracy) | 42.8% | Masks generation only; prefill representations remain intact |
| Prefill + Decode Masking (Top-58 heads) | MM-NIAH Image Needle (Accuracy) | 0.0% | Proves evidence localization occurs primarily during prefill |
| Top-1% Head Masking (MMLongBench-Doc) | Accuracy / Repetitive Rate | 23.31% / 1.04% | Insufficient threshold; leaves redundant retrieval heads active |
| Top-5% Head Masking (MMLongBench-Doc) | Accuracy / Repetitive Rate | 9.99% / 6.35% | Optimal balance isolating the functional retrieval circuit |
| Top-10% Head Masking (MMLongBench-Doc) | Accuracy / Repetitive Rate | 3.33% / 28.41% | Over-ablates general linguistic heads, triggering degeneration |
Key Findings¶
- Sparse retrieval heads exert catastrophic causal control over reasoning: Masking just 58 retrieval heads (5% of the total) in Qwen3-VL-8B induces an immediate collapse on 128K long document understanding, dropping MMLongBench-Doc by 42.5 points and SlideVQA by 62.3 points. On multimodal reasoning benchmarks (MMMU, MMMU-Pro, MathVista, MathVision), Chain-of-Thought (CoT) accuracy plummets by an average of 38.8 points (compared to 24.3 points in direct answering), showing that multi-step deductive derivations depend critically on iterative visual re-grounding mediated by these heads.
- Characteristic failure modes reflect broken information channels: When MMRetHeads are ablated, models do not degrade into gibberish. Instead, they exhibit two dominant failure behaviors: (1) false abstention, declaring "insufficient information" despite the complete diagram or table being present in the prompt (e.g., in MMMU-Pro thermodynamics problems); and (2) ungrounded hallucination, inventing numbers and grid lines that contradict the visual prompt, proving that retrieval heads serve as the indispensable conduit binding visual perception to reasoning.
- Selected-head attention outperforms all-head averaging in retrieval: On MMDocIR, scoring candidates with selected retrieval heads beats aggregating all attention heads by 2.9 macro and 3.2 micro points on page retrieval, and 2.7 macro and 2.1 micro points on layout retrieval. Filtering out diffuse, task-agnostic attention noise is essential for high-fidelity relevance ranking.
Highlights & Insights¶
- Bridging the string-copying divide for multimodal interpretability: While text-only retrieval heads relied on verbatim token copying, this work establishes a robust framework using calibrated attention mass, providing a mathematically sound and empirically verifiable methodology for detecting retrieval circuits over pure images.
- Content-driven semantic routing overrides perceptual input channels: Finding that rendered-text retrieval shares 79% of heads with pure text retrieval (versus 51% with natural images) offers profound insight: transformer layers route information based on the semantic abstraction of the content (alphanumeric text) rather than the raw modality format (pixel patches).
- Zero-cost transition from mechanistic insight to deployment-grade retriever: The paper demonstrates that internal attention weights from native pre-trained vision-language models can be harvested directly as an unsupervised, state-of-the-art document re-ranker, outperforming specialized contrastive models like ColPali without any fine-tuning.
Limitations & Future Work¶
- Forward-pass computational latency in large-scale search: Generating relevance scores via full LVLM forward passes requires substantial GPU memory and compute, making MMRetHeads impractical for initial first-stage retrieval over millions of candidate documents compared to lightweight dense embeddings or late-interaction indexers.
- Architectural and context-length frontiers: The analysis is focused on decoder-only LVLMs with dynamic vision tokenization up to 128K context. Extending this analysis to cross-attention architectures, native encoder-decoder models, and million-token video benchmarks remains an open challenge.
Related Work & Insights¶
- vs. Text-Only Retrieval Heads (Wu et al., 2025b; Zhang et al., 2025): Prior methods identify retrieval heads via token copy-paste matching or query-directed text attention, which fail on visual inputs; this paper generalizes retrieval head detection to multimodal evidence via post-softmax attention mass and null-question calibration.
- vs. Vision-Language Retrieval Models (ColPali, Col-Phi3, DSE): Dedicated visual document retrievers require extensive contrastive pre-training on document screenshot datasets; this work proves that off-the-shelf, general-purpose LVLMs already possess intrinsic retrieval circuits that surpass specialized models when properly isolated.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Extends retrieval head theory to multimodal inputs via an elegant calibrated attention mass framework]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Spans 6 models, 8K–128K lengths, causal path interventions, and real-world MMDocIR retrieval]
- Writing Quality: ⭐⭐⭐⭐⭐ [Exemplary clarity, compelling narrative structure, and rigorous causal evidence presentation]
- Value: ⭐⭐⭐⭐⭐ [Provides deep mechanistic insights into long-context LVLMs while delivering an effective zero-shot re-ranking tool]