Skip to content

Multiple Images Distract Large Multimodal Models via Attention Fragmentation

Conference: ECCV 2026
Paper: ECCV Official
Area: Interpretability
Keywords: Multimodal Large Language Models, Multi-image Understanding, Attention Fragmentation, Attention Sinks, Attention Remasking

TL;DR

This paper diagnoses the mechanism behind open-source LMM performance drops and recency bias across multi-image tasks as "attention fragmentation"—caused by recurrent background attention sinks and causal masking—and proposes Attention Remasking (AR), a training-free edit that suppresses visual sinks and dynamically routes attention to task-relevant forward links using an uncorrupted first-layer grounding prior.

Background & Motivation

Multi-image understanding requires Large Multimodal Models (LMMs) to perform holistic reasoning across visual contexts, such as multi-image comparison, cross-image retrieval, temporal progression tracking, and visual evidence synthesis. However, open-source LMMs exhibit an acute performance gap compared to proprietary commercial models on multi-image benchmarks. For instance, on the comprehensive MMIU benchmark, while leading commercial systems like Gemini-3-Pro reach 68.1% accuracy, top open-source models consistently struggle below 56%, demonstrating marked instability when required to integrate visual evidence distributed across images.

By dissecting the internal attention patterns of decoder-based LMMs operating under standard causal masking, the authors uncover a fundamental mechanistic bottleneck termed attention fragmentation. First, owing to the sum-to-one constraint of softmax normalization, Transformer representations naturally repurpose semantically empty background visual patches as attention sinks to store excess activation mass; in multi-image sequences, these sink tokens repeatedly emerge at identical spatial and background locations across distinct images. Second, autoregressive causal masking creates severe structural asymmetry across images: earlier images are exposed to substantially more downstream queries, accumulating disproportionately heavy sink attention shares. Supported by Shannon entropy measurements and bounded via Pinsker's inequality, the paper mathematically proves that this high-entropy dispersion across images, combined with stronger early-image sinks, strictly contracts the usable non-sink attention budget available to early images, thereby providing a formal mechanistic explanation for the long-standing recency bias and image-order sensitivity in LMMs.

Existing post-hoc remedies fall short of addressing this compounding failure. Proportional post-softmax reallocation techniques like VAR cannot bridge causal masks and directly inherit both dispersion and positional skew, while uniform causal relaxation methods like SoFA introduce unselective background noise by blindly opening bidirectional attention across all tokens. The authors' angle of attack stems from a key mechanistic insight: shallow transformer layers inherently exhibit high-fidelity, uncorrupted text-to-vision grounding prior to deep-layer representational drift and sink accumulation. Core idea: propose Attention Remasking (AR), a training-free pre-softmax edit that blocks background attention sinks and leverages the model's native first-layer text-visual grounding prior to dynamically unmask sparse, task-relevant cross-image links, channeling recovered attention directly into decisive evidence.

Method

Overall Architecture

Attention Remasking (AR) is a lightweight, training-free inference-time intervention designed for interleaved multi-image LMMs. It pursues two complementary objectives: (1) suppressing deep-layer visual attention sinks that absorb excess attention without contributing semantic value, and (2) counteracting attention fragmentation and causal isolation by dynamically establishing forward visual links conditioned on instruction relevance.

The method operates strictly on the visual-to-visual submatrix of the pre-softmax attention score matrix in deeper layers (\(\ell > 1\)), leaving text autoregression under standard causal masking untouched. End to end, AR executes three sequential stages: identification and column-wise blocking of visual sink tokens, extraction of an uncorrupted semantic grounding prior at Layer 1 to build a sparse candidate set, and closed-form redistribution of freed attention mass into forward cross-image links.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-image & Text Input Sequence"] --> B["Visual Attention Sink Blocking<br/>Column-wise mask on background sink tokens"]
    B --> C["Shallow Grounding-guided Sparse Unmasking<br/>Layer-1 text-to-vision attention prior"]
    C --> D["Freed Attention Mass Dynamic Routing<br/>Inject reclaimed sink mass into forward targets"]
    D --> E["Debiased Cross-image Reasoning Output"]

Key Designs

1. Visual Attention Sink Blocking: Column-wise mask on background sink tokens

In multi-image inputs, background tokens in each image repeatedly act as attention sinks due to softmax normalization constraints, soaking up large attention fractions despite near-zero semantic utility. To systematically detect and neutralize these non-informative anchors, AR follows activation-profile statistics on a calibration corpus to score the sink propensity \(\phi(x)\) of token state \(x\) across known sink dimensions \(\mathcal{D}_{\mathrm{sink}}\):

\[\phi(x) = \max_{d \in \mathcal{D}_{\mathrm{sink}}} \frac{x[d] - \mu_d}{\sigma_d}\]

A token \(j\) at layer \(\ell\) is flagged as a visual sink if its preceding layer hidden state satisfies \(\phi(x_j^{\ell-1}) \ge \tau\), forming the sink set \(\mathcal{S}^\ell = \bigcup_{m=1}^M \mathcal{S}_m^\ell\). For all subsequent layers \(\ell > 1\), AR enforces a column-wise block directly on the pre-softmax attention scores: \(\tilde{Z}_{i, j}^\ell = -\infty \quad (\forall i, \forall j \in \mathcal{S}^\ell)\). This completely eliminates incoming attention to sink keys; empirical answer-flip analysis under binomial Wilson bounds confirms that removing sink keys incurs near-zero prediction change, establishing that they serve purely as structural overflow buffers.

2. Shallow Grounding-guided Sparse Unmasking: Layer-1 text-to-vision attention prior

Causal masking prevents early image tokens from attending forward to subsequent images, creating an artificial barrier that severely hinders cross-image comparative reasoning. Rather than indiscriminately opening all forward links, AR exploits the empirical fact that LMMs exhibit clean, natural alignment between text instructions and visual patches in their initial decoder layer (\(\ell = 1\)), well before deep-layer sink fragmentation takes hold. The global task relevance of each non-sink visual token \(j \in \mathcal{V}_{\mathrm{valid}} = \mathcal{V} \setminus \mathcal{S}^1\) is derived from instruction tokens \(T\) as \(P_{\mathrm{global}}(j) = \frac{1}{|T|} \sum_{t \in T} \alpha_{t, j}^1\). By computing the mean relevance \(\mu_{\mathrm{valid}}\) across valid visual tokens, AR isolates a sparse, task-relevant candidate set:

\[\mathcal{U} = \left\{ j \in \mathcal{V}_{\mathrm{valid}} \mid P_{\mathrm{global}}(j) > \mu_{\mathrm{valid}} \right\}\]

For any visual query token \(i\) situated in image \(m\), AR selectively unmasks forward links only to the subset \(\mathcal{U}_i \subset \mathcal{U}\) residing in subsequent images (\(m' > m\)), defining an instruction-conditioned routing prior distribution \(\pi_{i, j} = \frac{P_{\mathrm{global}}(j)}{\sum_{k \in \mathcal{U}_i} P_{\mathrm{global}}(k)}\). This sparse forward connectivity bypasses causal blind spots without drowning representations in background clutter.

3. Freed Attention Mass Dynamic Routing: Inject reclaimed sink mass into forward targets

Simply masking sinks and falling back to naive softmax normalization redistributes the trapped sink budget proportionally across remaining keys, preserving both high entropy and the original causal skew. AR instead implements an explicit probability redistribution rule. For query \(i\) at layer \(\ell\), the trapped sink attention mass \(\eta_i^\ell = \sum_{j \in \mathcal{S}^\ell} \alpha_{i, j}^\ell\) is redirected specifically to the task-relevant candidate set \(\mathcal{U}_i\). The target row attention distribution is formulated as:

\[\hat{\alpha}_{i, j}^\ell = \left(1 - \eta_i^\ell\right) \frac{\alpha_{i, j}^\ell}{\sum_{k \notin \mathcal{S}^\ell \cup \mathcal{U}_i} \alpha_{i, k}^\ell} \mathbf{1}\left[j \notin \mathcal{S}^\ell \cup \mathcal{U}_i\right] + \eta_i^\ell \, \pi_{i, j} \, \mathbf{1}\left[j \in \mathcal{U}_i\right]\]

This redistribution is mapped to pre-softmax scores via \(\tilde{Z}_{i, j}^\ell = \log \hat{\alpha}_{i, j}^\ell + c_i^\ell\), ensuring full numerical consistency through softmax. Crucially, attention map extraction is restricted strictly to Layer 1, allowing all deeper layers (\(\ell > 1\)) to leverage PyTorch's scaled_dot_product_attention (SDPA) with Memory-Efficient attention backends. This avoids materializing full dense attention matrices and preserves high execution throughput.

Key Experimental Results

Main Results

The authors evaluate AR across five prominent multi-image benchmarks: MMIU (7 relationship types), MuirBench (12 multi-image tasks), MIRB (multi-hop and perception tasks), LIBench (LLaVA-Interleave Bench), and MIBench. Comparisons include the unedited baseline, SoFA (causal/bidirectional interpolation), and VAR (post-softmax sink redistribution).

Method & Backbone MMIU (Acc %) MuirBench (Acc %) MIRB (Acc %) LIBench (Acc %) MIBench (Acc %)
Commercial API
Gemini-3-Pro 68.1 65.7 72.5 70.3 69.9
GPT-5 65.3 65.9 69.6 70.2 70.0
LLaVA-OneVision-7B 36.9 35.0 47.9 53.5 53.8
+ VAR 38.0 35.5 48.7 54.2 55.0
+ SoFA 38.3 36.7 48.3 54.5 54.1
+ AR (Ours) 41.2 39.6 51.9 57.7 58.6
Qwen3-VL-8B 55.6 51.3 65.6 61.2 63.5
+ VAR 55.9 51.4 66.0 61.7 63.5
+ SoFA 56.7 52.3 66.2 61.6 64.8
+ AR (Ours) 59.4 54.9 69.2 65.5 67.6
InternVL3.5-8B 49.3 39.7 53.6 57.6 51.5
+ VAR 50.6 40.9 54.6 59.1 52.4
+ SoFA 50.5 41.3 54.9 59.3 52.7
+ AR (Ours) 54.1 43.8 57.9 61.9 55.6
DeepSeek-VL2-Small 51.4 40.5 45.8 58.3 53.7
+ VAR 52.1 41.1 46.6 59.4 54.6
+ SoFA 52.3 41.1 46.5 60.1 55.0
+ AR (Ours) 56.1 44.7 50.7 62.5 58.2

Ablation Study

1. Layer Selection for the Grounding Prior Ablation evaluating performance across benchmarks when extracting the guidance mask from Layer 1 (Ours), mid layer \(L/2\), and final layer \(L\):

Model Source Layer MMIU (Acc %) MuirBench (Acc %) MIRB (Acc %) LIBench (Acc %) MIBench (Acc %)
Qwen3-VL-8B – (Unedited) 55.6 51.3 65.6 61.2 63.5
+ AR Layer \(L\) (Last) 53.2 49.8 63.1 59.5 61.0
+ AR Layer \(L/2\) (Mid) 54.8 50.9 64.8 60.6 62.4
+ AR (Ours) Layer 1 (First) 59.4 54.9 69.2 65.5 67.6
InternVL3.5-8B – (Unedited) 49.3 39.7 53.6 57.6 51.5
+ AR Layer \(L\) (Last) 46.5 37.4 51.0 55.2 48.9
+ AR Layer \(L/2\) (Mid) 48.7 39.1 52.8 56.9 50.8
+ AR (Ours) Layer 1 (First) 54.1 43.8 57.9 61.9 55.6

2. Computational Overhead (NVIDIA A100 80GB, InternVL3.5-8B backbone)

Method Setting Latency mean / p90 (ms) Throughput (samples/s) Peak Memory (GB)
InternVL3.5-8B (Base) 280 / 330 3.6 15.2
+ SoFA 284 / 334 (+1.4%) 3.5 (-2.8%) 15.2 (+0.0%)
+ VAR 287 / 338 (+2.5%) 3.5 (-2.8%) 15.3 (+0.7%)
+ AR (Ours) 288 / 339 (+2.9%) 3.5 (-2.8%) 15.3 (+0.7%)

Key Findings

  • Layer 1 grounding is a structural requirement, not an empirical heuristic: Extracting routing priors from deep layers strictly degrades performance below baseline (e.g., dropping Qwen3-VL-8B MMIU to 53.2%). Because deep layers are dominated by background sinks, thresholding deep attention produces an inverted mask that reinforces noise.
  • Mutual necessity of sink masking and unmasking: Neither Mask-only nor Unmask-only variants match full AR. Mask-only merely diffuses sink mass across existing fragmented paths, while Unmask-only leaves sink drain intact.
  • Orthogonal compatibility with Delimiter Token Scaling (DTS): While DTS scales image delimiter representations to curb inter-image feature leakage, AR explicitly manipulates visual patch attention routing. Combining both yields compounding gains across MMIU and MuirBench.
  • Mitigating recency bias: Across image-order permutation tests, baseline models display a steep positive accuracy slope favoring later images; AR flattens this trajectory and substantially lowers answer-flip rates, demonstrating robust order-invariance.

Highlights & Insights

  • Formulates multi-image attention fragmentation mechanistically: Using Pinsker's inequality, the authors prove that high normalized entropy over images, paired with causal accumulation of background sinks in earlier images, mathematically strangles the usable non-sink attention budget of earlier inputs.
  • Lightweight, training-free intervention with negligible overhead: By materializing full probabilities only at Layer 1 and executing sparse custom masking via PyTorch SDPA memory-efficient kernels in deeper layers, AR incurs only +2.9% latency overhead while improving accuracy by up to 4.8 percentage points.
  • Elucidates why Gated Attention fails to eliminate multimodal visual sinks: Although text attention sinks are mitigated in recent LLMs via Gated Attention, visual sinks persist due to ViT background register phenomena and the absence of direct next-token prediction gradients on visual tokens.

Limitations & Future Work

  • Assumed validity of early semantic grounding: AR hinges on the observation that Layer 1 provides reliable instruction-vision grounding. For highly intricate visual queries requiring multi-step compositional reasoning before relevant regions can be recognized, early-layer attention may fail to capture all necessary patches.
  • Extreme sequence and image counts: In long-video or massively multi-image regimes (>50 images), simple mean thresholding across the visual candidate pool could experience distribution shifts, warranting hierarchical or cluster-based unmasking.
  • vs VAR (Visual Attention Redistribution): VAR operates entirely post-softmax within the confines of the existing causal mask, proportionally redistributing sink mass to local visual non-sink tokens. Consequently, it cannot build forward cross-image bridges and preserves positional skew. AR operates pre-softmax, selectively punches holes through the causal mask, and routes attention to task-relevant patches.
  • vs SoFA (Soft Attention): SoFA applies continuous linear interpolation between causal and fully bidirectional attention without discrimination. This floods early queries with irrelevant visual background tokens from later images. AR instead selectively unmasks only instruction-relevant tokens filtered via the Layer 1 grounding prior.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Rigorous theoretical and empirical identification of multi-image attention fragmentation with an elegant, training-free self-guided remasking solution.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 5 multi-image benchmarks, multiple parameter scales (7B to 32B), deep ablations on layer selection, and rigorous latency profiling.
  • Writing Quality: ⭐⭐⭐⭐⭐ Cohesive narrative progression from mechanistic mathematical formulation to efficient system realization.
  • Value: ⭐⭐⭐⭐⭐ Plug-and-play inference edit that substantially closes the multi-image reasoning gap between open-source models and commercial proprietary systems.