Skip to content

RAVE: Re-Allocating Visual Attention in Large Multimodal Models

Conference: EMNLP 2026
Paper: ACL Anthology
Area: Multimodal VLM
Keywords: Large Multimodal Models, Visual Attention Allocation, Cross-modal Attention Dilution, Pair-Gating Mechanism, Pre-ROPE Representations

TL;DR

RAVE addresses the pervasive bottlenecks of cross-modal visual attention dilution and intra-visual sink imbalances in decoder-only large multimodal models (LMMs) by introducing a minimalist, architecture-preserving pair-gating mechanism. With merely two layer-shared linear vectors operating on pre-ROPE query and key representations, RAVE injects a learned dynamic bias into pre-softmax logits over visual keys, achieving an average gain of 3 percentage points across diverse multimodal benchmarks with negligible parameter overhead and striking improvements on fine-grained perception-intensive tasks.

Background & Motivation

Modern large multimodal models (LMMs), prominently represented by the LLaVA family and the Qwen-VL series, predominantly employ a decoder-only architecture where visual features are projected into language embedding space and processed alongside textual tokens in a unified sequence. Under this formulation, the model relies entirely on the self-attention mechanism inherited from its pretrained language backbone. However, this inherited standard attention is modality-agnostic and lacks an explicit inductive bias or regulation mechanism for cross-modal interaction. Consequently, treating textual and visual tokens uniformly leads to severe visual attention misallocation, critically weakening the model's visual grounding capability during generation.

This misallocation manifests acutely along two fundamental axes: cross-modal imbalance and intra-visual distortion. At the cross-modal level, visual tokens as a collective entity are systematically under-attended relative to textual tokens. Across decoder depth, the attention mass allocated to visual tokens rapidly decays in deeper layers. More detrimentally, during autoregressive decoding, as historical answer tokens accumulate, the attention assigned to the visual block steadily diminishesโ€”often plunging toward near-zeroโ€”while attention becomes overwhelmingly concentrated on prior generated text, leading to severe hallucination snowballing. Concurrently, at the intra-visual level, attention within the image block is highly distorted: uninformative background or corner patches absorb disproportionate attention mass to form "visual sinks" or attention collapse, while standard 1D Rotary Position Embedding (RoPE) introduces distance-dependent attenuation and spatial bias that favor adjacent sequential patches over genuine semantic relevance.

Existing approaches to mitigate these problems face notable limitations. Training-free heuristic intervention methods (e.g., ATTNREAL, VAR, FARSIGHT) rely on hand-crafted rules that lack input-adaptive contextual sensitivity; architectural redesigns (e.g., D-ATTN, LLAVIT) decompose attention into fragmented modality paths, disrupting standard Transformer structures and complicating deployment; and visual revisiting methods (e.g., TVC, LOOK-BACK) demand auxiliary data curation pipelines, specialized tokens, or multi-stage reinforcement learning. The central challenge lies in developing an adaptive, content-driven mechanism that rectifies visual attention allocation within the native attention operator without altering the backbone architecture or breaking standard end-to-end training. Core idea: Introduce RAVE (Re-Allocating Visual Attention), a lightweight, architecture-preserving pair-gating mechanism that computes a content-aware bias from pre-ROPE query and key features and injects it additively into pre-softmax logits over visual keys, effectively eradicating visual attention dilution while preserving language reasoning priors.

Method

Overall Architecture

The objective of RAVE is to provide a drop-in correction to standard self-attention in decoder-only LMMs that directly recalibrates the competition for attention mass. The input sequence unifies system prompt tokens (\(I_{sys}\)), projected visual tokens (\(I_{img}\)), user instruction tokens (\(I_{que}\)), and autoregressively generated answer tokens (\(I_{ans}\)). Within each Transformer layer \(\ell\) and for any designated query head \(h\), pre-rotary query and key representations \(\bar{Q}_{\ell, h}, \bar{K}_{\ell, h} \in \mathbb{R}^{N \times d_k}\) and value representations \(V_{\ell, h}\) are computed. The standard attention path applies RoPE to obtain rotated features \(Q_{\ell, h}, K_{\ell, h}\) and evaluates standard dot-product attention logits \(L_{\ell, h}\). Concurrently, the RAVE auxiliary branch projects the pre-ROPE vectors using layer-shared scalar weights into 1D score vectors, computes an outer product to form a full pairwise interaction matrix, and maps it through a smooth \(\tanh\) nonlinearity into a gating bias \(G_{\ell, h}\). This bias is additively injected into the pre-softmax logits exclusively for visual key indices \(j \in I_{img}\). After standard causal masking and Softmax normalization, the calibrated attention weights aggregate values \(V_{\ell, h}\).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Input Sequence<br/>Text Tokens + Visual Tokens"] --> B["Pre-ROPE Feature Projections<br/>Extract unrotated query & key states"]
    B --> C["Pre-ROPE Pair-Gating Bias<br/>Layer-shared vectors compute outer-product gate"]
    B --> D["Standard RoPE & Dot-Product Logits<br/>Compute native scaled dot-product scores"]
    C --> E["Visual-Key Logits Recalibration<br/>Inject additive bias strictly to visual keys"]
    D --> E
    E --> F["GQA-Aware Partial-Head Gating<br/>Apply gate to p=0.25 heads to preserve priors"]
    F --> G["Full-Lifecycle Attention Aggregation<br/>Sustain visual evidence in Prefill & Decoding"]

Key Designs

1. Pre-ROPE Pair-Gating Bias: Decoupling positional attenuation to achieve content-driven dynamic scoring

Rotary Position Embedding (RoPE) inherently injects a distance-dependent attenuation into query-key dot products as the 1D sequence distance between tokens widens. In multimodal dialogue and long-chain reasoning, this mathematical property severely penalizes visual tokens located far earlier in the sequence relative to currently generated answer tokens, inevitably accelerating visual attention dilution. To overcome this limitation, RAVE deliberately computes its gating signal from the unrotated pre-ROPE representations \(\bar{Q}_{\ell, h}, \bar{K}_{\ell, h} \in \mathbb{R}^{N \times d_k}\). This decouples the recalibration signal from monotonic positional decay, ensuring that the gate modulates attention purely based on multimodal semantic context. To keep parameter overhead completely negligible, RAVE introduces only a single pair of learnable linear projection vectors per decoder layer \(\ell\):

\[W_{gq, \ell}, W_{gk, \ell} \in \mathbb{R}^{d_k \times 1}\]

These vectors are shared across all \(H\) query heads within that layer. For each query head \(h\), the scalar token scores are obtained by:

\[s_{\ell, h}^q = \bar{Q}_{\ell, h} W_{gq, \ell} \in \mathbb{R}^N, \quad s_{\ell, h}^k = \bar{K}_{\ell, h} W_{gk, \ell} \in \mathbb{R}^N\]

The pairwise gating bias is then constructed via their outer product and normalized via a hyperbolic tangent activation:

\[G_{\ell, h, ij} = \tanh\left(s_{\ell, h, i}^q \cdot s_{\ell, h, j}^k\right)\]

This pairwise formulation enables fine-grained query-key contextual interaction. Because the projection vectors are shared across all query heads in each layer, RAVE adds merely \(2 L d_k\) parameters in total across the entire networkโ€”less than a few tens of thousands of parameters for a 7B modelโ€”preserving the original backbone size almost exactly.

2. Visual-Key Logits Recalibration: Pre-softmax intervention governing cross-modal competition and intra-visual rebalancing

Conventional gating methods (such as GATED-ATTN) typically apply gating to the attention output hidden states. However, output-side modulation only adjusts the scaling of already-pooled representations and cannot intervene in the underlying competition for attention probability mass among diverse token modalities. RAVE repositions the intervention into the pre-softmax logits, selectively modifying the attention energy over visual key positions \(j \in I_{img}\):

\[\widetilde{L}_{\ell, h, ij} = L_{\ell, h, ij} + G_{\ell, h, ij} \cdot \mathbf{1}[j \in I_{img}]\]
\[\widetilde{A}_{\ell, h, ij} = \text{softmax}_j\left(\widetilde{L}_{\ell, h, ij} + M_{ij}\right)\]

where \(M\) denotes the standard causal attention mask. This formulation achieves two critical objectives simultaneously: first, by directly shifting visual-key logits before Softmax normalization, it allows visual tokens to compete more vigorously against historical textual tokens, preventing the visual block from being drowned out as generation progresses; second, because \(G_{\ell, h, ij}\) produces token-specific continuous adjustments across visual keys \(j\), it adaptively penalizes uninformative background patches ("visual sinks") while boosting semantically salient object regions. Crucially, non-visual keys (\(j \notin I_{img}\)) remain completely untouched, ensuring that textual self-attention dynamics and linguistic reasoning pathways are shielded from unnecessary interference. Initializing \(W_{gq, \ell}\) and \(W_{gk, \ell}\) near zero ensures \(G_{\ell, h, ij} \approx 0\) at initialization, guaranteeing that the model smoothly warms up from the pretrained checkpoint.

3. GQA-Aware Partial-Head Gating: Balancing visual evidence recalibration with pretrained language reasoning priors

Grouped-Query Attention (GQA) is ubiquitous in modern large models, where \(H_q\) query heads are organized into groups sharing \(H_{kv}\) key/value heads (\(r = H_q / H_{kv}\)). Forcibly overriding all query heads with visual recalibration risks compromising the language backbone's finely tuned linguistic and logical deduction capabilities. RAVE addresses this by implementing a GQA-aware partial-head intervention: within each GQA group, only a fraction \(p\) of the query heads are designated as gated heads (\(\mathcal{H}_{rave}\), with \(|\mathcal{H}_{rave}| = p H_q\)), while the remaining \((1-p)\) fraction of query heads execute standard vanilla attention:

\[\widehat{L}_{\ell, h, ij} = \begin{cases} \widetilde{L}_{\ell, h, ij}, & h \in \mathcal{H}_{rave} \\ L_{\ell, h, ij}, & h \notin \mathcal{H}_{rave} \end{cases}\]

Empirical analyses establish \(p = 0.25\) as the optimal Pareto operating point. Under this configuration, ungated heads maintain unimpeded linguistic flow and autoregressive textual continuity, while the gated heads act as dedicated sensory conduits that ground the generation process firmly in visual evidence.

4. Full-Lifecycle Attention Aggregation: Prefill-and-decoding joint modulation for stable visual conditioning

Visual attention degradation is not merely an artifact of long autoregressive generation; it also manifests during prompt prefilling when question tokens initially attend to image features. Many post-hoc intervention techniques only act during decoding, leaving the initial question-image multimodal representation suboptimal. RAVE operates seamlessly across both the prefill and decoding phases. During prefill, question tokens leverage RAVE to selectively emphasize informative visual regions, establishing a refined multimodal context. During decoding, newly generated tokens continuously utilize RAVE to overcome temporal attenuation and maintain persistent grounding. Ablation studies confirm that deploying RAVE across both prefill and decoding yields substantial performance gains over decoding-only intervention, validating the premise that early multimodal feature conditioning is essential for accurate downstream reasoning.

Loss & Training

RAVE requires no auxiliary loss functions, contrastive objectives, or gating-specific regularizers. The gate parameters \(W_{gq, \ell}, W_{gk, \ell}\) are trained strictly end-to-end alongside the vision projector and LLM backbone under the standard autoregressive next-token cross-entropy objective:

\[\mathcal{L} = - \sum_{t \in I_{ans}} \log P\left(x_t \mid x_{<t}\right)\]

In our controlled experimental setup based on Mistral-v0.3-7B, the training strictly follows the three-stage recipe of D-ATTN: Stage 1 aligns the visual projector on the 558K LLaVA LAION/CC/SBU subset; Stage 2 fine-tunes the model on 1.25M ShareGPT4V-PT dense captions; Stage 3 performs visual instruction tuning on a mixture of 665K LLaVA-1.5 instruction samples and 102K ShareGPT4V samples. Because the gating architecture is extraordinarily lightweight, it adds less than 10% wall-clock training overhead.

Key Experimental Results

Main Results

Under the controlled Mistral-v0.3-7B configuration, RAVE is compared against standard causal attention, VAR (post-hoc attention sink redistribution), D-ATTN (modality-decomposed attention), and GATED-ATTN (output-side attention gating) across nine diverse multimodal benchmarks in VLMEvalKit:

Attention Method Category MTVQA ChartQA DocVQA OCRBench InfoVQA TextVQA MathVista MMMU MMMU-Pro Perception Avg. Reasoning Avg. Overall Avg.
Standard Attention Baseline 6.51 23.20 35.85 40.60 24.86 62.17 24.60 30.56 20.00 32.20 25.05 29.82
VAR Inference-only 6.46 22.60 35.56 40.10 24.29 62.39 24.50 31.00 19.03 31.90 24.84 29.55
D-ATTN Arch. Redesign 6.23 23.92 35.14 39.40 26.21 62.18 27.90 39.33 20.29 32.18 29.17 31.18
GATED-ATTN Output-Gating 6.05 20.92 33.61 37.70 24.22 60.54 24.80 31.16 15.83 30.51 23.93 28.31
RAVE (Ours) Pre-Softmax Gate 9.50 26.28 43.55 44.20 28.21 65.90 25.40 32.22 20.16 36.26 25.93 32.81

Note: Perception-intensive benchmarks include MTVQA, ChartQA, DocVQA, OCRBench, InfoVQA, and TextVQA; Reasoning-oriented benchmarks include MathVista, MMMU, and MMMU-Pro-10c.

To directly evaluate the cross-modal attention mass redistribution, the table below shows the segment-wise attention mass percentage (%) assigned by generated answer tokens to the image and answer segments at the first and final decoding positions (averaged over 20 representative probes per task group):

Task Group Decoding Position Image Mass % (Std.) Image Mass % (RAVE) Gain (\(\Delta\)) Answer Mass % (Std.) Answer Mass % (RAVE)
Perception-intensive First Position 24.8% 39.6% +14.8% 4.8% 4.1%
Perception-intensive Final Position 20.9% 34.7% +13.8% 16.0% 15.5%
Reasoning / Knowledge First Position 12.8% 26.1% +13.3% 5.0% 4.3%
Reasoning / Knowledge Final Position 12.3% 25.3% +13.0% 7.2% 6.1%

Furthermore, to test backbone generalization beyond the LLaVA framework, RAVE was applied to Qwen2.5-VL-7B-Instruct via continued supervised fine-tuning. RAVE consistently outperformed the continued-SFT Standard Attention baseline across all nine benchmarks, improving Perception Avg. from 72.34% to 73.49% (ChartQA: 80.04% \(\to\) 82.44%; OCRBench: 82.50% \(\to\) 84.40%) and Overall Avg. from 63.95% to 64.97%, proving its robust adaptability to state-of-the-art commercial multimodal models.

Ablation Study

The systematic ablation across design dimensions on Mistral-v0.3-7B demonstrates the criticality of RAVE's architectural choices:

Variant Gating Location Gating Form Head Ratio \(p\) Application Stage DocVQA OCRBench TextVQA MMMU Overall Avg. Key Implication
(1) Main RAVE Pre-Softmax Additive 0.25 Prefill + Dec. 43.55 44.20 65.90 32.22 32.81 Optimal default configuration
(2) Gating Location Post-Softmax Additive 0.25 Prefill + Dec. 44.68 43.20 65.61 32.44 32.50 Fails to modulate pre-softmax token competition
(3) Gating Form Pre-Softmax Multiplicative 0.25 Prefill + Dec. 44.01 42.40 65.74 31.67 32.21 Multiplicative scaling destabilizes negative logits
(4) Head Ratio Pre-Softmax Additive 0.50 Prefill + Dec. 42.99 41.30 65.25 32.33 32.14 Excessive gating degrades text-side attention
(5) Head Ratio Pre-Softmax Additive 0.75 Prefill + Dec. 45.21 43.60 67.13 29.00 32.46 Strong OCR gains, but MMMU reasoning drops to 29.00
(6) Head Ratio Pre-Softmax Additive 1.00 Prefill + Dec. 44.42 44.40 65.97 29.78 32.13 Severe impairment of linguistic coherence
(7) Stage Variant Pre-Softmax Additive 0.25 Decoding Only 43.36 42.70 65.28 30.78 31.94 Omitting prefill visual alignment drops ~0.9 points

Key Findings

  • Enormous boosts on perception-intensive benchmarks: RAVE produces dramatic improvements on tasks requiring high-resolution grounding and text parsing, surging on DocVQA from 35.85 to 43.55 (+7.70 points), on MTVQA from 6.51 to 9.50 (+2.99 points), and on ChartQA from 23.20 to 26.28 (+3.08 points). This empirically proves that mitigating visual attention dilution directly revitalizes fine-grained perception.
  • Intervention locus is decisive: Despite introducing learned parameters, GATED-ATTN (which gates attention outputs) achieves an Overall Avg. of only 28.31, underperforming even the unaugmented Standard Attention baseline (29.82). In contrast, RAVE achieves 32.81 by modulating pre-softmax logits. This underscores that directly reshaping the competitive allocation of attention mass is fundamentally superior to post-hoc feature reweighting.
  • Controlled head intervention preserves language reasoning: Expanding the gated head ratio \(p\) from 0.25 to 1.00 severely harms multimodal reasoning, causing MMMU to drop from 32.22 down to 29.78. Maintaining 75% of query heads in their vanilla state provides sufficient capacity for the pretrained language prior to function without degradation.
  • Dramatic mitigation of late-stage visual dilution: Attention mass measurements reveal that by the final decoding position, RAVE maintains 34.7% image attention mass on perception tasks (compared to 20.9% for Standard Attention) and 25.3% on reasoning tasks (compared to 12.3%), effectively doubling visual evidence retention at the tail end of long generation sequences.

Highlights & Insights

  • Decoupling RoPE positional attenuation via Pre-ROPE features: Standard RoPE penalizes distant tokens across 1D sequence lengths, which inherently disadvantages visual tokens during prolonged decoding. RAVE's derivation of query-key gating biases from unrotated pre-ROPE features represents an insightful circumvention of positional bias, anchoring attention reallocation strictly in semantic relevance.
  • Dual rebalancing via a minimalist outer-product bias: By introducing a single pair of layer-shared projection vectors per layer, RAVE simultaneously resolves cross-modal competition (enabling visual keys to outcompete historical text keys) and intra-visual distortion (suppressing uninformative visual sinks) through a unified additive scalar interaction.
  • Extreme parameter efficiency and seamless drop-in integration: Adding merely \(2 L d_k\) parameters and training stably under standard cross-entropy loss without auxiliary rewards or architectural restructuring makes RAVE exceptionally attractive for practical LMM deployment.

Limitations & Future Work

  • Lack of fused FlashAttention kernel optimization: The current implementation relies on PyTorch's native SDPA and explicitly materializes the intermediate \(N \times N\) pairwise gate tensor, resulting in approximately 17% inference latency overhead. Implementing a custom fused FlashAttention/FlashDecoding CUDA kernel that computes the outer-product bias on-chip in SRAM will be vital for zero-overhead deployment.
  • Exploration in full pretraining from scratch: While verified on continued SFT and standard instruction tuning, RAVE has not yet been incorporated into large-scale multimodal pretraining from scratch, where deeper synergistic co-adaptation between visual encoders and gated attention heads could unlock even greater performance.
  • Bounds on hallucination mitigation: While RAVE consistently ensures strong visual evidence retention, enhanced visual attention alone does not guarantee the complete elimination of hallucinations in complex logical deduction or high-stakes domain-specific queries.
  • vs VAR / ATTNREAL / FARSIGHT (Training-free Inference Interventions): VAR and ATTNREAL rely on static statistical heuristics to prune or redistribute attention from sink tokens during inference, lacking context-dependent adaptability. RAVE learns a parametric, input-adaptive recalibration module end-to-end, achieving substantially superior benchmark scores (32.81 vs 29.55).
  • vs D-ATTN (Modality-Decomposed Attention): D-ATTN splits self-attention into separate visual-to-visual, text-to-visual, and text-to-text matrices, incurring structural fragmentation and engineering complexity. RAVE retains the unified causal attention operator, outperforming D-ATTN by a wide margin on perception-intensive tasks (36.26 vs 32.18).
  • vs GATED-ATTN (Output-Side Attention Gating): GATED-ATTN applies a sigmoid gate to the output projection of attention heads without changing the pre-softmax distribution. RAVE demonstrates that shifting the gate to pre-softmax visual logits is the necessary mechanism to fundamentally rectify cross-modal attention dilution.

Rating

  • Novelty: โญโญโญโญ Elegant identification of RoPE's role in visual attention dilution and a beautifully minimalist pre-ROPE pair-gating formulation.
  • Experimental Thoroughness: โญโญโญโญโญ Rigorous controlled experimental setting matching D-ATTN protocols, extensive ablations across locations, forms, and ratios, plus strong validation on Qwen2.5-VL.
  • Writing Quality: โญโญโญโญโญ Clear narrative progression, insightful diagnostic visualizations (Figures 1 & 2), and concise mathematical framing.
  • Value: โญโญโญโญโญ Outstanding drop-in applicability for any modern decoder-only vision-language model suffering from long-generation visual forgetting.