Rethinking Attention Reallocation for Multimodal Emotion Recognition¶

Conference: ECCV 2026
Paper: ECCV 2026
Code: https://github.com/yzl77/ReAR
Area: Multimodal VLM
Keywords: Multimodal Large Language Model, Multimodal Emotion Recognition, Attention Reallocation, Shallow Structural Prior, Training-Free Inference

TL;DR¶

Uncovers the counterintuitive phenomenon where attention reallocation from generated to modality tokens helps in two-modality settings but hurts in three-modality settings, and proposes a training-free attention rectification method leveraging shallow attention priors to regularize entangled deep-layer attention.

Background & Motivation¶

With the rapid development of Multimodal Large Language Models (MLLMs), Multimodal Emotion Recognition (MER) has progressively transitioned from conventional categorical classification or captioning toward an open-ended generative paradigm. Under this paradigm, models consume heterogeneous multimodal signals—including video facial expressions, vocal audio, and transcribed dialogue text—to autoregressively generate descriptive and interpretable emotional deductions. However, existing studies commonly observe that autoregressive decoders tend to over-allocate attention to newly generated text tokens while systematically neglecting the multimodal context tokens that harbor critical affective cues. To alleviate the resulting hallucinations and modal collapse, recent popular strategies advocate inference-time attention reallocation, transferring attention mass from generated tokens back to modality tokens.

Crucially, prior investigations have implicitly assumed that this reallocation heuristic universally holds regardless of the multimodal input configuration. Through extensive empirical exploration, this paper discovers a striking and counterintuitive phenomenon: reallocating attention from generated tokens to modality tokens consistently boosts performance under two-modality configurations (e.g., video–text or audio–text), but severely degrades performance in three- or more-modality settings (e.g., video–audio–text). Even more surprisingly, in three-modality scenarios, executing the exact opposite reallocation direction—diverting attention from modality tokens toward generated tokens—inversely yields noticeable performance gains.

To understand the core mechanism driving this contradiction, the authors evaluate token informativeness by jointly assessing attention weights and value vector \(L_2\) norms. In two-modality settings, modality tokens preserve strong intrinsic signal magnitude, so amplifying their attention directly boosts task-relevant information. In contrast, under three or more modalities, intricate cross-modal interactions cause the attention distributions in deeper Transformer layers to become heavily entangled and disordered, making it difficult for the model to assign high attention to tokens that truly carry rich information. Core idea: leverage relatively clean and structured shallow-layer attention as an inference-time structural prior, dynamically extracting the most informative premature layer via Jensen–Shannon divergence and injecting an attentional residual map into the final layer.

Method¶

Overall Architecture¶

To rectify the degraded attention allocation caused by entangled cross-modal interactions in deep layers without incurring retraining overhead, the authors introduce a training-free attention rectification framework (ReAR). Built upon an MLLM-based MER backbone (such as AffectGPT, which integrates Qwen3 with CLIP ViT-L and HuBERT-L), ReAR operates purely during the autoregressive decoding stage at inference time, leaving backbone parameters completely untouched.

The overall pipeline comprises two complementary components: first, a Shallow Layer Extraction module that monitors the layer-wise attentional trajectory divergence using Jensen–Shannon divergence (\(\mathcal{D}_{JS}\)), dynamically identifying the optimal premature layer \(l^*\) that best preserves fine-grained multimodal structures; second, an Attention Refinement module that constructs an attentional residual map \(\Delta A = A^{(l^*)} - A^{(L)}\) and injects it as a gradient-like regularization signal into the final-layer attention distribution \(A^{(L)}\), followed by row-wise normalization before next-token generation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Input<br/>Video / Audio / Text / Pre-fused Tokens"] --> B["Shallow Transformer Blocks<br/>Compute intermediate attention maps"]
    B --> C["Shallow Layer Extraction<br/>Select premature layer l* via max JS divergence"]
    C --> D["Deep Transformer Blocks<br/>Compute entangled final-layer attention A(L)"]
    D --> E["Attention Refinement<br/>Inject attentional residual ΔA = A(l*) - A(L)"]
    E --> F["Row Normalization & Next-Token Decoding<br/>Accurate open-ended emotion generation"]

Key Designs¶

1. Shallow Layer Extraction: Capturing structured multimodal priors via attentional trajectory divergence

As input tokens traverse stacked Transformer layers, representations shift from fine-grained cross-modal feature grounding to task-aligned semantic representations. This semantic transition creates an information bottleneck in deeper layers, where nuanced acoustic and visual affective signals are progressively attenuated. To identify the exact layer that retains the maximal volume of these vanishing multimodal cues, the framework defines a candidate layer range \(\mathcal{L} = \{l \mid l \in [l_s, l_e)\}\) corresponding to the early formative phase of multimodal fusion. The discrepancy between the final layer \(A^{(L)}\) and an intermediate layer \(A^{(l)}\) is measured via Jensen–Shannon divergence (\(\mathcal{D}_{JS}\)):

\[\mathcal{D}_{JS}(A^{(L)} \parallel A^{(l)}) = \frac{1}{2}\mathrm{KL}(A^{(L)} \parallel M) + \frac{1}{2}\mathrm{KL}(A^{(l)} \parallel M)\]

where \(M = \frac{1}{2}(A^{(L)} + A^{(l)})\) is the consensus distribution, and \(\mathrm{KL}(\cdot)\) represents the Kullback–Leibler divergence. The optimal premature layer \(l^*\) is selected by maximizing this divergence:

\[l^* = \arg\max_{l \in \mathcal{L}} \mathcal{D}_{JS}(A^{(L)} \parallel A^{(l)})\]

This selection ensures that \(l^*\) captures the largest attentional variance suppressed by deeper layers, preserving critical multimodal cues for subsequent reasoning.

2. Attention Refinement: Regularizing entangled deep attention via residual injection

Once the premature layer \(l^*\) is identified, the next objective is to restore the attenuated multimodal signals without disrupting the high-level semantic reasoning established in the final layer. Naively substituting the final-layer attention with shallow attention disrupts deep contextual synthesis, whereas uniform regularization lacks modality-specific awareness. To achieve adaptive correction, the authors formulate the Attentional Residual Map \(\Delta A\) as:

\[\Delta A = A^{(l^*)} - A^{(L)}\]

Here, \(\Delta A\) acts as an attentional gradient signal highlighting modality tokens that are underrepresented in deeper layers. The final-layer attention is recalibrated by injecting this scaled residual signal governed by parameter \(g\):

\[\tilde{A}^{(L)} = A^{(L)} + g \cdot \Delta A\]

To restore a valid probability distribution, a row-wise normalization operator \(\Psi(\cdot)\) is applied:

\[\tilde{A}^{(L)} \leftarrow \Psi(\tilde{A}^{(L)})\]

This residual injection rectifies misallocated attention weights while maintaining the probabilistic validity of the self-attention map, ensuring that autoregressive predictions draw upon both contextual reasoning and clean, well-structured multimodal evidence.

Loss & Training¶

The proposed method is strictly training-free during inference. In the underlying MLLM training phase (following AffectGPT), unimodal encoders (CLIP ViT-L and HuBERT-L) remain frozen, while LoRA parameters within the Qwen3 LLM backbone, modality projectors, and the cross-modal pre-fusion module are fine-tuned using AdamW with a learning rate of 1e-5 and batch size of 3 for up to 60 epochs. During inference, ReAR modifies only the final-layer self-attention computation on the fly with zero gradient computation and zero additional parameters.

Key Experimental Results¶

Main Results¶

The model is evaluated on the MER-UniBench benchmark, encompassing 9 datasets across basic emotion recognition (MER2023, MER2024, MELD, IEMOCAP), sentiment analysis (CMU-MOSI, CMU-MOSEI, CH-SIMS, CH-SIMS v2), and fine-grained emotion recognition (OV-MERD+). Representative results under two-modality and three-modality settings are summarized below:

Input Modality	Methods	MER2023 (Basic)	MELD (Basic)	MOSI (Sentiment)	SIMS (Sentiment)	OV-MERD+ (Fine-grained)	Average Score (Avg)
Audio, Text	SALMONN	55.53	45.62	81.00	68.69	45.00	57.89
Audio, Text	AffectGPT	72.94	56.63	83.46	82.99	59.98	72.18
Audio, Text	Ours (ReAR)	75.78	62.74	81.79	87.48	62.33	75.38 (+3.20)
Video, Text	VideoChat2	33.67	36.64	66.84	69.49	39.21	52.67
Video, Text	AffectGPT	74.58	57.63	82.39	87.20	61.65	74.31
Video, Text	Ours (ReAR)	79.48	62.06	82.27	90.72	63.52	76.98 (+2.67)
Video, Audio, Text	PandaGPT	40.21	37.88	61.92	68.38	37.12	52.92
Video, Audio, Text	R1-Omni	64.17	43.20	58.02	71.82	55.24	59.61
Video, Audio, Text	Emotion-LLaMA	59.38	46.76	66.13	78.32	52.97	64.17
Video, Audio, Text	AffectGPT	78.54	55.65	81.30	88.49	62.52	74.77
Video, Audio, Text	Ours (ReAR)	84.49	62.25	83.31	89.77	65.32	78.08 (+3.31)

Ablation Study¶

Ablations conducted under the full three-modality setting validate each component and benchmark against alternative refinement strategies:

Configuration / Strategy	Fine-grained	Basic	Sentiment	MER-UniBench Avg	Note
Baseline	64.50	67.07	84.06	74.34	Unmodified AffectGPT baseline
+ Shallow Layer Guidance	65.23	71.19	84.60	76.48	Guidance prior from premature layer
+ Residual Refinement (Ours Full)	65.32	74.06	85.30	78.08	Complete residual injection and rectification
(a) Shallow Layer Replacement	65.26	73.64	84.81	77.68	Direct replacement perturbs high-level semantics
(b) Uniform Prior Injection	65.27	72.50	85.04	77.27	Lacks modality-specific structural differentiation
(c) Layer-wise Residual Aggregation	65.28	70.05	84.65	76.01	Heuristic multi-layer aggregation introduces noise
Length-Constrained Decoding	63.90	66.03	79.26	71.67	Forcing longer generation does not yield gains
Layer-Guided Logit Refinement (DeCo)	64.82	67.07	83.98	74.33	Logit-level contrasting fails to fix attention bias
Ours w/o Fused Token	61.94	69.36	83.48	74.81	Consistently outperforms baseline w/o fused token (73.47)

Key Findings¶

Resolution of the reallocation paradox: In two-modality settings, the baseline scores 78.17 on MER2023; transferring attention \(G \to M\) increases it to 79.04 while \(M \to G\) drops to 78.11. Under three modalities, \(G \to M\) drops to 76.52, whereas \(M \to G\) increases to 80.53. ReAR resolves this discrepancy by tapping into early structural priors, achieving superior scores of 79.48 in two-modality and 84.49 in three-modality configurations.
Contribution ratio correlates with performance: Defining the token contribution ratio as \(C_{\{G\}} / C_{\{M\}}\) (incorporating attention weight and value L2 magnitude), every configuration yielding empirical gains exhibits a decreased ratio (higher relative modality contribution). ReAR achieves the lowest ratio across both two-modality (1.3480 vs. baseline 1.4015) and three-modality (0.2099 vs. baseline 0.2128) setups.
Optimal depth for prior extraction: Layer-selection analysis demonstrates that extremely shallow layers (layers 0–6) lack cross-modal semantic interaction, whereas intermediate formative layers (layers 6–18) yield peak performance (exceeding 80% Hitrate), validating the choice of candidate selection range \(\mathcal{L}\).

Highlights & Insights¶

Counterintuitive empirical discovery: Demonstrates that the widely adopted heuristic of reallocating attention to modality tokens fails once more than two modalities are involved, offering crucial insights into multimodal token dynamics in autoregressive architectures.
Training-free, zero-parameter integration: The residual injection mechanism operates purely at inference time without requiring gradient backward passes or architectural modifications, making it easy to deploy on any decoder-only MLLM.
High generalizability across tasks: By tackling the root cause of deep cross-modal attention entanglement, the approach shows broad applicability across fine-grained categorization, sentiment polarity, and complex affective reasoning.

Limitations & Future Work¶

Author-admitted limitations: The current validation is primarily centered around MLLM-based emotion recognition architectures (AffectGPT/Qwen3) on MER-UniBench; its scalability to extreme multimodal systems involving tactile, physiological, or dense sensor streams remains to be explored.
Self-identified limitations: Evaluating the Jensen–Shannon divergence across all intermediate candidate layers during decoding adds computational overhead that could increase latency on ultra-long multimodal sequences.
Future directions: Developing lightweight heuristic routing or learned sparse gates to predict the optimal premature layer index without full divergence sweeps across every candidate block.

vs. Attention Reallocation (Tu et al. / Liu et al.): Prior works only examine two-modality (vision-language) setups and apply unidirectional attention boost toward image tokens; this paper identifies the multi-modality reversal paradox and resolves it via bidirectional shallow-prior residual regularization.
vs. DeCo (Dynamic Correction Decoding): DeCo manipulates output logits using contrastive decoding, leaving internal attention entanglement unresolved; ReAR operates directly on intermediate attention maps before final prediction logits are computed.
vs. DoLA (Decoding by Contrasting Layers): DoLA contrasts layer-wise output logits, achieving only 68.38 on MER-UniBench; ReAR leverages structural attention priors inside the self-attention blocks, outperforming DoLA by 9.7 points (78.08).

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Uncovers a surprising reversal in multimodal attention dynamics and addresses it with an elegant training-free shallow prior mechanism.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated on 9 diverse datasets covering basic, sentiment, and fine-grained tasks across two- and three-modality configurations with thorough ablations.
Writing Quality: ⭐⭐⭐⭐⭐ Well-structured narrative with crisp mathematical formulations, convincing empirical evidence, and clear visualization.
Value: ⭐⭐⭐⭐⭐ Training-free, lightweight, and effective; provides valuable foundational insights into multi-modality scaling behavior for future MLLM architectures.