ADAPT: Attention Dynamics Alignment with Preference Tuning for Faithful MLLMs¶

Conference: ECCV 2026
arXiv: 2606.31054
Code: https://github.com/yao-ustc/ADAPT
Area: Multimodal VLM
Keywords: MLLM Hallucination, Cross-Attention Degradation, Visual Anchor, Attention-Supervised Decoding, Visually-Conditioned Preference Optimization

TL;DR¶

ADAPT distills semantically aligned visual anchors from the multi-layer cross-attention of early decoding steps. During inference, it uses these anchors as references to monitor attention fidelity online and applies sparse corrections only when significant degradation is detected. Additionally, it strengthens the model's reliance on visual evidence through visually-conditioned contrastive preference optimization, reducing hallucination rates by 40–60% on multiple hallucination benchmarks with only a 1.42x inference overhead.

Background & Motivation¶

The hallucination issue of Multimodal Large Language Models (MLLMs) is the primary bottleneck for trustworthy deployment. Models tend to describe non-existent objects, attribute incorrect properties, or create non-existent spatial relations in their generations, and even mainstream models like GPT-5, Qwen-VL2, and LLaVA are not immune. Existing mitigation strategies fall into two lines of work: training-time alignment, which uses RLHF/DPO and their vision-aware variants (mDPO, RLHF-V, SIMA) to fine-tune model parameters for factual responses; and inference-time intervention, which modifies the output distribution at the final layer using contrastive decoding (VCD), penalty strategies (OPERA), or attention head suppression (SPIN). However, these methods share a common characteristic: they retrospectively adjust the distribution at the token output level, ignoring the internal generation process of hallucinations. Even if a model passes a hallucination benchmark, it does not guarantee that it truly "saw" the corresponding regions in the image—language priors and positional biases interweave, causing hallucinations to recur in complex scenarios.

In this paper, the authors analyze the internal signals of MLLMs, systematically examining the dynamic changes in text-to-image cross-attention during generation. They discover a highly consistent degradation phenomenon: the generation process irreversibly slides from the early "grounding stage" to the "prior-dominated stage". The cross-attention exhibits three quantifiable failure modes—attention drifts away from text-related visual evidence to become dispersed or develop spatial preferences, becomes insensitive to newly generated tokens, and the total attention on visual tokens decreases layer by layer. This suggests that the degradation of cross-attention occurs before the hallucinated tokens are selected. If the precursors of such degradation can be detected in real-time at the attention level, it is possible to intervene before the model actually outputs hallucinations.

The key insight of this work is to track and correct cross-attention as an actionable internal signal, rather than modifying the output probability after incorrect generations occur.

Core Idea: Distill semantically aligned visual anchors from reliable cross-attentions in early decoding, use these anchors as references to monitor attention fidelity online during inference, apply sparse corrections only when significant degradation is detected, and use visually-conditioned contrastive preference optimization to teach the model to distinguish between "seeing" and "guessing" correctly.

Method¶

Overall Architecture¶

ADAPT designs a three-stage collaborative framework centered around a single visual anchor. First, a pre-processing step is performed on the multi-layer cross-attention of the first \(K\) tokens—fusing them with weights based on three complementary criteria (spectral, smoothness, and focus) and subtracting the model's inherent spatial bias—to obtain a query-adaptive, stable visual anchor. This anchor is then utilized in two parallel interventions. During inference, the deviation (AMC metric) between the current attention and the anchor is computed at each decoding step, triggering logit bias corrections with the anchor weights only when the metric falls below a threshold. During training, anchor-enhanced images and pure noise images are used to construct visually-conditioned contrastive preference pairs, encouraging the model to be confident when visual evidence is sufficient and cautious when visual information is absent. The three stages share the same anchor but use it differently—inference is "passive monitoring" while training is "active shaping".

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image + Text Query"] --> B["Short Prefix Inference (First K tokens)<br/>Collect multi-layer cross-attention"]
    B --> C["Three-criterion Fusion + Debiasing Mask"]
    C --> D["Visual Anchor A_anchor"]

    D --> E["Decode each step, compute AMC fidelity"]
    E -->|AMC >= τ| F["No intervention, normal decoding"]
    E -->|AMC < τ| G["Sparse Attention Correction<br/>log anchor bias"]
    F --> H["Generate Output"]
    G --> H

    D --> I["VAG-DPO Training"]
    I --> J["i. Anchor-enhanced image + ground-truth response<br/>ii. Noise image + hallucinated response"]
    J --> K["Updated MLLM Parameters"]
    K --> E

Key Designs¶

1. Visual Anchor via Three-Criterion Fusion and Debiasing: Distilling reliable query-adaptive references from multi-layer cross-attention

Original cross-attention has two main defects as an intervention signal: it is highly noisy, with many semantic-irrelevant high-frequency components mixed in; and it suffers from positional bias—the "attention sink" phenomenon in Transformers causes certain spatial locations to receive excessively high focus regardless of image content. This paper's strategy is to first collect \(L\)-layer cross-attention maps during the first \(K=5\) tokens (which are in a reliable visual grounding phase) and score each layer using three complementary criteria. The spectral criterion uses 2D FFT to detect the high-frequency energy ratio and suppress unstable grounding; the smoothness criterion uses total variation regularization to penalize fragmented distribution and encourage focus on contiguous object regions; the focus criterion dynamically adjusts the target entropy based on query semantics—for fine-grained queries like "what is the color", it focuses on local regions, while for global queries like "describe the whole scene", it diffuses attention. The scores from the three criteria are fused via learnable projection weights, which are grid-searched on the AMBER validation set to be \(\omega_{spec}=0.4\), \(\omega_{smooth}=0.3\), and \(\omega_{focus}=0.3\). The optimal region is a gentle valley rather than a sharp peak, confirming that the three criteria are complementary rather than redundant. After fusion, positional bias must be removed: a pure noise image is input to the model with the same query to calculate the "content-absent" attention prior \(M_{bias}\), and its reciprocal is used as a debiasing mask to perform element-wise multiplication with the fused map, yielding the final query-adaptive anchor \(A_{anchor}\).

2. Online Attention Degradation Monitoring and Conditional Triggered Correction Based on the AMC Metric

Attention degradation does not occur uniformly during decoding—the model may maintain good visual grounding in most steps and only become unstable at a few key steps. This paper defines the Anchor-Modulated Concentration (AMC) metric to quantify the alignment between the current attention and the anchor:

\[S_{AMC}(A_t \| A_{anchor}) = \frac{\sum_i (a_{i,t}^2 \cdot w_i^{anchor})}{(\sum_i a_{i,t})^2 + \epsilon}\]

The core of AMC is embedding the anchor weights into the concentration calculation: when attention is concentrated on regions where the anchor itself also has high weights, the numerator is large and the score is high; when attention diffuses to background or positional-biased regions where the anchor weights are low, the numerator is small and the score is low. AMC is computed at each decoding step, and correction is triggered only when it falls below a threshold \(\tau=0.6\), adding a mild guidance by applying a logarithmic bias of the anchor weights to the attention logits. This conditional trigger mechanism avoids extra computation at every step, keeping ADAPT's inference overhead at only 1.42x, which is significantly lower than OPERA (7.20x) and GF-SCD (3.96x).

3. Visually-Conditioned Preference Optimization: Teaching the model to distinguish "seeing" from "guessing" correctly

Standard DPO performs preference contrast at the text level—comparing the probabilities of correct and incorrect responses given the same image. The model learns "which text looks more like the standard answer" but does not differentiate whether the faithfulness of the answer stems from visual evidence or language priors. VAG-DPO alters the contrasted variables: it fixes the correctness label of the response but changes the dependent image condition. For a factual response \(y_c\), the chosen branch views its generation probability under the anchor-enhanced image; for a hallucinated response \(y_r\), the rejected branch views its probability under a pure noise image. The optimization target encourages the model to be confident when visual evidence is sufficient and cautious when visual evidence is missing—VAG-DPO forces the model to distinguish between two fundamentally different states: "I indeed saw it in the image" versus "I am just guessing an answer based on language priors", reinforcing its reliance on visual evidence from a mechanistic perspective rather than output statistics.

Loss & Training¶

The loss function of VAG-DPO shares the same form as standard DPO, but the core difference lies in the conditions of the two probability terms:

\[\mathcal{L}_{\text{VAG-DPO}} = -\mathbb{E}_{\mathcal{D}}\left[\log\sigma\left(\beta\log\frac{P_\theta(y_c|I_{\text{VE}},Q)}{P_{\text{ref}}(y_c|I_{\text{VE}},Q)} - \beta\log\frac{P_\theta(y_r|I_{\text{noise}},Q)}{P_{\text{ref}}(y_r|I_{\text{noise}},Q)}\right)\right]\]

where \(I_{\text{VE}}\) is the anchor-enhanced image, and \(I_{\text{noise}}\) is the pure noise image. The preference dataset is constructed based on subsets of RLAIF-V and RLHF-V, keeping the original response pair labels and only replacing the image conditions. Training hyperparameters: \(\beta=0.1\), learning rate 1e-6, 1 epoch.

Key Experimental Results¶

Main Results¶

Backbone	Method	Obj-Hal Chairs↓	AMBER Chair↓	AMBER Hal↓	MM-Hal Cog↓	POPE-Adv Prec↑
LLaVA-1.5-7B	Baseline	54.4	7.8	36.4	4.2	76.1
	+ADAPT-TF	35.6	4.0	16.7	1.1	90.3
	+ADAPT	30.7	3.8	15.2	1.2	91.9
Qwen2.5-VL-3B	Baseline	43.2	8.0	44.4	4.3	93.2
	+ADAPT	15.2	3.1	13.5	0.5	94.0
Qwen2.5-VL-7B	Baseline	32.3	4.7	21.9	1.2	93.5
	+ADAPT	16.8	3.5	13.1	0.7	93.5
LLaVA-1.5-13B	Baseline	49.8	7.0	29.0	3.2	84.5
	+ADAPT	34.8	4.2	17.7	1.4	90.6

Ablation Study¶

Configuration	Chair↓	Hal↓	Cog↓	Description
LLaVA-7B Baseline	7.8	36.4	4.2	No intervention
+Visual Enhance (only)	4.7	18.7	1.8	Only anchor-enhanced input, no decoding correction
+Attention-Supervised (only)	6.3	27.8	2.9	Only inference-time attention monitoring and correction
+VAG-DPO (only)	6.0	25.0	2.1	Only training-time visually-conditioned DPO
+ADAPT (full)	3.8	15.2	1.2	Synergy of all three components

Key Findings¶

Strong synergy among the three components rather than simple addition: Visual Enhance alone achieves the best individual effect (Chair 7.8 \(\to\) 4.7), but the full ADAPT (3.8) significantly outperforms any single component, showing a complementary rather than redundant superimposing effect among a "good starting point" (anchor), "real-time correction" (monitoring), and "internal preference shaping" (DPO).
Inference efficiency superior to similar methods: ADAPT's 1.42x overhead is significantly lower than OPERA (7.20x), GF-SCD (3.96x), and VCD (2.03x). The core lies in conditional triggering—when attention is normal at most decoding steps, AMC passes detection and triggers no extra operations.
Anchors themselves serve as high-precision visual grounding features: Applying the ADAPT anchor to RefCOCOg grounding achieves Acc 50.3 / mIoU 57.0, significantly outperforming LLaVA-7B (32.6/41.8) and even CLIP (45.9/51.3), proving that anchors fused and debiased via the three criteria can directly serve as high-quality, transferable semantic localization features.
Better preservation of general capabilities: The decline of VAG-DPO on MME (-51.52) is about 25% less than standard DPO (-68.27), indicating that the contrastive strategy of "fixing responses and changing visual conditions" is milder than "fixing images and changing responses"—it does not force the model to assign high probabilities to low-confidence answers when visual evidence is weak.

Highlights & Insights¶

Paradigm shift from "output-level post-hoc remedy" to "internal-signal pre-emptive prediction": Almost all existing mainstream methods (VCD, OPERA, SPIN) operate on token probabilities. ADAPT is the first to systematically treat cross-attention degradation as an actionable intervention signal. This perspective itself is highly inspiring—intervening in internal states before incorrect output even occurs.
Engineering ingenuity of the AMC metric: Embedding anchor weights into the attention concentration calculation allows a single formula to achieve both drift detection and noise filtering. It is more accurate and robust against positional bias than simply looking at attention entropy or maximum attention values.
The contrastive strategy of "fixing responses and changing visual conditions": The core insight of VAG-DPO is profound—standard DPO teaches the model what is correct to say, but not how to distinguish "correctness stemming from seeing" versus "guessing". Swapping the contrast variables from response correctness to the strength of visual evidence is the first direct optimization targeted at the fundamental conflict of "language priors vs. visual evidence".
Design philosophy of conditionally-triggered intervention: ADAPT does not seek to actively correct at every step, but "intervenes only when necessary"—preserving the fluency of normal decoding while keeping extra computational overhead to a minimum.

Limitations & Future Work¶

Hallucinations do not purely originate from attention degradation—reasoning errors, missing knowledge, and instruction-following failures also generate hallucinations. ADAPT only targets the single failure mode of cross-attention degradation, and is not guaranteed to be effective against non-degrading hallucinations. A more comprehensive framework requires joint monitoring of multiple internal signals.
The assumption that the first \(K\) tokens are in a "reliable grounding phase" might violate extreme scenarios—if the model starts hallucinating within the first few tokens (e.g., misidentifying the subject in complex scenes), the extracted anchor itself will be incorrect, potentially amplifying hallucinations in reverse. The robustness of \(K\) and automatic validation mechanisms of anchor quality require further study.
VAG-DPO's negative sample pairs with noise images might disrupt key structural information for tasks sensitive to precise spatial relationships (e.g., table understanding, chart reading, OCR), creating unreasonable contrastive pairs and interfering with learning. A gentler visual condition degradation strategy needs to be designed for these tasks.

vs VCD (Contrastive Decoding): VCD suppresses language priors by contrasting the original distribution with a distorted input distribution, requiring two forward passes per step as an output-level post-processing. ADAPT performs positive anchor guidance in the attention dimension and fundamentally reinforces visual reliance during the training stage.
vs OPERA: OPERA checks for attention pattern anomalies of candidate tokens at each step and penalizes them, resulting in an extremely high overhead of over 7x. ADAPT's conditional triggering achieves a better balance between efficiency and hallucination suppression.
vs SPIN (Attention Head Suppression): SPIN identifies and statically suppresses attention heads that undervalue image tokens prior to training. It lacks the flexibility and query-adaptiveness of ADAPT's dynamic, on-demand correction.
vs IBD (Image Biased Decoding): IBD globally amplifies image-biased attention to combat language priors. ADAPT selectively amplifies attention via AMC anchor weights, intervening only where necessary.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Starting from the internal signal of cross-attention degradation to systematically analyze and design a three-stage intervention framework is highly paradigm-shifting.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Coverage of 4 benchmarks and 4 backbones, with comprehensive and clear four-dimensional analysis across ablation, efficiency, grounding, and general capabilities.
Writing Quality: ⭐⭐⭐⭐⭐ First reveals the degradation phenomenon and then designs the intervention; the methodology progresses step-by-step with well-placed figures/tables and self-consistent logit.
Value: ⭐⭐⭐⭐⭐ Hallucination is the primary trustworthiness bottleneck of MLLMs. ADAPT achieves SOTA hallucination mitigation with only 1.42x overhead, demonstrating high practical utility.