Looking Back and Forth: Cross-Image Attention Calibration and Attentive Preference Learning for Multi-Image Hallucination Mitigation¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/yyyxcleo/CAPL
Area: Multimodal VLM
Keywords: multi-image hallucination, cross-image attention calibration, direct preference optimization, attention mask, vision-language model

TL;DR¶

This paper attributes multi-image hallucination to the unidirectional cross-image information flow induced by autoregressive causal attention, and proposes CAPL: it calibrates attention so that only key tokens become mutually visible across images, then builds preference pairs by generating positives and negatives under the full cross-image mask and a fully truncated one, trained with DPO plus NLL — yielding consistent 1–3.6 point gains on BLINK/MUIRBench without hurting single-image ability.

Background & Motivation¶

Multi-image inputs are increasingly common in real applications — multi-view comparison, cross-image information integration, and interleaved image-text dialogue all require a model not only to understand each image but also to establish relations and consistency across images. Models such as Idefics and Qwen-VL already support multi-image inputs, yet their hallucinations take a different form in the multi-image setting than in the single-image one: rather than object-existence hallucinations of the "is this object in the picture" kind, the failures are cross-image mismatches — information from image A attributed to image B, entities and attributes that do not exist introduced during relational reasoning, or a fluent-looking answer that simply ignores the decisive differences between images. Most existing mitigation methods were designed for single images: decoding strategies (such as over-trust penalties in the OPERA line), visual contrastive decoding, and alignment training. In the multi-image regime these either do not apply or only adjust the decoding distribution through local structural modifications, leaving the question of how images actually interact untouched. Later work did target multi-image settings with dedicated decoding methods (e.g. the MIHBench line) and carefully designed multi-image training schemes (MIA-DPO, PERL), but most still treat each image as an independent context, so inter-image relations are never explicitly modeled.

The structural root of the problem lies in attention itself. Existing Transformer-based autoregressive LVLMs place text tokens and image tokens in a single sequence under a unified causal attention mask: multi-image inputs are processed in order, so later images can attend to earlier ones while earlier images have no access to later ones at all. This one-directionality introduces an inherent positional bias and breaks the symmetry that cross-image relational modeling requires — the model struggles to build stable, symmetric associations at the image-token level. When cross-image visual interaction is insufficient, multi-image reasoning degenerates into superficial correlation matching over text tokens rather than genuine relational modeling grounded in visual evidence, and predictions become increasingly reliant on language priors and autoregressive generation bias. The paper illustrates this with a consistency question in Fig. 1(c): the incorrect answers share a common trait of ignoring inter-image distinctions, whereas the correct answer leverages the key cross-image information.

The paper's angle is direct: if the bias comes from the one-directional attention mask, then remove the causal constraint between images so that tokens from different images see each other — the "Looking Back and Forth" in the title refers precisely to this mutual visibility, where later images can look back at earlier ones and earlier images can look forward to later ones. But modifying the mask only at inference time is a stopgap: the models were pretrained under the causal paradigm and may not adapt to this mutual-interaction scheme, while ordinary SFT merely imitates positive samples and cannot penalize the model's own hallucinatory reasoning paths. Core idea: first calibrate the inference-time information flow with a selective cross-image attention that keeps intra-image causality but opens inter-image causality, then use the two extreme forms of that same attention (all cross-image connections open vs. all truncated) to induce positive samples and "hallucination-prone" negative samples, writing the use of cross-image evidence into the parameter space — namely Cross-Image Attention calibration plus Preference Learning (CAPL).

Method¶

Overall Architecture¶

CAPL addresses one thing — that under multi-image input the model does not look at relations between images — and does it in two parts: modifying the mask along the inference path, and using the two masks produced by that modification to build preference data and align the model along the training path. The input is N images plus interleaved text (question and images); the output is the model's generated response. Three stages lie in between: first select each image's "key tokens" by visual-token response intensity and open the causal mask among them to obtain the enhanced mask; then cut all cross-image connections entirely to obtain the truncated mask, and generate once under each mask for the same question to serve as positive and negative samples; finally pull the two apart with DPO while an NLL term makes the model imitate the token-level generation trajectory of the positive samples. At inference time the enhanced mask is used directly, and the trained model has internalized the habit of "attending across images when it should."

One point worth stating up front: what is modified here is the decoder self-attention mask, not a new attention module or an extra network branch — image tokens stay in the original sequence and only "who can see whom" changes. The backbone structure therefore needs no modification, and the method drops onto existing models such as Qwen2.5-VL, InternVL2.5, and GLM4.1VBase directly.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-image input + text question"] --> B["Selective cross-image token interaction<br/>energy-based key tokens, cross-image mask"]
    B -->|enhanced mask yields positives / truncated mask yields negatives| C["Adversarial preference pair construction"]
    C --> D["Cross-image attention guided preference optimization<br/>DPO contrast + NLL imitating positives"]
    D --> E["CAPL-tuned LVLM"]

Key Designs¶

1. Selective cross-image token interaction: make only key tokens mutually visible instead of opening everything

The pain point is the one-directional information flow described above: tokens of later images can see earlier images, but earlier images cannot see later ones. The paper redefines the mask — removing the causal constraint altogether between tokens of different images, while preserving the causal mask exactly within each image, so intra-image positional structure survives while the cross-image one-directionality is broken. Opening all cross-image connections, however, would introduce a great deal of redundant interaction, since information density and semantic relevance vary widely across images and bidirectional attention between irrelevant tokens is pure noise. The paper therefore adds an embedding-energy filter: for the visual tokens of image k, compute the response intensity (L2 norm) \(s_{k,i}=\|\mathbf{h}_{k,i}\|_2\) and keep only the top \(\lfloor \rho \tau_k \rfloor\) tokens (\(\rho\) is the selection ratio and \(\tau_k\) the token count of that image) as the key-token set \(\mathcal{S}_k\). The resulting selective cross-image mask is set to zero (i.e. visible) only where both sides belong to their respective images' key-token sets, and falls back to the original causal mask everywhere else:

\[ \mathbf{M}^{\mathrm{cross\_sel}}_{ij}= \begin{cases} 0, & g(i)\neq g(j),\ i\in\mathcal{S}_{g(i)},\ j\in\mathcal{S}_{g(j)}\\ \mathbf{M}^{\mathrm{causal}}_{ij}, & \text{otherwise} \end{cases} \]

where \(g(i)\) is the index of the image token \(i\) belongs to. Two implementation details round this off. First, the final attention weights are an equal-weight fusion of the selective cross-image attention and the original causal attention rather than a wholesale replacement — multi-image tasks also include temporally dependent queries and single-image queries, for which preserving sequential structure remains necessary. Second, an alternating scheme across decoder layers: odd-numbered layers apply the selective cross-image mask and even-numbered layers keep the original causal mask, so cross-image interaction does not disrupt the autoregressive generation pathway. On its own this design gives modest but stable gains (0.2–0.7 points on every backbone in the ablation); its greater value is providing the generation condition for the preference learning that follows.

2. Adversarial preference pair construction: forcing the model's own hallucinations out by blocking cross-image attention

DPO depends heavily on negative-sample quality — negatives further from the true distribution give stronger optimization signals, whereas simply using the model's raw outputs as negatives tends to yield mediocre errors that fail to expose its hallucination patterns. The paper's idea is to turn the causal mechanism against itself: since causal attention allows only one-way flow across images in multi-image settings, push that asymmetry to the extreme by defining a truncated mask \(\mathbf{M}^{\mathrm{trunc}}\) that keeps causality within each image but sets all attention between different images to \(-\infty\), making the images representationally independent. The model can then only rely on a single image and language priors, so cross-image relational reasoning degenerates into inference from text priors and the hallucination rate rises markedly — exactly what is wanted for rejected samples. Fig. 3 compares negatives produced by the original structure against those produced by the truncated structure; the latter are about 20% less accurate on GLM4.1VBase, confirming that truncation really manufactures harder error cases. On the positive side, responses are generated with the enhanced selective cross-image attention and then refined with feedback from a stronger model (Qwen3) to guarantee quality. The paper also adds a special class of negatives: responses whose final answer is correct but whose reasoning path is wrong because cross-image attention was blocked are also fed into DPO, teaching the model to avoid such "right conclusion, wrong process" errors.

3. Cross-image attention guided preference optimization: DPO separates the probabilities, NLL restores trajectory imitation

The preference pair consists of an answer produced under bidirectional cross-image attention and one produced with cross-image attention fully blocked, which makes the pair a controlled contrast over "whether cross-image evidence was used." Given the image-text input \(x_{\text{I,T}}\), with \(y^+\) the output under the enhanced mask and \(y^-\) the output under the truncated mask, the training objective contains a standard DPO term:

\[ \mathcal{L}_{\text{DPO}}(\pi_\theta;\pi_{\text{ref}})=-\log\sigma\Big(\beta\log\tfrac{\pi_\theta(y^+\mid x_{\text{I,T}})}{\pi_{\text{ref}}(y^+\mid x_{\text{I,T}})}-\beta\log\tfrac{\pi_\theta(y^-\mid x_{\text{I,T}})}{\pi_{\text{ref}}(y^-\mid x_{\text{I,T}})}\Big) \]

(⚠️ Eq. 12 of the original paper is typographically corrupted; this is organized in the standard DPO form, and the symbols should be checked against the original.) The authors stress the key difference from SFT: SFT only imitates positives and cannot penalize the model's own erroneous reasoning patterns, whereas DPO explicitly contrasts positive and negative generation probabilities and can write "prefer cross-image evidence" into the parameter space rather than leaving it as an inference-time mask adjustment.

DPO, however, only orders positives above negatives; it does not force the model to reproduce the high-quality positive trajectory token by token, so the model may learn the preference ordering without internalizing the structured cross-image reasoning process. The paper therefore adds a negative log-likelihood term on the positive samples:

\[ \mathcal{L}_{\text{NLL}}(\pi_\theta)=-\sum_{t=1}^{T}\log\pi_\theta\big(y^+_t\mid x_{\text{I,T}},y^+_{<t}\big) \]

The two terms are combined into \(\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{DPO}}+\lambda\mathcal{L}_{\text{NLL}}\), with \(\lambda\) weighting the imitation term. The whole procedure trains LoRA only (rank 16, learning rate 1e-4) and leaves the backbone untouched.

Loss & Training¶

The training set contains 3.6K self-built samples drawn from WikiArt, MDVP, Mantis-Instruct, and VLM2Bench, with the guarantee that no data overlaps the evaluation benchmarks. Because the backbones differ architecturally, hyper-parameters are set per model: the selection ratio \(\rho\) is 0.95 for Qwen2.5-VL and InternVL2.5 and 0.9 for GLM4.1VBase; the NLL weight \(\lambda\) is 2.5 / 2.5 / 2 respectively. Training uses LoRA (rank 16, learning rate 1e-4); Qwen2.5-VL and GLM4.1VBase are trained with LLaMAFactory and InternVL2.5 with MS-Swift, and all models are evaluated with VLMEvalKit on NVIDIA L20 GPUs.

Key Experimental Results¶

Main Results¶

Across three backbones (Qwen2.5-VL-7B-Instruct, InternVL2.5-8B, GLM4.1VBase-9B), comparison on the multi-image hallucination benchmarks BLINK / MUIRBench and multi-image general benchmarks:

Model	Params	BLINK↑	MUIRBench↑	NLVR2	QBench2	MIBench	MIRB
Qwen2VL	7B	53.17	39.57	87.41	76.8	69.20	31.68
InternVL2	7B	50.34	45.61	77.68	69.8	59.37	53.15
Qwen2.5-VL	7B	54.60	58.42	79.85	71.1	72.42	52.73
+CAPL (Ours)	7B	57.76	62.00	80.05	72.4	71.06	56.55
InternVL2.5	8B	54.81	48.54	90.42	75.5	63.42	54.18
+CAPL (Ours)	8B	55.76	52.12	90.13	75.3	65.11	56.55
GLM4.1VBase	9B	58.17	57.84	84.98	74.4	70.86	59.96
+CAPL (Ours)	9B	61.33	60.57	84.87	73.6	71.70	60.06

Comparison with the recent multi-image alignment methods SOFA and MIA-DPO under the same architectures:

Model	Benchmark	Base	SOFA	MIA-DPO	CAPL
Qwen2.5-VL	BLINK	54.60	54.92	56.50	57.76
Qwen2.5-VL	MUIRBench	58.42	59.34	56.15	62.00
InternVL2.5	BLINK	54.81	55.02	54.45	55.76
InternVL2.5	MUIRBench	48.54	49.07	47.88	52.12
GLM4.1V	BLINK	58.18	58.18	59.76	61.34
GLM4.1V	MUIRBench	57.84	57.23	57.23	60.27

Performance on single-image benchmarks (↑ higher is better, ↓ lower is better):

Model	POPE↑	CHAIRi↓	CHAIRs↓	MMB↑	AMBER↑
Qwen2.5-VL	81.23	7.2	29.6	88.15	85.13
+CAPL (Ours)	82.94	7.5	28.6	87.48	85.25
InternVL2.5	88.94	6.4	24.4	84.11	88.99
+CAPL (Ours)	89.08	7.3	23.8	84.27	89.79
GLM4.1VBase	84.79	7.2	22.0	84.36	89.20
+CAPL (Ours)	86.20	6.5	18.4	84.62	88.49

Ablation Study¶

Progressively adding components (Base → cross-image attention only → full CAPL):

Config	Qwen2.5-VL BLINK	Qwen2.5-VL MUIRBench	InternVL2.5 BLINK	InternVL2.5 MUIRBench	GLM4.1VBase BLINK	GLM4.1VBase MUIRBench
Base	54.60	58.42	54.81	48.54	58.17	57.84
+Attn	55.34	58.96	55.02	49.07	58.23	58.07
Ours (Attn+DPO+NLL)	57.76	62.00	55.76	52.12	61.33	60.57

Negative-sample construction compared across all MUIRBench subtasks (GLM4.1VBase):

Config	Overall	Action	Similarity	Cartoon	Counting	Diagram	Difference	Geographic	I-T	Ordering	Scene	Grounding	Retrieval
GLM4.1VBase	57.9	46.3	58.7	46.2	39.3	73.9	57.9	41.0	68.8	20.3	66.1	28.6	59.6
+Attn	58.1	47.0	56.6	46.2	40.2	74.9	57.4	41.0	68.8	23.4	65.6	28.6	61.0
⊕ Original DPO	59.5	46.3	62.8	47.4	40.6	75.6	58.5	44.0	73.3	20.3	69.4	31.0	55.8
⊕ Truncated DPO	60.6	48.2	59.7	48.7	42.7	76.9	60.3	48.0	72.6	21.9	70.4	31.0	59.6

(⚠️ The first header label of Table 5 in the original does not fully match the number of remaining columns; the table here follows the subtask names and value correspondence indicated in the prose, and subtask naming should be checked against the original.)

Key Findings¶

Attention calibration and preference learning are complementary, not redundant. Adding cross-image attention alone gives small gains on both benchmarks for all three backbones (e.g. Qwen2.5-VL on MUIRBench: 58.42 → 58.96), and only after preference training is stacked on top does a clear jump appear (→ 62.00). The paper explains this as the former strengthening the model's ability to capture cross-image dependencies while the latter further regularizes the generation process with preference signals.
Gains are larger on MUIRBench than on BLINK, exceeding 3.5 points in the best case. That benchmark emphasizes complex cross-image relational reasoning, precisely where insufficient cross-image information flow hurts most, whereas BLINK gains are generally 1–3 points. Even a strong baseline such as GLM4.1VBase-9B still improves steadily, suggesting that advanced models still rely on fairly naive attention mechanisms in multi-image settings.
Worse negatives work better. Negatives from truncated attention are about 20% less accurate than those from the original structure and yield larger gains across MUIRBench subtasks, with Geographic Understanding +7.0, Scene Understanding +4.3, and Difference Spotting +2.4, beating original DPO on most subtasks (only Retrieval is flat).
Single-image ability does not degrade. CAPL is trained only on multi-image samples, yet it holds steady or slightly improves on POPE, CHAIR, MMBench, and AMBER (Qwen2.5-VL POPE 81.23 → 82.94; GLM4.1VBase CHAIRs 22.0 → 18.4). The authors conjecture that pushing the model away from negative samples also suppresses its latent single-image hallucination tendencies, while preference learning over multi-image visual information enriches visual knowledge. Note that a few individual metrics do slip (e.g. InternVL2.5 CHAIRi 6.4 → 7.3), so the effect is not uniformly positive.
The selection ratio \(\rho\) has an optimum. Performance rises with \(\rho\) but declines slightly as it approaches 1, indicating that a small share of image tokens is noise; keeping most but not all key tokens is the better trade-off, with optimal values of 0.95 for Qwen2.5-VL / InternVL2.5 and 0.9 for GLM4.1VBase. For the NLL weight \(\lambda\), too small a value makes the model over-rely on the DPO signal and weakens its language modeling ability, while too large a value dilutes the preference signal; the optimum is 2.5 / 2.5 / 2.

Highlights & Insights¶

It locates the cause of hallucination in the symmetry of the attention mask rather than in data or decoding. The change is a one-line mask redefinition (drop causality across images, keep it within), yet it directly matches what multi-image tasks need — cross-image relational modeling requires symmetric visibility. This way of tracing a bias into the architectural prior is more reusable than another round of training data.
The negative-sample construction is cleverly inverted. The usual approach lets the model run free and picks the wrong outputs as negatives; here the state the model is worst at (cross-image information fully invisible, degenerating to pure language priors) is turned into a generation condition that actively forces hallucinations out. It also forms two extremes of the same variable (cross-image visibility), making the preference pair a very clean contrast that needs no extra annotation or reward model.
The "right answer, wrong path" negatives are a transferable idea. In many multimodal tasks a correct final answer does not imply reliable reasoning; folding such samples into preference pairs adds a layer of process supervision on top of outcome supervision, and can transfer to chart QA, multi-hop VQA, or any setting where process reliability matters more than the answer.
Combining an inference-time change with training-time consolidation is worth borrowing. Changing the mask at inference time alone gives limited gains (only a few tenths of a point for +Attn in the ablation), but it also serves as the generator of high-quality positive samples, aligning the training signal with the deployment configuration. This "turn an inference trick into a data-construction device" move applies to any test-time method.

Limitations & Future Work¶

It presupposes that cross-image visibility is desirable. The method assumes the multi-image input genuinely contains information that needs cross-referencing. For "quasi single-image" settings (several semantically unrelated images that each need their own answer), forcing cross-image attention open may introduce interference; the paper mitigates this with equal-weight fusion and layer alternation but provides no quantitative analysis on such cases.
Positive-sample quality depends on external feedback from Qwen3, and the paper only states that positives are "refined with Qwen3" without a before/after comparison or any quality assessment, which weakens reproducibility there.
The selection ratio \(\rho\) is tuned per model (0.95 / 0.95 / 0.9); the three values are very close, and the sensitivity conclusion (decline near 1) rests on a single curve in Fig. 4, leaving unclear how much re-tuning a new model would cost.
The training set is small (3.6K) and, although deliberately disjoint from the evaluation sets, covers a limited range of task types; the reported gains concentrate on BLINK and MUIRBench, with no exploration of longer contexts (8+ images, multi-turn interleaved dialogue).
Table 1 and Table 2 disagree slightly for GLM4.1V (BLINK 58.17/61.33 vs 58.18/61.34; MUIRBench 60.57 vs 60.27) with no explanation, so care is needed when quoting.

vs training-free multi-image debiasing (e.g. MIHBench-style attention averaging): they average attention across images to reduce bias toward a particular image but do not update parameters, so the adjustment stays local to the decoding distribution; CAPL redefines the cross-image mask architecturally and writes that interaction into the parameters, which is why its advantage is clearer on a strongly relational benchmark such as MUIRBench.
vs MIA-DPO: MIA-DPO builds preference data through multi-image augmentation, but in these experiments it varies considerably across models and tasks (Qwen2.5-VL even drops from 58.42 to 56.15 on MUIRBench), and it mainly addresses a "quasi single-image within multi-image input" setting where images are often semantically unrelated; CAPL's preference pair uses "is cross-image evidence available" as the contrast variable, targeting genuine inter-image relational modeling.
vs ordinary SFT / RLHF: SFT uses positives only and cannot penalize the model's own hallucination patterns; RLHF and RLAIF require costly negative-sample construction and struggle to capture the model's latent hallucination directions. CAPL obtains highly targeted positives and negatives from the same model under two masks, sidestepping external annotation cost.
vs single-image hallucination mitigation (contrastive decoding, alignment training, etc.): those target object-existence style hallucinations and address a different failure form from cross-image mismatch, so transferring them to multi-image settings is of limited use — which is exactly the paper's starting point for arguing that multi-image hallucination needs explicit cross-image modeling.

Rating¶

Novelty: ⭐⭐⭐⭐ Attributing multi-image hallucination to the cross-image one-directionality of the causal mask and building preference pairs from the two opposite states of the same mask is a clean and uncommon angle; DPO and attention-mask modification themselves are combinations of existing components.
Experimental Thoroughness: ⭐⭐⭐⭐ Three backbones, three families of benchmarks (hallucination / general / single-image), and both component and negative-sample ablations are fairly complete; the positive-sample refinement process, long multi-image contexts, and the subtask header consistency are under-reported.
Writing Quality: ⭐⭐⭐⭐ The motivation chain (one-way information flow → positional bias → language priors) is clear and the figures map well to the text; corrupted formula typesetting and the slight Table 1 / Table 2 discrepancy are deductions.
Value: ⭐⭐⭐⭐ The method is lightweight (mask change plus LoRA), transfers across architectures, and does not sacrifice single-image ability; the "truncated attention as a negative-sample generator" trick can be borrowed directly by multi-image and multimodal reasoning work.