Revealing Multi-View Hallucination in Large Vision-Language Models¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: Multimodal VLM
Keywords: Multimodal Large Language Models, Multi-View Hallucination, Contrastive Decoding, Visual Interference, Attention Perturbation
TL;DR¶
To tackle "Multi-View Hallucination" (MVH) where large vision-language models (LVLMs) mistake attributes across concurrent instances or camera viewpoints in multi-view scenarios, this paper introduces MVH-Bench (a rigorous 4.8k QA benchmark) and proposes Reference Shift Contrastive Decoding (RSCD)โa training-free inference strategy that suppresses visual distraction by counteracting negative logits induced by masked intermediate text-to-text attention.
Background & Motivation¶
Large vision-language models (LVLMs) are increasingly deployed as autonomous visual assistants in embodied AI, digital twins, multi-camera surveillance, and egocentric assistive platforms. In these real-world settings, models are routinely supplied with multi-view visual inputsโsuch as synchronized first-person (egocentric) and third-person (exocentric) video streams, or multi-angle indoor and outdoor camera feeds. While multi-view inputs provide comprehensive geometric and context-rich cues that cannot be captured by a single image, they also drastically escalate visual complexity. Existing multi-view LVLM literature has predominantly focused on unifying viewpoint embeddings into a shared representation space or aggregating scene-level spatial features, assuming that richer vision leads directly to superior reasoning.
However, empirical examination reveals that current LVLMs suffer from an insidious and pervasive failure mode that the authors term Multi-View Hallucination (MVH). Unlike conventional object hallucination, which primarily stems from strong linguistic priors and world knowledge (e.g., stereotypically predicting that a banana is yellow despite contrary visual evidence), MVH arises because the model is seduced by genuine yet non-target visual evidence in the multi-view input. For instance, when queried about what an individual in a white shirt is doing, the model may conflate that person with a co-occurring individual in a black shirt and attribute the latter's action (holding a watering can) to the former; or, when asked about spatial orientation in View 1, the model inadvertently grounds its answer on the perspective of View 2. Under multi-view distraction, conventional remedies that blindly steer more attention toward visual tokens backfire by simply amplifying the interference from non-target visual cues.
Deeper mechanistic probing shows that MVH is intimately tied to the formation of query context within the LVLM's Transformer decoder. When inter-text token communication is hindered or incomplete, the model fails to construct a holistic representation of the query; instead, it falls back onto shallow, fragmented phrase matching and binds to the wrong visual anchor. Consequently, mitigating multi-view hallucination demands both a diagnostic benchmark designed to prevent spurious shortcut heuristics and a lightweight decoding intervention that penalizes partial-query visual grounding without prohibitive model fine-tuning. Core idea: This paper establishes the MVH-Bench benchmark with strict pair and quadruplet metrics to diagnose cross-instance and cross-view interference, and introduces Reference Shift Contrastive Decoding (RSCD), which disrupts text-to-text attention in intermediate layers to generate negative logits reflecting incomplete query understanding and subtracts them during decoding to suppress multi-view visual interference.
Method¶
Overall Architecture¶
The framework comprises two complementary contributions: MVH-Bench, a rigorously filtered evaluation benchmark tailored to decouple and expose multi-view grounding errors, and Reference Shift Contrastive Decoding (RSCD), an inference-time decoding algorithm that requires neither training nor architectural modification. MVH-Bench extracts instance-descriptor tuples from synchronized multi-view datasets (Ego-Exo4D and LEMMA) across action, object, numerical, and spatial subcategories, constructing paired binary and multiple-choice questions with strict metric protocols (\(p\text{-Acc}\) and \(q\text{-Acc}\)). Drawing on the insight that blocking intermediate text-to-text attention predictably shifts the model's focus toward non-target visual distractors, RSCD dynamically generates negative logits via a reference shift mask and subtracts this misleading bias during auto-regressive generation.
The end-to-end pipeline is illustrated below:
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multi-View Image Inputs & Query Prompt"] --> B["MVH-Bench Diagnostic Evaluation<br/>Cross-Instance & Cross-View Paired Benchmarks"]
B --> C["Layer-Wise Probing Analysis<br/>Sliding-Window Mask Identifies Sensitive Layers"]
C --> D["Reference Shift Mask Generation<br/>Mask Top-P Text-to-Text Attention at Intermediate Layers"]
D --> E["Dual-Path Forward Pass<br/>Original Base Logits & Reference Shift Negative Logits"]
E --> F["Adaptive Plausibility Filtering & Contrastive Decoding Output"]
Key Designs¶
1. MVH-Bench Diagnostic Protocol: Eliminating Shortcut Biases via Strict Pair and Quadruplet Metrics To systematically differentiate the two fundamental axes of multi-view failureโcross-instance hallucination (conflating distinct entities) and cross-view hallucination (confusing identical entities across differing viewpoints)โMVH-Bench curates 4,800 question-answer pairs spanning four representative subcategories: action, object, numerical, and spatial. Because standard binary questions are notoriously susceptible to a "Yes" bias (where models answer "Yes" simply because an queried object appears somewhere in the multi-view scene), the benchmark pairs every question with an adversarial counterpart. For binary questions, the evaluation computes pair accuracy (\(p\text{-Acc}\)), which requires correct answers to both the "Yes" and "No" questions, and quadruplet accuracy (\(q\text{-Acc}\)), which demands simultaneous correctness across all four related queries derived from the same image pair: $\(q\text{-Acc} = \frac{1}{N} \sum_{n=1}^N \prod_{x, y \in \{i, j\}} \mathbb{I}\left(m(q_{xy}^n) = a_{xy}^n\right)\)$ For multiple-choice questions, options are randomized and evaluated via paired accuracy along with the Adversarial Error Ratio (AER), ensuring that models cannot achieve high scores through single-view shortcuts or blind affirmative guessing.
2. Intermediate Layer Identification and Reference Shift Mechanism: Uncovering the Locus of Multi-View Contextual Grounding In LVLMs, visual and textual tokens are concatenated into a joint sequence processed by a causal Transformer decoder. To isolate how query context directs visual attention across multiple views, the authors employ a sliding-window text-to-text attention blocking mask \(M_{t2t}\). By evaluating reference accuracy on multi-image captioning and multi-hop reasoning (HotpotQA), a striking U-shaped degradation pattern emerges: models exhibit a severe dip in visual grounding accuracy specifically when text-to-text attention is severed in intermediate layers \(\Lambda^\star\) (e.g., layers 13โ20 in LLaVA-OneVision-7B and layers 12โ20 in Qwen2.5-VL-7B). This reveals that intermediate layers are the critical computational locus where the full semantics of the query are synthesized to steer visual grounding. When text tokens are isolated at this stage, the model reverts to superficial keyword matching, predictably drifting toward salient yet non-target visual regions across alternative viewpoints.
3. Reference Shift Contrastive Decoding (RSCD): Direct Logit Counteraction with Full KV Cache Reuse Capitalizing on the discovery that incomplete query context induces predictable visual drift, RSCD generates an intentional negative distribution directly in logit space. For each designated intermediate layer \(l \in \Lambda^\star\), RSCD constructs a reference shift mask (RSM) \(M_{rs}\) that masks the top \(\rho\%\) of attention weights from each text token \(i \in \mathcal{T}\) toward all other text tokens: $\(M_{rs, i, j} = \begin{cases} -\infty, & i \in \mathcal{T} \text{ and } j \in \mathcal{J}_i \\ 0, & \text{otherwise} \end{cases}\)$ where \(\mathcal{J}_i = \{j \mid A_{i, j} \in \text{TopP}(A_{i, \mathcal{T}}, \rho)\}\). Performing a forward pass with \(M_{rs}\) yields a negative distribution \(\text{logit}(y_{n+1} \mid y_{\le n}, M_{rs})\) that encapsulates the model's distracted, partially grounded tendencies. The final generation distribution amplifies the contrast against this negative drift: $\(\text{logit}_{rscd} = (1 + \alpha) \text{logit}(y_{n+1} \mid y_{\le n}) - \alpha \text{logit}(y_{n+1} \mid y_{\le n}, M_{rs})\)$ subject to adaptive plausibility filtering to truncate low-probability noise tokens. Crucially, because \(M_{rs}\) modifies only text-to-text attention without altering image inputs or image token representations, the entire multi-view image Key-Value cache (KV Cache) is preserved intact, resulting in near-zero computational overhead.
Benchmark & Evaluation Protocol¶
MVH-Bench comprises 4,800 QA pairs across 1,274 multi-view images extracted from Ego-Exo4D (66%) and LEMMA (34%). The dataset captures diverse viewpoint combinations (35% FPV&TPV, 34% TPV&TPV, and 31% FPV&FPV). Evaluated models include 5 closed-source commercial APIs (GPT-4o, GPT-4o mini, Gemini 2.5 Flash, Gemini 2.0 Flash, Claude Sonnet 4.5) and 10 state-of-the-art open-source LVLMs. Decoding comparisons focus on LLaVA-OneVision-7B and Qwen2.5-VL-7B against leading baselines (VCD, ICD, AvisC), with additional cross-architecture verification on Qwen3-VL-8B, InternVL3.5-14B, and Llama-3.2-11B-Vision-Instruct.
Key Experimental Results¶
Main Results¶
The performance of leading closed-source and open-source models, as well as contrastive decoding interventions on MVH-Bench, is summarized below (MVH Score denotes the sum of all accuracy metrics, excluding YER and AER):
| Model / Method | Cross-Instance \(p\text{-Acc}\) / \(q\text{-Acc}\) | Cross-View \(p\text{-Acc}\) / \(q\text{-Acc}\) | Cross-Instance M.C. \(p\text{-Acc}\) | Cross-View M.C. \(p\text{-Acc}\) | Overall MVH Score |
|---|---|---|---|---|---|
| Closed-Source LVLMs | |||||
| GPT-4o | 52.41 / 32.04 | 46.48 / 23.43 | 36.11 | 32.31 | 478.32 ยฑ 0.32 |
| Gemini 2.5 Flash | 39.68 / 14.91 | 51.16 / 28.06 | 38.15 | 48.89 | 486.64 ยฑ 2.90 |
| Claude Sonnet 4.5 | 36.94 / 13.61 | 49.35 / 23.61 | 36.02 | 49.42 | 471.68 ยฑ 4.05 |
| LLaVA-OneVision-7B | |||||
| Base model | 32.36 / 10.93 | 27.17 / 4.26 | 36.29 | 29.72 | 373.49 ยฑ 6.16 |
| + VCD (CVPR 2024) | 32.59 / 11.94 | 29.31 / 2.59 | 43.98 | 28.43 | 394.42 ยฑ 1.34 |
| + ICD (ACL 2024) | 33.19 / 11.30 | 28.47 / 3.61 | 44.81 | 27.87 | 393.20 ยฑ 5.95 |
| + AvisC (ACL 2025) | 29.31 / 10.28 | 26.43 / 2.04 | 44.35 | 28.61 | 384.90 ยฑ 2.39 |
| + RSCD (Ours) | 38.75 / 14.35 | 33.06 / 5.65 | 43.61 | 27.96 | 410.55 ยฑ 6.41 |
| Qwen2.5-VL-7B | |||||
| Base model | 41.30 / 16.29 | 47.45 / 21.95 | 43.52 | 50.56 | 489.89 ยฑ 9.85 |
| + VCD (CVPR 2024) | 38.52 / 13.52 | 45.51 / 19.63 | 44.72 | 46.48 | 475.64 ยฑ 47.62 |
| + ICD (ACL 2024) | 32.13 / 10.46 | 39.31 / 16.39 | 42.04 | 47.41 | 450.42 ยฑ 44.95 |
| + AvisC (ACL 2025) | 36.43 / 11.39 | 38.66 / 16.02 | 45.93 | 45.65 | 458.51 ยฑ 33.08 |
| + RSCD (Ours) | 45.19 / 17.69 | 56.02 / 31.02 | 53.70 | 58.33 | 553.33 ยฑ 2.73 |
Ablation Study¶
Ablation experiments exploring layer range selection, attention masking ratios, inference runtime, and viewpoint scaling are detailed below:
| Dimension | Configuration | Key Metric Performance | Note & Insight |
|---|---|---|---|
| Layer Range \(\Lambda^\star\) | Layers 0โ12 (Early) | MVH Score drops to <340 | Disrupting low layers cripples basic syntax and token representation |
| (LLaVA-OV-7B) | Layers 21โ27 (Late) | MVH Score ~375 | Late layer representations are saturated; negative logit divergence is too weak |
| Layers 13โ20 (\(\Lambda^\star\), Intermediate) | MVH Score 410.55 | Optimal window where query context synthesis is selectively disturbed | |
| Layers 12โ21 / 14โ19 | MVH Score 405โ411 | Performance remains stable; insensitive to precise layer boundary offsets | |
| Masking Ratio \(\rho\) | \(\rho = 0.6 \to 0.9\) | MVH Score monotonically climbs | Stronger text perturbation induces sharper negative drift away from true target |
| \(\rho = 1.0\) (Complete masking) | Dramatic score collapse | Total masking degrades negative logits into arbitrary noise, harming contrast | |
| Inference Cost | Base model | 0.81 sec / query | Baseline auto-regressive decoding time |
| (Multi-view inputs) | VCD (Visual noise) | 1.38 sec / query | Incurs +70% overhead due to re-encoding corrupted visual tokens |
| AvisC (Attentional calibration) | 1.37 sec / query | Re-computes image token attention and representations | |
| RSCD (Ours) | 0.89 sec / query | Only 0.08s overhead (+9.8%); reuses cached image Key-Value states directly | |
| Viewpoint Scaling | Two-View (Default) | Base: 345.23 / RSCD: 380.99 | Standard pairwise setting |
| (Max-View, avg 4.3 views) | Max-View (All views) | Base: 317.10 / RSCD: 380.99 | MVH intensifies with more views; RSCD maintains a +63.89 lead over base |
Key Findings¶
- Conventional Hallucination Mitigations Deteriorate in Multi-View Contexts: On Qwen2.5-VL-7B, widely adopted methods like VCD and ICD degrade performance below the unaugmented base model (VCD drops to 475.64, ICD drops to 450.42, vs. Base 489.89). Injecting visual noise or textual disturbance instructions disrupts the fragile inter-view spatial correspondences, proving that multi-view hallucination requires targeted query-level contrast rather than generic visual corruption.
- Divergence between Standard and Multi-View Benchmarks: Strong performance on POPE or CHAIR fails to predict multi-view grounding competence. For example, InternVL3-8B ranks highest on POPE (89.63%) and achieves the lowest CHAIRi (8.80), yet scores poorly on MVH-Bench (379.39). Conversely, Qwen2.5-VL-7B achieves the highest base MVH Score (489.89) despite moderate POPE rankings, establishing MVH as an independent orthogonal evaluation dimension.
- Broad Zero-Shot Generalization: Without hyperparameter recalibration, RSCD delivers positive transfers across standard multimodal benchmarks on LLaVA-OV-7B, enhancing MMBench (85.22% to 85.63%), SEED-Bench (68.50% to 70.00%), and POPE (87.38% to 87.98%).
Highlights & Insights¶
- Inverting Incomplete Semantics into a Self-Supervised Negative Anchor: Rather than relying on external heuristic prompts or artificial image noise to induce negative logits, RSCD exploits the intrinsic architecture of Transformersโdisrupting intermediate text-to-text attention to naturally generate logits biased toward non-target visual anchors, which can then be subtracted away.
- Zero-Footprint KV Cache Compatibility: Because the reference shift mask exclusively intervenes in the text-to-text attention sub-matrix during decoder intermediate layers, the cached Key-Value tensors of all multi-view image patches remain 100% valid and untouched. This preserves near-baseline inference latency (0.89s vs 0.81s), making it uniquely viable for multi-camera robotics.
- Ultra-Lightweight Layer Calibration: Finding the optimal layer interval \(\Lambda^\star\) requires probing as few as 3 to 30 sample image pairs on a simple captioning task, with wide empirical tolerance across adjacent layer boundaries.
Limitations & Future Work¶
- Evaluation Centered on Paired Views: While the Max-View experiment demonstrates efficacy across an average of 4.3 viewpoints, the primary MVH-Bench benchmark is structured around two-view pairs, leaving ultra-dense setups (e.g., 360-degree autonomous driving rings or 50+ camera array captures) for future benchmark iterations.
- Model-Specific Layer Profiling: Although the U-shaped degradation pattern is universal across model families, the exact numerical layer index \(\Lambda^\star\) varies with model depth and architectural topology (e.g., layers 13โ20 in 28-layer models vs. layers 13โ29 in 36-layer models), requiring a brief one-off profiling step for novel architectures.
- Future Directions: Integrating geometric priors (such as camera extrinsic matrices and epipolar constraints) directly into the reference shift attention mask, and distilling the contrastive decoding benefits into model weights during post-training alignment.
Related Work & Insights¶
- vs VCD (Leng et al., CVPR 2024): VCD applies Gaussian noise to input images to counteract linguistic prior hallucinations. In multi-view scenarios, degrading image inputs destroys subtle viewpoint distinctions and lowers accuracy; RSCD keeps visual fidelity pristine and instead perturbs text context.
- vs AvisC (Woo et al., ACL 2025): AvisC constructs negative logits by conditioning on low-information image tokens. In MVH, however, distractors are prominent, high-information visual entities residing in alternate views; RSCD directly tackles this semantic binding failure, outperforming AvisC by 25.7 points on LLaVA-OV-7B and 94.8 points on Qwen2.5-VL-7B.
- vs Multi-View Scene Representation Models (3D-LLM, Ego-Exo4D Baselines): Earlier works prioritize learning invariant features across viewpoints but overlook inference-time cross-view attentional drift; this paper establishes the diagnostic benchmark and inference-time mitigation necessary for reliable multi-view grounding.
Rating¶
- Novelty: โญโญโญโญโญ [Pioneering identification of multi-view hallucination and an elegant contrastive decoding strategy derived from intermediate text-to-text attention disruption]
- Experimental Thoroughness: โญโญโญโญโญ [Comprehensive 4.8k-sample benchmark evaluated on 23 models with strict pair/quadruplet metrics, runtime benchmarking, and cross-task verification]
- Writing Quality: โญโญโญโญโญ [Clear problem formulation, meticulous mechanistic exposition, and strong self-consistency between hypotheses, visualizations, and quantitative tables]
- Value: โญโญโญโญโญ [Training-free, KV Cache-preserving, highly effective, and directly deployable across embodied AI and multi-view vision-language systems]