Skip to content

PVFH-Bench: Benchmarking Premise-Induced Visual Fixation Hallucinations in Vision-Language Models

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/NEUIR/PVFH-Bench
Area: Hallucination Detection
Keywords: Hallucination Detection, Vision-Language Models, Visual Fixation Hallucination, False Premise, Chain-of-Thought Reasoning

TL;DR

This paper identifies a critical compounded failure mode termed Premise-Induced Visual Fixation Hallucination (PVFH) along with a diagnostic benchmark PVFH-Bench, revealing that contemporary LVLMs anchor onto visually confusable regions under misleading textual premises and that explicit chain-of-thought reasoning exacerbates rather than mitigates this hallucination trap.

Background & Motivation

Large Vision-Language Models (LVLMs) such as Qwen-VL, InternVL, and Gemma have achieved remarkable progress across diverse multimodal perception and reasoning tasks. However, object hallucination remains a pervasive challenge that undermines their trustworthiness in critical real-world deployments such as embodied navigation, autonomous driving, and clinical decision support. Mainstream hallucination benchmarks, including POPE and MME, evaluate object-level faithfulness predominantly through binary existence polling or open caption checks. Such formulations fail to isolate how modern reasoning-oriented LVLMs interact with leading conversational contexts, especially when false textual premises subtly distort visual perception.

Classical cognitive science studies on eyewitness testimony by Elizabeth Loftus established that presuppositions embedded within queries can inject deceptive misinformation into human memory, compelling observers to falsely confirm the presence of nonexistent items. When applied to multimodal foundation models, this perceptual vulnerability manifests acutely: if an image contains an object visually similar to an absent queried target (such as an actual newspaper when asked about a magazine), a model in a vanilla setting can readily reject the absent magazine. Yet, once a question embeds a false premise concerning the missing object's spatial relation (such as inquiring whether the magazine lies in the upper or lower half), the model anchors its visual attention onto the confusable newspaper region and constructs a detailed, premise-consistent answer about the nonexistent entity.

The primary obstacle in existing benchmarks is their inability to disentangle basic recognition failure from visual fixation and linguistic premise acceptance. Core idea: construct PVFH-Bench with visually confusable true-false object pairs across four controlled diagnostic tiers, systematically tracing how misleading spatial premises coerce models into rationalizing ambiguous visual evidence into fluent hallucinations, and demonstrating that explicit reasoning expands this vulnerability.

Method

Overall Architecture

PVFH-Bench introduces a paired diagnostic framework that isolates Premise-Induced Visual Fixation Hallucination through controlled contrastive setups across 258 MSCOCO 2014 validation images, yielding 1,032 balanced evaluation instances. Each instance pairs a visible true object with an absent, visually confusable counterpart and evaluates model responses through an automated LLM-as-a-judge protocol verifying semantic premise rejection.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image (MSCOCO 2014)"] --> B["Identify Constrained Object Pairs<br/>True Visible Object vs Confusable Absent Object"]
    B --> C["Instantiate 8 Spatial Relation Templates<br/>Object-Relative and Image-Relative Relations"]
    C --> D["Construct Four Paired Diagnostic Subsets<br/>Vanilla, Visual Fixation, False Premise, Core PVFH"]
    D --> E["Semantic LLM-as-a-Judge Evaluation<br/>Discriminative Premise Rejection & Trace Alignment"]

Key Designs

1. Constrained Extraction of Mutually Exclusive Confusable Object Pairs

Standard negative object generation in prior benchmarks relies on statistical co-occurrence or random negative sampling, which lacks fine-grained visual confusability with localized scene elements. PVFH-Bench employs Qwen3.5-Plus with strict prompt constraints to identify true-false object pairs satisfying four criteria: visual unambiguity (the true object is clearly visible and uncontroversial), strict absence (the false category has zero valid instances in the entire image), mutual exclusivity (the true instance cannot truthfully belong to the false category, precluding synonyms or hyponyms like substituting wine bottle with bottle), and maximum visual confusability (retaining the pair with the closest appearance, e.g., newspaper vs. magazine, lemon vs. lime, or tennis ball vs. baseball). This setup guarantees that subsequent hallucinations arise purely from visual-linguistic fixation rather than annotation artifacts.

2. Four-Tier Decoupled Diagnostic Evaluation Hierarchy

To track how models progress from normal visual recognition to compounded hallucination, PVFH-Bench fixes the identical image and object pair across four systematically stratified query settings: - Vanilla (Raw Query): Queries the existence of the visible true object (e.g., "Is the newspaper in the image?"), where the expected answer is Yes, serving as a baseline for visual recognizability. - Visual Fixation Query: Queries whether the absent confusable counterpart is present (e.g., "Is the magazine in the image?"), where the correct answer is No, testing vulnerability to visual similarity in isolation. - False-Premise Diagnostic Query: Queries a spatial relation concerning a randomly chosen absent object that bears no visual resemblance to visible entities, isolating pure linguistic premise pressure from visual confusions. - Premise-Induced Visual Fixation Query (Core Setting): Queries a spatial relation presupposing the existence of the absent confusable object (e.g., "Is the magazine mainly in the upper half or the lower half?"), where the only correct behavior is premise rejection (declaring the magazine is absent).

3. Spatial Relation Template Pool & Premise Rejection Protocol

The benchmark establishes eight spatial relation templates spanning object-relative relationships (left/right, above/below, closer/farther, larger/smaller, relative horizontal positioning) and image-relative coordinates (left/right side, upper/lower half, foreground/background). For evaluation, open-ended textual generations are graded via Qwen3.5-Flash (and cross-verified with GPT-5.4-mini) using strict criteria: answers that simply output a binary selection or choose a spatial position are marked incorrect, whereas responses explicitly stating that the target object is absent or declining the choice based on premise invalidity are graded as correct.

Loss & Training

As PVFH-Bench is an evaluation benchmark and diagnostic study, models were evaluated zero-shot at a decoding temperature of 0.1 to minimize sampling variance. For models supporting controllable reasoning budgets (e.g., the Seed-2.0 series), reasoning effort was modulated across minimal, low, medium, and high settings to dissect the mechanistic effect of thinking depth on hallucination rates.

Key Experimental Results

Main Results

PVFH-Bench evaluates 10 state-of-the-art LVLMs spanning 4 model families (Qwen-VL, Gemma, LLaMA, and Seed) across open-source and proprietary platforms in standard and reasoning modes.

Model Vanilla (Actual Object) Visual Fixation (Confusable) False Premise (Absent Object) Premise-Induced VF (Core Setting) Overall Avg. Acc.
Qwen3.5-Plus 98.84% 79.84% 93.80% 74.81% (-5.04) 86.82%
Qwen3.5-Plus (+Reasoning) 100.00% 78.29% 63.18% 36.82% (-41.47) 69.57%
Seed-2.0-Lite 97.29% 82.56% 82.95% 68.60% (-13.95) 82.85%
Seed-2.0-Lite (+Reasoning) 96.12% 82.95% 77.13% 52.33% (-30.62) 77.13%
Seed-2.0-Mini 97.67% 79.46% 76.74% 44.96% (-34.50) 74.71%
Seed-2.0-Mini (+Reasoning) 98.06% 77.91% 62.40% 27.13% (-50.78) 66.38%
Qwen3-VL-32B 97.67% 74.03% 88.37% 58.14% (-15.89) 79.55%
Qwen3-VL-32B (+Reasoning) 98.06% 82.95% 82.17% 44.19% (-38.76) 76.84%
Qwen3-VL-8B 98.45% 80.62% 81.40% 58.14% (-22.48) 79.65%
Qwen3-VL-8B (+Reasoning) 96.12% 84.11% 87.21% 40.31% (-43.80) 76.94%
Gemma-3-27B-IT 98.84% 67.05% 60.08% 28.29% (-38.76) 63.57%
Gemma-3-12B-IT 94.96% 75.58% 44.19% 18.99% (-56.59) 58.43%
Gemma-3-4B-IT 96.51% 62.79% 19.38% 1.94% (-60.85) 45.16%
LLaMA-4-Scout 93.80% 82.17% 86.05% 51.16% (-31.01) 78.29%
LLaMA-3.2-11B 98.06% 76.36% 86.43% 46.90% (-29.46) 76.94%
All-Model Average 97.36% 77.78% 72.76% 43.51% (-34.26) 72.86%

Ablation Study

Controlled variations isolating false-premise confusability, cross-judge verification, and reasoning effort demonstrate consistent degradation patterns.

Setting / Variant Evaluator / Judge Vanilla Visual Fixation False Premise Core PVFH
Main Leaderboard Mean (10 Base) Qwen3.5-Flash 97.36% 77.78% 72.76% 43.51%
Independent Audit Leaderboard GPT-5.4-mini 97.48% 76.24% 70.43% 42.44%
Seed-2.0-Lite (Minimal Effort) Qwen3.5-Flash 96.51% 84.88% 84.50% 66.28%
Seed-2.0-Lite (High Effort) Qwen3.5-Flash 96.12% 82.95% 77.13% 52.33%
Seed-2.0-Mini (Minimal Effort) Qwen3.5-Flash 98.06% 78.29% 77.13% 42.25%
Seed-2.0-Mini (High Effort) Qwen3.5-Flash 98.06% 77.91% 62.40% 27.13%
5-Model Base Subset Mean Qwen3.5-Flash 97.98% 79.30% 84.65% 60.93%
5-Model Reasoning Subset Mean Qwen3.5-Flash 98.06% 81.24% 74.42% 40.16%

Key Findings

  • Compounding Penalty of Premise and Visual Fixation: On Vanilla queries, all models exceed 90% (average 97.36%), confirming high baseline visual recognition. Performance drops to 77.78% under isolated visual confusability and 72.76% under random false premises. When both merge in the Core PVFH setting, average accuracy collapses to 43.51% (a 34.26-point deficit relative to visual fixation). Replacing an ordinary absent target with a confusable one under false premises costs an immediate 29.25 accuracy points.
  • Explicit Reasoning Exacerbates Hallucination: Across the 5 models evaluated with thinking modes, average Core PVFH accuracy plummets from 60.93% down to 40.16%. Qwen3.5-Plus plunges from 74.81% to 36.82% (-41.47 points), and Seed-2.0-Mini drops from 44.96% to 27.13% (-50.78 points relative to visual fixation). Error analyses reveal that over 66.7% of PVFH failures occur on samples that models answered correctly in Visual Fixation Query; reasoning chains seize upon the false premise and rationalize the confusable true object as proof of the nonexistent entity.
  • Scaling Thinking Depth Fails to Resolve Premise Traps: Increasing reasoning effort from minimal to high in Seed models does not restore faithfulness (Seed-2.0-Lite drops from 66.28% to 52.33%, Seed-2.0-Mini drops from 42.25% to 27.13%). More thinking tokens merely provide more headroom for models to follow textual bait, search for visual surrogates, and fabricate spatial relations.

Highlights & Insights

  • Formulates Premise-Induced Visual Fixation Hallucination grounded in classic cognitive psychological principles, presenting a compelling diagnosis of how conversational premises corrupt multimodal perception.
  • The 4-tier paired evaluation decouples visual recognizability, similarity confusion, and premise acceptance on identical images, establishing a diagnostic benchmark architecture.
  • Challenges the pervasive assumption that test-time thinking universally improves faithfulness: explicit chain-of-thought without premise verification mechanisms acts as an accelerator for confabulatory rationalization.

Limitations & Future Work

  • Diagnostic Focus Without Active Mitigation: The work focuses on exposing and benchmarking PVFH rather than introducing training-time or inference-time mitigation techniques.
  • Bounded Premise Forms: Queries are constrained to 8 spatial relation templates; realistic user interactions may introduce conversational false premises in far more diverse forms.
  • Domain Confinement to MSCOCO: Sourced exclusively from general natural scenes; evaluating PVFH in high-stakes domains such as surgical robotics, medical diagnostics, or aerial monitoring remains open.
  • vs POPE (Li et al., 2023b): POPE tests binary presence with adversarial negative samples but remains isolated to existence polling; PVFH-Bench extends to spatial relations and demonstrates a dramatic performance collapse on the identical model families (e.g., Qwen3-VL drops from ~89% on POPE to 58.14% on PVFH).
  • vs HaloQuest (Wang et al., 2024): HaloQuest evaluates false-premise VQA but lacks fine-grained diagnostic decoupling; PVFH-Bench isolates the exact transition from correct visual rejection to premise-induced fixation on matched image-object pairs.

Rating

  • Novelty: โญโญโญโญโญ Elegant cognitive science inspiration yielding an acute characterization of reasoning vulnerabilities.
  • Experimental Thoroughness: โญโญโญโญโญ Comprehensive 10-model evaluation spanning open/proprietary systems, cross-judge audits, attention tracking, and reasoning effort ablations.
  • Writing Quality: โญโญโญโญโญ Rigorous conceptual structure, lucid methodology, and insightful error decomposition.
  • Value: โญโญโญโญโญ Delivers an essential benchmark and benchmark insights for training trustworthy reasoning-oriented vision-language models.