No Place to Hide: Benchmarking Video Hallucination with Background-Controlled Pairs¶

Conference: ECCV 2026
arXiv: 2606.31933
Code: Public on project page (the paper states "code and data are available at the project page", no specific GitHub link provided)
Area: Video Understanding / Multimodal Hallucination Evaluation
Keywords: Video Hallucination, Adversarial Video Pairs, Background-Controlled, Benchmark, Generative Data Synthesis

TL;DR¶

This paper proposes VidPair-Halluc, a video hallucination benchmark that utilizes adversarial video pairs with "highly similar backgrounds but significantly different foreground semantics" to decouple model errors from background variations and cleanly attribute them to foreground hallucinations. The accompanying PairFlow three-stage generation pipeline leverages T2I and video generation models to automatically synthesize 1K high-quality video pairs and 11K spatiotemporal QA pairs. Evaluations indicate that mainstream large video models generally suffer from a severe degradation in performance under this controlled setting.

Background & Motivation¶

Video hallucination refers to scenarios where Large Video Models (LVMs) confidently generate content inconsistent with visual evidence. The root cause often lies in video encoders being significantly weaker than large-scale pre-trained LLMs, with weak visual signals leading to "overconfident but incorrect" outputs. To diagnose this issue, evaluation benchmarks have continuously evolved. Early benchmarks like Vript-HAL and EventHallusion ask spatiotemporal questions on a single video to examine basic understanding of object relationships, actions, and event sequences. VideoHallucer further introduces adversarial text on single videos (paired binary questions with one true and one false, misleading distractors) to target language priors. Later, HallusionBench and VidHalluc began utilizing paired visual inputs to test fine-grained semantic alignment. However, these works share a common weakness—they either rely on text perturbations to mislead the LLMs or use CLIP/DINO feature retrieval to find video pairs that are "visually similar but still exhibit overall background variations", never explicitly controlling background consistency.

This introduces a fatal attribution confusion: when a model answers incorrectly, one cannot determine whether it genuinely misunderstood the foreground semantics (true hallucination) or was merely misled by overall shifts in the background and camera motion. The background acts as "query-irrelevant" visual context, whereas the foreground serves as the "query-relevant" crucial evidence. Only by locking down the background as much as possible while modifying only the foreground can error attribution be cleanly isolated to foreground semantic misinterpretation. More troublesome is that constructing such adversarial pairs with "nearly identical backgrounds but different key foregrounds" from real videos is extremely costly. Real videos possess fixed storylines and complex spatiotemporal dependencies; frame-by-frame editing or frame replacement often breaks background/entity consistency, and video editing techniques lack stability and fidelity in complex scenes. Generating videos after editing keyframes is also constrained by scarce frame-level captions and high annotation costs. While the image VQA domain has long leveraged inpainting and automated pipelines to scale the production of adversarial samples, the video domain has been stuck at this scalability bottleneck, which is precisely the technical barrier of "why nobody has done this systematically before".

The key insight of this paper is: since Text-to-Image (T2I) and video generation/editing models have become sufficiently powerful, we can bypass real videos and use a synthetic pipeline that "writes a story first, then generates clips, and finally assembles them into pairs" to produce background-consistent, foreground-divergent adversarial video pairs with minimal human intervention. Core Idea: Turn "hallucination attribution" into a controlled experiment—isolating the sources of video hallucination completely from background drift by using synthetic means to freeze query-irrelevant backgrounds and only modify the queried foreground values or temporal orders, leaving models with "No Place to Hide" and yielding a cleaner diagnostic signal than real video mining.

Method¶

Overall Architecture¶

The core problem VidPair-Halluc aims to solve is "how to synthesize background-controlled adversarial video pairs at scale and strictly evaluate fine-grained LVM video hallucinations based on them." The entire system consists of two interlocking components: the data generation engine PairFlow, and a three-tier adversarial evaluation protocol defined on its products (video-level / temporal-level / spatial-level).

PairFlow is a three-stage serial pipeline. Stage 1 Story Orchestration: For a target semantic aspect (e.g., action, color, count), GPT-4.1 automatically writes a three-act micro-story containing placeholder {values}, and prepares several candidate values for the placeholders that are "substantially different from each other yet seamlessly fit into the plot" (e.g., "choose" vs. "fold"). Stage 2 Video Clip Generation: First, a T2I model (FLUX) generates the original keyframe based on the script, and then an image editing model (SeedEdit) refines the foreground while preserving the background to yield both the "original" and "adversarial" versions of the keyframe. Subsequently, a video generation model (Wan2.1/2.2-14B) generates videos segment-by-segment, conditioned on the "last frame of the previous clip + current segment script" to ensure coherence across clips. Stage 3 Video Assembly: Clips are concatenated to form positive video samples, and an adversarial function \(\phi(\cdot)\) is applied to impose temporal inversion, clip replacement, occlusion, etc., to produce negative video samples, creating adversarial pairs at three granularities: video-level, temporal-level, and spatial-level. Finally, each approved video pair is mapped to QA templates covering 10 categories of spatiotemporal hallucinations to generate binary QA, multiple-choice questions (MCQ), and open-ended descriptions, followed by a four-level human audit.

The final dataset contains 2,000 15-second video clips, 1K adversarial video pairs, and 11,523 QA pairs (4,000 binary + 4,000 MCQ + 3,523 open-ended), covering two major categories of temporal (33%) and spatial (67%) domains across 10 semantic perspectives.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Target Semantic Aspect<br/>Action/Color/Count…"] --> B["Story Orchestration<br/>GPT-4.1 writes 3-act micro-stories<br/>with {values} placeholders + candidate values"]
    B --> C["Video Clip Generation<br/>T2I generates original frame → Image edit<br/>alters foreground for adversarial frame → Segment-by-segment generation"]
    C --> D["Video Assembly φ(·)<br/>Temporal inversion / Clip replacement / Occlusion<br/>creates positive/negative video pairs"]
    D -->|Video-level / Temporal-level / Spatial-level| E["QA Templates<br/>10 Hallucination Classes × Binary/MCQ/Open-ended"]
    E --> F["4-level Human Audit<br/>Script→Clips→Adversarial Pairs→QA"]
    F --> G["VidPair-Halluc<br/>Evaluates 15 LVMs"]

Key Designs¶

1. Background-Controlled, Foreground-Divergent Adversarial Pairs: Isolating the Culprit of Errors

This is the absolute foundation of the benchmark, distinguishing it from all prior works. In previous benchmarks like VidHalluc which use CLIP/DINO to retrieve video pairs, the background, camera motion, and foreground often vary simultaneously, making error attribution impossible when a model fails. This work deliberately freezes the background as much as possible and only alters the "queried foreground semantics", cleanly isolating failure to foreground hallucination. To quantify "how similar the backgrounds actually are", the authors divide adversarial pairs into two categories based on scene dependence and measure them separately: for scene-independent pairs (e.g., changes to objects/actions within the same background), the full frame similarity is computed; for scene-dependent pairs (where the human subject itself acts as part of the background), the consistent main subject is first masked out, and metrics are computed only within the masked area. In non-scene-dependent cases, three metrics—DINOv2, LPIPS, and SSIM—reach DINOv2 0.94, LPIPS 0.15, and SSIM 0.76. In masked scene-dependent cases, SSIM goes as high as 0.90, which, compared to random pairs (DINOv2 of only 0.21), verifies that the background is indeed controlled to extremely high similarity. Since the background is no longer a variable, if a model fails, it can only be due to its failure to comprehend the foreground—this is the literal meaning of "No Place to Hide".

2. PairFlow Generation Pipeline: Bypassing the Scalability Bottleneck of Real Videos via "Story Orchestration + Keyframe Editing + Segment-by-Segment Continuation"

Building adversarial pairs from real videos is prohibitively expensive, while three naive synthetic pipelines have their own severe drawbacks: direct generation of video pairs or frame-by-frame replacement destroys background/entity consistency and introduces abrupt transitions; video editing lacks stability and fidelity in complex scenes; and keyframe-based generation is limited by scarce frame-level captions and high annotation costs. PairFlow cleverly solves these issues together: it parameterizes "one plot, multiple values" with placeholder stories, ensuring candidate values are both significantly different and logically sound; it injects foreground differences via image editing rather than video editing to bypass video editing instability; and it chains clips together via "last-frame continuation" to ensure temporal coherence. The mathematical formulation of segment-by-segment generation is:

\[V_i = f(V_{i-1}^{L}, S_i), \quad i \ge 2\]

where \(f(\cdot)\) is the video generation model, which takes the last frame \(V_{i-1}^{L}\) of the previous clip \(V_{i-1}\) and the current segment script \(S_i\) as input, and outputs the next clip \(V_i\) of equal length. This anchors each segment to the last frame of the preceding one, preserving the background and entities while shifting the foreground values according to the script. The authors also provide empirical proof of the "feasibility of synthesis": as video generators strengthen, the human first-pass rate rises from ~63% to ~83%, and Wan2.2 achieves a pass@1 of ~70–100% across various dimensions, showing that this methodology will become increasingly frictionless with the advancement of generation models.

3. Three Levels of Adversarial Granularity (Video/Temporal/Spatial): Descending from "Global Semantics" to "Clip-level Details"

A single granularity is insufficient to capture the full picture of hallucinations. This work uses the adversarial function \(\phi(\cdot)\) to create three progressively harder levels of difficulty. Video-level replaces or rewrites entire segments to make the global semantics of positive and negative videos completely different, examining whether the model can capture global semantic consistency—this is the most basic discrimination. Temporal-level primarily uses "full sequence reversal" to rearrange clips in reverse order \([V_N, V_{N-1}, \dots, V_1]\). Reversal is chosen over random shuffling because it preserves temporal semantics to the greatest extent and tests temporal logical reasoning in its purest form. Spatial-level introduces local inconsistencies like "object disappearance or scene change" by applying occlusion, replacement, or subtle modifications to specific segments. It uses binary QA to measure sensitivity to spatial details and object-level coherence, and MCQ to require models to rank clip-level descriptions, forcing them to perform more fine-grained spatiotemporal reasoning. These three levels, from coarse to fine, perfectly map to the three dimensions—"global semantics, temporal order, and local foreground"—where models are most prone to hallucinations.

4. 4-Level Human Audit: Preventing "Generators from Grading Themselves" and Ensuring Error Traceability

The greatest risk of synthetic data is that the generator acts as both the athlete and the referee. This work stifles this risk with a robust 4-level "model-assisted + human-double-checked" audit pipeline: trained reviewers first audit the coherence and outcome differentiability of the GPT-4.1 scripts; annotators verify on Label Studio whether generated clips align with the descriptions and meet basic visual quality standards; for adversarial pairs, reviewers keep only those specimens with "highly similar backgrounds and clear foreground semantic differences", filtering out background drift, foreground ambiguity, identity/attribute shifts, temporal inconsistencies, and visible artifacts; finally, for all 11,523 QA pairs, another set of reviewers confirms strict question-video alignment. The critical outcome of the 4-level audit is that "the generator never evaluates its own output," and the preserved model errors can be fully traced back to "deliberately altered foreground values or temporal orders"—making the causal attribution of the evaluation metrics highly credible.

Loss & Training¶

Since this paper introduces an evaluation benchmark and does not train models, there is no loss function. To characterize the model's robustness against adversarial pairs, the evaluation side defines several paired metrics (detailed in the Experiments section): Question Pair Accuracy (qAcc, requiring the model to correctly answer all instances within an adversarial question pair to be considered correct), Video Pair Accuracy (vAcc, requiring the model to correctly answer all questions associated with a video to be considered correct), wAcc (obtained by weighting the two metrics based on sample size), and Yes Percentage Difference (measuring the deviation of the model's "yes" ratio relative to ground truth, where closer to 0 indicates less bias).

Key Experimental Results¶

Main Results¶

Evaluations were conducted on 15 mainstream LVMs (including both open-source and closed-source APIs), with human performance serving as the upper bound. The core findings show that closed-source models are better calibrated, with Gemini-2.5-Pro being the overall strongest, though all models remain far below human level.

Model	Params	Binary wAcc ↑	Binary FP ↓	MCQ F1 ↑	MCQ vAcc ↑	Open-ended Desc ↑
Video-ChatGPT	7B	24.58	33.66	6.10	0.0	27.70
Qwen2.5-VL-Instruct	7B	41.66	48.71	61.50	42.86	45.07
Video-LLaMA2	7B	21.48	43.00	62.91	42.86	40.85
GPT-4o	-	26.97	29.16	59.15	38.10	47.89
GPT-5-mini	-	29.33	19.65	64.82	45.28	49.33
Gemini-2.5-Pro	-	49.15	13.07	67.32	43.36	54.68
Human (Upper Bound)	-	74.32	9.28	89.21	79.66	-

Gemini-2.5-Pro achieves the highest binary wAcc, the lowest FP, and the strongest MCQ and open-ended description scores, while GPT-5-mini scores the highest on vAcc. However, even for the strongest model, the binary wAcc (49.15) still exhibits a massive 25-point gap compared to humans (74.32), and the MCQ vAcc plummets from human-level 79.66 to just around 40, indicating that current LVMs are generally fragile under background-controlled, fine-grained settings. Among open-source models, Qwen2.5-VL scores passably on wAcc but suffers from high FP, exposing calibration deficiencies.

Ablation Study¶

While there is no traditional module ablation table due to the nature of a benchmark paper, the authors provide several convincing comparative analyses.

Comparative Dimension	Key Results	Explanation
Text Pairs vs. Video Pairs Hallucination	Video pairs are significantly harder	Adversarial video pairs with highly similar backgrounds are more prone to eliciting hallucinations than text perturbations (Fig.6).
VidHalluc vs. VidPair-Halluc	Performance of most models drops sharply	Fine-grained foreground differences are much harder to recognize once the background is controlled (Fig.9).
Wan2.1 vs. Wan2.2 Synthesis Quality	First-pass rate: ~63% → ~83%	Stronger generators yield a higher human passage rate for synthesized data, verifying pipeline feasibility (Fig.7b).
Qwen2.5-VL Representation Analysis	Heavy overlap in positive/negative t-SNE	Under strict background control, the model nearly fails to separate "relational polarities", biasing towards background clues rather than foreground evidence (Fig.7a).

Key Findings¶

Background control leaves hallucinations "nowhere to hide": The t-SNE analysis of Qwen2.5-VL reveals that when the background is controlled, positive and negative samples heavily overlap in both video and text representations. This indicates that the learned representations rely heavily on scene context and style, rather than the fine-grained action/object evidence needed for hallucination detection—directly validating the main argument that "backgrounds act as confounding variables".
Low FP is crucial for safety-critical scenarios: The lowest FP of Gemini-2.5-Pro implies fewer "out of thin air" fabrications, which is highly valuable in decision pipelines where false alarms are costly. Meanwhile, a high wAcc represents broad robustness across adversarial video-text pairs rather than isolated successes.
Reasoning models are more stable on binary judgments: Case studies reveal that when tasks are simplified to binary decisions without CoT, the reasoning model ThinkLite-VL exhibits richer attention and more precise localization, distinguishing subtle differences more reliably than Qwen2.5-VL. This contradicts the conventional wisdom that "more thinking leads to more hallucinations."
Frontier models still fail to reach human levels: Supplementary tests in the appendix on next-generation flagships like Gemini-3.1-Pro and GPT-5.4 push the binary wAcc to 60+ (Gemini-3.1-Pro 63.84), closing the gap to humans (74.32) significantly, but still fail to reach parity. Each model displays complementary, rather than strictly superior, capability profiles.

Highlights & Insights¶

Formulating "hallucination attribution" as a "controlled experiment": The most ingenious design is utilizing "background-consistent, foreground-divergent" setups to decouple "background drift" and "foreground misinterpretation" which used to be intertwined. This allows the benchmark, for the first time, to cleanly attribute failures to foreground hallucinations. This "controlled variable" approach is transferable to any multimodal diagnostic task requiring attribution.
Repositioning synthetic data from "training-oriented" to "eval-oriented": While most works utilize generative models for training data, this work explicitly targets evaluations. By freezing non-target backgrounds and altering only the queried values or order, it isolates hallucination sources much more cleanly than scraping real video pairs—a pragmatic repositioning of generative data usage.
The combination of "last-frame continuation + image-edited foreground injection": Instead of relying on unstable video editing, editing keyframes via image editing and continuing them segment-by-segment secures both foreground variations and temporal coherence. This is a highly reusable recipe for adversarial video synthesis.
The "generator does not judge itself" principle in 4-level auditing: Separating the referee from the athlete and making retained errors traceable provides a valuable engineering paradigm for synthetic benchmarks' credibility.

Limitations & Future Work¶

Constraint by the physical priors of video generators: The authors acknowledge that PairFlow is currently constrained by the limited physical priors of existing video generation models, posing a bottleneck for larger scale expansion—if the generator lacks physical realism, the credibility of synthetic adversarial pairs degrades.
Short videos with limited segments: The current release only includes 3 segments of 5 seconds each, limiting total duration to 15 seconds, and mostly explores the "few-segment" scenario. Generating and evaluating longer, multi-segment compositional videos is left for future work, which would align more closely with real-world complexity.
Complementary to, rather than replacing, real video benchmarks: The authors clarify that this benchmark serves as a "controlled setting." While real videos capture natural diversity, controlled pairs isolate query foregrounds; both should be used in tandem—solely relying on controlled pairs would underestimate the hallucinations caused by background variations in real-world scenarios.
Avenues for improvement: PairFlow could be extended from "evaluation use" to contrastive learning/preference training (e.g., generating hard negatives for DPO). The paper also hints that this pipeline will become increasingly valuable as generation models improve.

vs. VideoHallucer: VideoHallucer generates "one true, one false" binary question pairs on single videos, which primarily induces text hallucination (perturbing LLMs); this work synthesizes video pairs and strictly controls backgrounds, aiming directly at visual hallucinations. The advantage of this work lies in isolating errors to the foreground instead of language priors, though at the cost of relying on synthesis and being constrained by generator quality.
vs. VidHalluc: VidHalluc uses CLIP/DINO to retrieve "visually different but semantically similar" video pairs to test temporal hallucinations, where backgrounds and camera angles change entirely; this work deliberately freezes backgrounds and only edits foregrounds. Experiments (Fig. 9) show that most models experience a sharp drop in performance when moving from VidHalluc to VidPair-Halluc, proving that background control is tougher and more effective at exposing fine-grained weaknesses.
vs. HallusionBench: HallusionBench uses meticulously styled image-question pairs to decouple language and visual illusions; this work extends this concept to video and introduces additional adversarial pairs along the temporal axis.
vs. Preference-Alignment Works such as VistaDPO / Hound-DPO: These works alleviate hallucinations by training on visual preference pairs, but high-quality video preference pairs are heavily reliant on human annotations of keyframes/segments, suffering from poor scalability. PairFlow follows the same "adversarial video pairs" concept but leverages generative synthesis to bypass human labor and focuses on evaluation, opening up potential interfaces for low-cost preference pair generation in the future.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ First video hallucination benchmark to explicitly control background consistency; the "background-controlled attribution" experimental design strikes at the heart of prior works' vulnerability.
Experimental Thoroughness: ⭐⭐⭐⭐☆ Evaluates 15 LVMs + 7 frontier flagships in the appendix, with robust multi-dimensional comparative analyses across three QA classes; however, lacks a systematic ablation study of each PairFlow stage.
Writing Quality: ⭐⭐⭐⭐☆ Clear motivation and pipeline explanation, rich illustrations; notation for some metric definitions (qAcc/vAcc) is slightly dense.
Value: ⭐⭐⭐⭐⭐ Provides a clean controlled probe for fine-grained video hallucination diagnosis, paving the way for evaluation/training reuse of synthetic adversarial pairs.