Spotlight: Identifying and Localizing Video Generation Errors Using VLMs¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://spotlight-video.github.io
Area: Video Generation / Multimodal VLM / Evaluation Benchmark
Keywords: Video Generation Evaluation, Fine-Grained Error Localization, Vision-Language Models, Temporal Grounding, Physical & Semantic Consistency

TL;DR¶

To tackle nuanced, transient generation failures in modern high-fidelity text-to-video models, this paper introduces Spotlight—a benchmark containing 1,604 human-annotated errors across 600 videos from top generators with timestamps, error categories, and natural language explanations—revealing that current leading VLMs lag behind human-level fault localization and reasoning by nearly 2× under bipartite matching metrics.

Background & Motivation¶

Text-to-video (T2V) generation has advanced dramatically due to large-scale diffusion and transformer architectures. Frontier models such as Veo 3, Seedance, and LTX-2 synthesize remarkably high-fidelity, temporally coherent, and photorealistic videos. Having largely eliminated the obvious, macroscopic visual artifacts that plagued earlier generative pipelines, modern models now exhibit subtle, transient, and spatio-temporally localized failures. Such errors manifest as momentary physical impossibilities, subtle anatomical distortions, or anomalous material morphing embedded within otherwise aesthetically pleasing footage, forcing viewers to repeatedly pause and scrutinize the scene to understand what went wrong.

To reduce reliance on labor-intensive human evaluation, the community has actively explored Vision-Language Models (VLMs) as automated judges and reward models. However, existing benchmarks primarily focus on global holistic scoring, overall text-video alignment, or coarse classification of rendering defects across entire clips. This creates a critical tension: automated evaluation and post-training reinforcement learning urgently require evaluators that pinpoint exactly when and why an error occurs with granular timestamps and causal explanations, yet it remains unexplored whether state-of-the-art VLMs possess the fine-grained temporal perception and commonsense reasoning needed to localize and explain these subtle generation flaws.

This paper tackles this gap by formulating a rigorous temporal localization and natural language explanation task for generative video defects. Core idea: construct Spotlight, a benchmark encompassing 1,604 fine-grained temporal error annotations across physics, semantics, and anatomy from frontier video generators, and introduce a bipartite matching evaluation protocol combining temporal precision and semantic reasoning to diagnose the perception bottlenecks and hallucinations of VLMs.

Method¶

Overall Architecture¶

The Spotlight task evaluates the capability of VLMs to identify, temporally localize, and explain fine-grained errors in videos synthesized by modern T2V models. Given an AI-generated video $V$ generated from prompt $P$ with duration $d$ seconds, the model is tasked with predicting an error set $\hat{E} = \{\hat{e}_1, \dots, \hat{e}_j\}$, where each predicted error $\hat{e}_j = (\hat{t}^s_j, \hat{t}^e_j, \hat{c}_j, \hat{r}_j)$ includes temporal boundary timestamps $(\hat{t}^s_j, \hat{t}^e_j)$, error category $\hat{c}_j$, and a natural language rationale $\hat{r}_j$. The evaluation protocol maps predictions against human ground truth $E = \{e_i\}$ using bipartite matching under combined temporal precision and semantic similarity constraints.

The pipeline comprises three major stages: dataset collection and hierarchical taxonomy curation, inference-time reasoning and task decomposition strategies, and maximum-weight bipartite matching evaluation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Generated Video V and Prompt P"] --> B["Stage 1: Multi-Dimensional Error Taxonomy Construction<br/>Physics / Semantics / Anatomy 3 Types across 6 Categories"]
    B --> C["Stage 2: Inference-Time Decomposition & Multi-Agent Auditing<br/>Category Divide-and-Conquer / Sliding Window / Sequential Grounding"]
    C --> D["Stage 3: Maximum-Weight Bipartite Matching Evaluation<br/>Temporal Precision P + Reason Similarity S Joint Thresholding"]
    D --> E["Output: Exact Diagnostic Alignment Scores & Coverage"]

Key Designs¶

1. Multi-Dimensional Error Taxonomy Construction: Mapping the Failure Boundaries of Modern Generators

Prior video quality benchmarks primarily capture coarse artifacts (such as color banding or blur) or clip-level misalignment, failing to capture subtle physical and logical anomalies produced by frontier generators. Spotlight defines a structured hierarchy spanning 3 core types and 6 fine-grained categories: - Physics: encompasses "Physical Violations" (defying gravity, inter-object penetration, unphysical fluid dynamics) and "Appearance/Disappearance" (objects abruptly popping in or vanishing without occlusions). - Semantics: covers "Motion" (unnatural acceleration or sudden trajectory jumps), "Prompt Adherence" (omitted actions or attributes specified by the prompt), and "Logical Errors" (causal incoherence or object functionality contradictions). - Anatomy: focuses on "Body Anatomy" (extra or missing limbs, impossible joint inversions, facial morphing in humans and animals).

To curate diverse and challenging scenarios, 200 prompts were sourced across StoryEval, VBench2, VidProM, BlackSwan, and LVD-2M, spanning creative, dynamic, and realistic domains. 600 videos were generated using Veo 3 Fast, Seedance 1.0, and open-weight LTX-2. Annotations were collected via a two-round qualified crowd-sourcing pipeline on CloudConnect: the first round filtered annotators by requiring at least 3 errors per video and rationales of $\ge 8$ words; the second round engaged these vetted annotators to thoroughly re-annotate the 600 videos and assign a difficulty rating from 1 to 5 (partitioned into Easy, Medium, and Hard). Overlapping annotations (IoU $\ge 0.6$) were semantically merged via an LLM, yielding 1,604 validated errors (average video length 6.5s, average error duration 2.49s, inter-annotator type agreement of 83%, and human validation accuracy of 93% for rationales and 99% for temporal boundaries).

2. Inference-Time Decomposition & Multi-Agent Auditing: Mitigating Temporal Collapse in Long-Context VLMs

Direct zero-shot prompting of VLMs (predicting an entire JSON list of errors in one pass) often leads to temporal collapse—where models default to marking the entire video duration as an error or completely miss transient, micro-second defects. Spotlight investigates four structured inference-time strategies: - Chain-of-Thought (CoT): prompts the VLM to explicitly describe frame-by-frame observations and potential discrepancies before formatting the structured output, reducing hallucinated timestamp bounds. - Sliding Window (SW): partitions the video into 2-second sub-clips and processes each at an elevated density of 4 fps. Temporal intervals are mapped back to the global timeline, and an LLM in text-only mode performs transitive semantic clustering and interval merging (taking unions) to resolve redundancies while capturing localized transient errors. - Sequential: Reason then Localize (Seq): decouples error detection from timestamp regression by mimicking human cognitive flow. In the first pass at 2 fps, the VLM scans the entire video to extract textual error descriptions without timestamps. In the second pass, each identified error acts as a query prompt, directing the VLM to re-inspect the video specifically to locate start and end timestamps. - Multi-Agent (MA): executes a divide-and-conquer strategy by launching 6 category-specific VLM query instances, each focused exclusively on identifying one of the 6 error classes. The outputs are aggregated and consolidated, effectively preventing attention dilution across long prompts.

3. Maximum-Weight Bipartite Matching Evaluation: Rigorous Dual-Constraint Temporal and Semantic Alignment

Because predicted errors $\hat{E}$ and ground-truth errors $E$ differ in cardinality, temporal boundaries, and semantic wording—with possible overlapping errors within single frames—simple global IoU or string matching fails to fairly evaluate performance. Spotlight introduces an optimal bipartite matching formulation: - Temporal Precision ($P$) and Recall ($R$): for ground truth error $e_i = (t^s_i, t^e_i, c_i, r_i)$ and prediction $\hat{e}_j = (\hat{t}^s_j, \hat{t}^e_j, \hat{c}_j, \hat{r}_j)$, precision measures the fraction of the predicted interval that falls inside the ground truth, penalizing overly broad predictions; recall measures the fraction of the ground truth interval covered: $$P(e_i, \hat{e}_j) = \frac{|[t^s_i, t^e_i] \cap [\hat{t}^s_j, \hat{t}^e_j]|}{\hat{t}^e_j - \hat{t}^s_j}, \quad R(e_i, \hat{e}_j) = \frac{|[t^s_i, t^e_i] \cap [\hat{t}^s_j, \hat{t}^e_j]|}{t^e_i - t^s_i}$$ - Reason Similarity ($S$): an impartial evaluator (GPT-4o-mini) grades the semantic alignment between ground truth rationale $r_i$ and predicted rationale $\hat{r}_j$ on a 1-to-10 scale, normalized to $[0, 1]$: $$S(e_i, \hat{e}_j) = \frac{\text{LLM-Match}(r_i, \hat{r}_j)}{10}$$ - Joint Threshold Bipartite Cost Matrix ($S+P$): since ground truth errors are compact (averaging 2.49s), models that artificially inflate interval length to boost recall must be penalized. Spotlight enforces a hard threshold $\tau = 0.7$. A candidate edge between $e_i$ and $\hat{e}_j$ is non-zero only if both precision and semantic similarity exceed $\tau$: $$F_{i,j} = \begin{cases} \frac{P_{i,j} + S_{i,j}}{2}, & \text{if } P_{i,j} \ge \tau \land S_{i,j} \ge \tau \\ 0, & \text{otherwise} \end{cases}$$ - Optimal Assignment Solving: Kuhn-Munkres maximum-weight bipartite matching solves for the optimal assignment set $\mathcal{M}$, yielding the final average score $M(E, \hat{E}) = \frac{1}{|\mathcal{M}|}\sum_{(e_i, \hat{e}_j)\in\mathcal{M}} F_{i,j}$ and ground-truth coverage $M^\%(E, \hat{E}) = \frac{|\mathcal{M}|}{|E|}$.

Key Experimental Results¶

Main Results¶

The authors benchmarked both open-weight and closed-source state-of-the-art multimodal models across zero-shot and inference-time strategies at threshold $\tau = 0.7$, alongside non-VLM baselines (optical flow, CLIP similarity) and expert human performance.

Model / Strategy	Variant	S+P Score $M$	S+P Coverage $M^\%$	Similarity $M$	Similarity Coverage $M^\%$	Precision $P$	Precision Coverage $M^\%$	Recall $R$	Recall Coverage $M^\%$
Non-VLM Baselines	Optical Flow	-	-	-	-	0.462	32.2%	0.049	3.0%
	CLIP Sim.	-	-	-	-	0.663	38.1%	0.029	1.2%
Open-Weight Zero-Shot	InternVL3-8B	0.041	2.2%	0.150	8.6%	0.408	22.8%	0.593	35.8%
	MiniCPM-V-4.5 (8B)	0.118	5.7%	0.248	14.0%	0.621	34.5%	0.530	28.6%
	Molmo 2 (8B)	0.087	5.0%	0.291	19.7%	0.443	26.3%	0.845	74.8%
	Qwen3-VL-8B-Instruct	0.148	8.0%	0.352	23.3%	0.472	27.1%	0.903	73.1%
	Qwen3-VL-30B-A3B-I	0.137	6.1%	0.339	21.5%	0.510	29.9%	0.910	73.9%
Closed-Source Zero-Shot	GPT-5.1 (Multi-Frame)	0.218	11.6%	0.340	23.3%	0.506	33.6%	0.599	43.5%
	Gemini 2.5 Pro	0.250	12.6%	0.422	26.7%	0.691	44.3%	0.836	68.4%
	Gemini 3 Pro	0.250	12.1%	0.387	24.1%	0.807	56.3%	0.681	45.9%
Inference-Time Baselines	Qwen3-8B - CoT	0.163	7.9%	0.281	17.4%	0.617	40.2%	0.630	48.7%
	Qwen3-8B - Sliding Window (SW)	0.189	10.0%	0.386	24.6%	0.664	41.8%	0.931	67.6%
	Qwen3-8B - Sequential (Seq)	0.189	9.3%	0.394	25.7%	0.645	39.4%	0.822	60.9%
	Qwen3-8B - Multi-Agent (MA)	0.254	13.3%	0.405	30.6%	0.675	49.8%	0.767	61.7%
	Gemini 2.5 Pro - MA	0.308	17.0%	0.514	33.8%	0.812	57.7%	0.829	67.6%
Human Benchmark	Human Baseline	0.508	26.8%	0.599	34.4%	0.840	48.5%	0.880	50.0%

Ablation Study and Category Breakdown¶

Performance across the 6 specific error categories under the joint S+P metric at threshold $\tau = 0.7$ provides deep diagnostic insights into model strengths and failure modes:

Category Super-Type	Specific Category	Qwen3-8B-MA Score $M$ ($M^\%$)	Gemini 2.5 Pro Score $M$ ($M^\%$)	Human Score $M$ ($M^\%$)	Failure Characteristics
Physics	Physical Violations (Phy. Viol.)	0.205 (20.3%)	0.149 (14.4%)	0.367 (33.9%)	Mechanics and collision dynamics; open-weight MA beats Gemini zero-shot
	Appearance/Disappearance (App/Dis)	0.017 (1.5%)	0.099 (9.0%)	0.139 (12.1%)	Extremely fleeting flickering; lowest detection rate across all models
Semantics	Motion	0.122 (13.0%)	0.117 (11.1%)	0.264 (30.0%)	Jerky kinematic shifts; bottlenecked by temporal frame sampling rate
	Prompt Adherence (Adherence)	0.243 (21.4%)	0.246 (20.8%)	0.532 (45.0%)	Macro semantic misses; easiest for both humans and models to catch
	Logical Errors (Logical)	0.132 (11.4%)	0.083 (5.9%)	0.210 (21.4%)	Causal incoherence; requires long-range common-sense reasoning
Anatomy	Body Anatomy (Body An.)	0.068 (7.0%)	0.078 (8.3%)	0.153 (17.6%)	Micro-level limb/facial defects; prone to misses and false positives

Key Findings¶

Human Performance Outperforms Top VLMs by 2×: On the rigorous S+P metric, human evaluators achieve a score of 0.508 with 26.8% coverage, whereas the best zero-shot models (Gemini 2.5 Pro and Gemini 3 Pro) score 0.250 with 12.1%–12.6% coverage. Even with multi-agent inference enhancement, Gemini 2.5 Pro reaches 0.308, leaving a substantial gap to human capability.
The Illusion of High Recall: Open-weight models like Qwen3-VL and Molmo 2 show remarkably high isolated recall ($73\%\sim75\%$), but this stems from an overly permissive bias to predict entire video durations as erroneous. When joint precision and semantic thresholding ($S+P$) are enforced, their performance collapses to $6\%\sim8\%$ coverage.
Multi-Agent Decomposition Yields Outsized Gains: Partitioning the auditing process into 6 category-dedicated agents allows an 8B open model (Qwen3-8B-MA) to surge from 0.148 to 0.254 on S+P, matching and slightly exceeding the zero-shot performance of closed-source Gemini 2.5 Pro.
Failure Root-Cause Analysis: An audit of 50 failure cases from Gemini 2.5 Pro reveals that 78% of errors exhibit clear failures: 29% due to perceptual limitations (e.g., missing fine grass blade motion), 29% due to hallucinations (fabricating events absent from the video), and 19% due to flawed physical commonsense. In addition, 35.9% of predicted timestamps were overly broad.

Highlights & Insights¶

From Holistic Scores to Localized Causal Diagnosis: While prior benchmarks offer single overall quality scores, Spotlight formalizes video evaluation as a temporal localization and explanation task, establishing an interpretable foundation for training process-supervised reward models (PRMs) for generative video.
Dual-Threshold Bipartite Matching Rigor: Coupling temporal precision and rationale semantic similarity via a strict $\tau = 0.7$ threshold prevents models from gaming metrics through unbounded timestamp ranges or vague, generic explanations.
Effective Task Decomposition for Fine-Grained Perception: Decomposing monolithic video auditing into category-specific agents significantly mitigates attention dilution and hallucinations, demonstrating that lightweight open-source models can achieve competitive diagnostic utility via inference-time orchestration.

Limitations & Future Work¶

Author-Admitted Limitations: Despite aggregating 1,604 annotations across 600 videos, human cognitive biases, differing thresholds of visual tolerance, and annotator fatigue mean the dataset may not capture every fleeting defect in the generated clips.
Nuanced Observations: Temporal boundary annotation is inherently fuzzy for continuous non-rigid deformations or gradual physical transitions (e.g., slow melting or color morphing), where hard cutoffs at $\tau = 0.7$ might penalize reasonable boundary estimates; additionally, semantic similarity scoring relies on GPT-4o-mini as an automated judge.
Future Directions: Extending temporal intervals into spatio-temporal bounding tubes (3D spatio-temporal grounding) to indicate "where and when", and distilling Spotlight annotations into lightweight, specialized video critic models to directly guide diffusion denoising steps.

vs VBench / VBench 2.0: VBench provides macro-level capability rankings via dimensional prompt suites and clip-level ratings; Spotlight shifts focus to micro-level temporal error localization and causal explanation within state-of-the-art videos.
vs VideoPhy / Physics-IQ: These benchmarks evaluate physical plausibility via binary classification or multiple-choice QA; Spotlight embeds physical, semantic, and anatomical defects into a unified localization and natural language justification paradigm.
vs BrokenVideos / DAVID-XR1: BrokenVideos focuses on lower-level rendering distortions in older generators, and DAVID-XR1 targets synthetic video detection; Spotlight specifically addresses fine-grained, subtle failures in modern photorealistic generators (Veo 3, Seedance, LTX-2).

Rating¶

Novelty: ⭐⭐⭐⭐ [Pioneering fine-grained temporal error localization and rationalization for generative videos]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Spans 3 leading video generators, extensive open/closed VLMs, 4 inference strategies, difficulty splits, and human baselines]
Writing Quality: ⭐⭐⭐⭐⭐ [Clear problem formulation, elegant bipartite matching metric derivation, and comprehensive error analysis]
Value: ⭐⭐⭐⭐⭐ [Crucial diagnostic benchmark for the video generation community to steer next-generation VLM evaluators and process reward models]