Skip to content

VGI-Bench: Probing Visual Intelligence in Video Generation Models

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Video Generation Models, Visual Intelligence, Process-Sensitive Evaluation, VLM-as-Judge, Visual Reasoning

TL;DR

VGI-Bench introduces a photorealistic, process-sensitive benchmark with 27 tasks and 810 instances to evaluate procedural visual reasoning in video generation models, revealing that the strongest model Seedance 2.0 achieves only 51.0% under a multi-factor judge protocol while intermediate denoising analysis demonstrates virtually absent (<1%) visual self-correction.

Background & Motivation

With the rapid emergence of high-capacity diffusion transformers and autoregressive architectures, video generation models are increasingly transitioning from passive visual synthesis tools toward physical world simulators capable of hallucinating plausible scene evolutions. Recent empirical studies further suggest that scaling video pre-training allows models to capture spatial-temporal geometry, physical affordances, and structured environmental rules, sparking enthusiasm for their potential as zero-shot visual reasoners. Beyond high-fidelity video rendering, treating video generation models as reasoning engines that solve procedural tasks by explicitly "thinking in frames" could transform downstream robot manipulation, embodied policy learning, and multimodal deliberate planning.

However, reliably evaluating the emergent visual intelligence encoded in current video models remains severely impeded by three systemic deficiencies in existing benchmarks. First, existing suites predominantly rely on line-art schematics, synthetic diagrams, or abstract game grids for programmatic scalability; such inputs deviate substantially from the rich naturalistic image priors learned during web-scale pre-training, causing catastrophic visual collapse and constraint ignorance that conflate visual domain mismatch with intrinsic reasoning deficiencies. Second, many existing benchmarks lack process sensitivity: tasks are frequently solvable directly from the initial frame without requiring dynamic visual rollouts, allowing models to cheat via plausible static configurations or abrupt non-physical teleportations that bypass intermediate rule constraints. Third, task difficulty and feasibility are poorly calibrated, frequently incorporating intractable multi-minute horizons exceeding modern video clip durations or knowledge-intensive queries requiring non-visual domain expertise, resulting in saturated near-zero performance floors that offer negligible diagnostic value.

To bridge these gaps, a rigorous evaluation framework must operate within the realistic visual domain, enforce strict fidelity across intermediate procedural state transitions, and anchor task difficulty directly near current model capability boundaries. Core idea: construct VGI-Bench, a photorealistic, process-sensitive, and feasibility-calibrated benchmark covering 4 domains and 7 cognitive skill tags across 27 procedural tasks, evaluated via a dual-factor VLM-as-Judge protocol that couples global goal completeness with fine-grained adaptive checklist verification to diagnose the true boundaries, sensitivity, and internal denoising dynamics of video reasoning.

Method

Overall Architecture

VGI-Bench standardizes video reasoning evaluation around an image-to-video (I2V) formulation: given a photorealistic starting frame (standardized to 16:9 aspect ratio) and a text prompt specifying the operational objective alongside scene-level and procedural constraints, the model must roll out a 5–10s continuous video completing the required multi-step execution. The methodology is structured around three primary pillars: a two-level domain-and-skill taxonomy, pre-generation feasibility calibration with human-in-the-loop review, and a dual-factor VLM-as-Judge scoring protocol that aggregates macro-level completeness with micro-level rubric adherence.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Photorealistic Image ⊕ Goal & Constraint Prompt"] --> B["Two-Level Capability Taxonomy<br/>4 Mutually Exclusive Domains ⊕ 7 Non-Exclusive Skill Tags"]
    B --> C["Pre-Generation Feasibility Calibration & Quality Control<br/>SOTA Model Filter + Human-in-the-Loop 16:9 Natural Scene Synthesis"]
    C --> D["Model Rollout: 5–10s Continuous Temporal Generation"]
    D --> E["Macro Goal Completeness Assessment<br/>2 fps Uniform Sampling into 3-Tier Discrete Standard"]
    D --> F["Micro Process Rule Checklist Verification<br/>Adaptive Sliding Window (4/8 fps) ⊕ Inverse Decay Penalty"]
    E --> G["Multiplicative Dual-Factor Aggregation & Denoising Diagnosis"]
    F --> G

Key Designs

1. Two-Level Capability Taxonomy: Disentangling Task Contexts from Cognitive Competencies Prior benchmarks frequently confound task categories with underlying cognitive requirements, masking the precise nature of model failure modes. VGI-Bench establishes a two-level taxonomy comprising four mutually exclusive task domains and seven non-exclusive cognitive skill tags. At the domain level, 27 tasks are categorized into Visual Organization (arranging or grouping items based on visual cues), Physical Manipulation (contact-rich interactions like block assembly, object packing, or tool application), Structured Puzzles (rule-governed logic problems such as maze solving, Tower of Hanoi, and polyform tiling), and Spatiotemporal Dynamics (tracking temporal dependencies and state mutations such as clock progression and Eulerian path tracing). At the skill tag level, each task is annotated with one to three cognitive requirements grounded in visual cognitive science: Planning (12 tasks), Physics (8 tasks), Spatial (7 tasks), Attribute Grounding (6 tasks), Affordance (5 tasks), Topology (3 tasks), and Temporal (3 tasks). This dual-level structure enables granular failure attribution across both practical application categories and foundational cognitive primitives.

2. Pre-Generation Feasibility Calibration & Quality Control: Enforcing Boundary Diagnosticity To eliminate uninformative floor and ceiling effects, VGI-Bench implements an empirical pre-generation filter before final instance curation. For every candidate task proposal, two instances at the easiest level are probed across leading commercial video models (including Sora 2, Veo 3.1, and Kling 3.0). A task is accepted into the benchmark if and only if it is successfully resolved by at least one model and simultaneously failed by at least one other model; tasks that are universally trivial or universally intractable are discarded or redesigned. Following acceptance, high-fidelity input scenes are generated via a human-in-the-loop pipeline with advanced image models (GPT-Image-2 and Nano Banana Pro) and real-world collections, enforcing realistic materials, natural lighting, drop shadows, and 16:9 framing. Each task is expanded across three difficulty tiers (Easy, Mid, Hard) with 10 instances per tier, yielding a calibrated suite of 810 evaluation instances fitting within typical 5–10s generation durations.

3. Macro Goal Completeness Assessment: Discrete Tiered Verification of State Progress In procedural reasoning, evaluating terminal states alone rewards static hallucinations or spurious final frames. The Completeness metric (\(Comp\)) provides a macro-level assessment of whether the model made genuine progress toward the intended target state. Sampled uniformly at a low frame rate of 2 fps, the video frames are evaluated by a calibrated VLM judge (Gemini-3-Flash) conditioned on the starting frame, optional ground-truth visual references, and a task-specific tiered rubric. The judge maps the trajectory into one of three discrete tiers: \(0\) (Failed: static scene, severe input drift, or immediate invalid pathing), \(0.5\) (Partial: meaningful progress made along valid corridors without reaching the terminal goal), and \(1.0\) (Complete: full target realization while preserving core scene elements). This discrete formulation ensures that global trajectory fidelity is recognized without being dominated by localized visual noise.

4. Micro Process Rule Checklist Verification: Adaptive Coarse-to-Fine Sliding-Window Audit While a video may appear to reach a target state, critical violations—such as transient wall clipping, impossible phase transitions, or duplicate entity spawning—frequently occur in brief temporal intervals. To catch localized infractions without incurring prohibitive context lengths, VGI-Bench introduces an adaptive coarse-to-fine verification protocol. The VLM judge first inspects the video at 4 fps using a 10-frame sliding focus window with a 1-frame boundary overlap. Frames flagged as suspicious or containing ambiguous interactions are dynamically resampled at a refined rate of 8 fps for targeted re-inspection. Local window reports are subsequently integrated by an image-free Polish call that deduplicates persistent infractions. For each checklist rule \(i\), its item score is calculated via an inverse-decay penalty:

\[s_i = \frac{1}{x_i + 1}\]

where \(x_i\) denotes the final count of distinct substantive violations of rule \(i\). The overall Rubric Score (\(Rub\)) is the unweighted arithmetic mean across all \(N\) checklist items:

\[Rub = \frac{1}{N} \sum_{i=1}^{N} s_i\]

The final aggregated instance score enforces joint necessity via multiplication: \(\text{Final} = Comp \times Rub\). Videos that remain static achieve high rubric scores but near-zero completeness, whereas videos that cheat to reach the goal incur heavy rubric penalties, penalizing both degeneration paths.

Benchmark & Evaluation Protocol

The primary video evaluation benchmarks 9 representative models across commercial proprietary APIs (Seedance 2.0, Kling 3.0, Gen 4.5, Sora 2, Wan 2.7, Veo 3.1) and open-source models deployed locally on NVIDIA H20 GPU clusters (MiniMax-H3, Wan 2.2-I2V-A14B, HunyuanVideo-1.5). All models generate at \(1280 \times 720\) resolution (\(960 \times 544\) for MiniMax-H3) with 5s or 10s clip durations. In addition, an auxiliary single-image adaptation suite across 16 tasks evaluates 11 advanced image generation models (e.g., Nano Banana Pro, GPT-Image-2, Flux.2-Max) to isolate static target-state deduction from temporal rollout maintenance. Finally, a synthetic fine-tuning transfer experiment investigates whether scaling LoRA tuning on 1M abstract synthetic videos (via the VBVR framework) transfers to naturalistic visual reasoning tasks.

Key Experimental Results

Main Results: Comprehensive Video Reasoning Performance

The aggregated performance (\(\text{Final} = Comp \times Rub\), normalized to 0–100) across all 9 video generation models on VGI-Bench is summarized below:

Domain Level Seedance 2.0 Kling 3.0 Gen 4.5 Sora 2 Wan 2.7 Veo 3.1 MiniMax-H3 Wan 2.2 HunyuanVideo 1.5
Visual Organization Easy 72.6 67.6 60.5 55.6 37.0 50.3 42.8 30.4 26.6
(5 tasks) Mid 56.4 47.5 38.3 37.0 32.2 43.3 42.3 15.3 17.9
Hard 53.5 43.7 41.9 39.1 32.5 44.0 39.8 18.5 24.7
Avg. 60.8 52.9 46.9 43.9 33.9 45.9 41.6 21.4 23.1
Spatiotemporal Dynamics Easy 54.6 45.2 39.6 45.3 38.9 28.1 44.3 38.7 34.7
(7 tasks) Mid 47.4 35.2 34.6 34.6 42.6 21.2 40.7 32.0 27.2
Hard 33.4 28.2 29.5 24.4 28.9 16.9 35.9 19.9 23.9
Avg. 45.3 36.5 34.8 34.8 36.8 22.3 40.3 30.2 28.7
Structured Puzzles Easy 46.9 45.3 29.2 40.5 25.6 31.8 51.9 17.1 9.9
(8 tasks) Mid 45.1 38.3 20.7 25.8 23.3 21.8 52.3 8.4 8.8
Hard 41.8 28.9 18.7 22.4 24.7 13.5 47.6 5.6 7.9
Avg. 44.6 37.5 22.9 29.5 24.5 22.4 50.6 10.4 8.9
Physical Manipulation Easy 64.4 60.6 52.1 47.9 55.8 49.6 58.6 33.7 22.5
(7 tasks) Mid 59.2 51.2 43.4 41.8 48.2 40.4 41.5 22.1 16.7
Hard 44.4 45.6 39.5 32.6 37.1 37.3 33.0 17.5 11.9
Avg. 56.0 52.5 45.0 40.8 47.1 42.5 44.4 24.4 17.0
Overall Mean Total 51.0 44.0 36.6 36.7 35.7 32.0 44.4 21.6 19.1

Note: Bold highlights the top performing model per row. Commercial models consistently lead open-source competitors, though MiniMax-H3 exhibits outstanding reasoning capabilities in Structured Puzzles (50.6).

Reliability and Ablation of Evaluator Protocol

The validity of the VLM-as-Judge system was benchmarked against 1,100+ human pairwise preference annotations across ROC-AUC and Pairwise Agreement:

Judge Configuration & Base Model ROC-AUC Pairwise Agreement (%) Core Mechanism Impact
Full Evaluator (Gemini-3-Flash) 0.803 73.2% Coarse-to-fine adaptive sampling + 10-frame sliding focus window
w/o adaptive fps sampling 0.772 69.5% Misses transient wall-penetrations and high-speed teleportations (-3.7%)
w/o sliding focus window 0.753 68.3% Evaluating full sequence in single pass causes attention dilution (-4.9%)
w/ GPT-5-mini 0.690 64.3% Degraded spatiotemporal bounding and fine-grained violation detection
w/ Qwen3.6-Plus 0.624 59.1% Lower consistency in temporal continuity checks
w/ Claude-Haiku-4.5 0.478 47.9% Failure on rule checklist tracking, near random agreement

Internal Denoising Dynamics & Synthetic Transfer Limits

Intermediate latent decodings across 40 denoising steps (117 instances across 4 open-source models, yielding 702 checkpoint transitions) uncover how reasoning hypotheses evolve:

Denoising Step Pair Unrecognizable (♠) Stable State (♣) Correct \(\to\) Wrong Wrong \(\to\) Correct (Self-Correction) Wrong \(\to\) Wrong'
1 \(\to\) 2 69.2% 17.9% 2.6% 0.9% 9.4%
2 \(\to\) 3 41.9% 49.6% 0.9% 0.0% 7.7%
3 \(\to\) 4 22.2% 70.1% 0.0% 0.9% 6.8%
4 \(\to\) 10 6.8% 69.2% 0.9% 0.0% 23.1%
10 \(\to\) 20 0.0% 75.2% 0.0% 0.0% 24.8%
20 \(\to\) 40 0.0% 90.6% 0.0% 0.0% 9.4%

Synthetic Fine-Tuning Transfer (VBVR-Wan2.2 vs. Base Wan2.2-I2V): - Structural Overlap Tasks: Final score jumped from 15.3 to 55.8 (\(\uparrow 40.5\)), including Maze (29.0 \(\to\) 87.8) and Tower of Hanoi (3.2 \(\to\) 52.9). - Semi-Overlap Tasks: Modest gains from 17.8 to 34.9 (\(\uparrow 17.1\)), e.g., Untie Knot improving from 1.8 to 34.0 (primarily through reduced erratic action). - Non-Overlap Tasks: Highly bounded transfer from 29.2 to 35.4 (\(\uparrow 6.2\)), with 3D spatial unwrapping stagnating (16.3 \(\to\) 16.7) and Tool Use degrading (58.8 \(\to\) 44.0).

Key Findings

  • Current Video Models Remain Far from Robust General Reasoners: Even the top commercial system, Seedance 2.0, obtains only 51.0 on composite scoring. Under the zero-tolerance Strict Success Rate (requiring both flawless goal arrival and zero checklist infractions), Seedance 2.0 plummets to 14.0%, Kling 3.0 to 9.5%, and Wan 2.2 to 2.3%, contrasted with a human baseline of 97.1%.
  • Severe Cognitive Blind Spots in Topology and Temporal Coherence: Across cognitive skill tags, models score highest on short-term Planning and general Spatial cues, but exhibit severe vulnerabilities in Topology (maintaining unbroken structural connectivity) and Temporal State Tracking (preventing vanishing or resetting objects mid-trajectory).
  • Denoising Trajectories Exhibit Negligible Self-Correction: Beyond step 4 (the initial 10% of sampling), transition probability from an incorrect hypothesis to a correct state is precisely 0.0%. Subsequent state mutations consist almost entirely of hopping between different invalid configurations (\(\text{Wrong} \to \text{Wrong}'\), ~24%), with 90.6% trajectory freezing in the final half of denoising, proving that diffusion models refine initial commitments rather than performing search-like error correction.
  • Oracle Prompting Yields Limited Ceiling Relief: Supplying full ground-truth step-by-step action sequences in prompts provides minimal performance gains; text-level descriptions cannot easily convey continuous physical micro-dynamics, and generative models struggle to simultaneously maintain physical cohesion while obeying strict multi-constraint instructions.

Highlights & Insights

  • Disentangling Domain Appearance from Reasoning Capability: By demonstrating that abstract line-art inputs cause open-source models to collapse on the exact same underlying logic puzzles, VGI-Bench establishes that photorealistic conditioning is vital to accurately measure a video foundation model's learned world priors.
  • Adaptive Frame Sampling as an Efficient VLM-Judge Paradigm: Combining coarse initial sweeps with fine-grained zoom windows on flagged intervals captures transient physical clipping at negligible inference cost compared to dense end-to-end video decoding.
  • Empirical Refutation of Denoising Reasoning Emergence: By decoding intermediate latents across multiple diffusion backbones, the work provides concrete empirical evidence countering the speculative narrative that iterative diffusion steps perform endogenous chain-of-thought self-correction.

Limitations & Future Work

  • Temporal Horizon and Modality Constraints: All tasks are scoped around 5–10s generation clips, excluding long-horizon multi-step planning (e.g., multi-stage mechanical repair) and multi-image or video-conditioned procedural generation.
  • Monolingual Protocol: Prompts and rubrics are exclusively formulated in English, leaving multilingual instruction grounding unexamined.
  • Future Directions: Developing test-time planning techniques in latent diffusion space (such as tree search or value-guided sampling) to steer early structural commitments and enable true mid-trajectory self-correction.
  • vs. PhysGenBench & WorldSimBench: While prior world-simulation benchmarks focus primarily on passive physical plausibility (fluid dynamics, falling objects), VGI-Bench evaluates active, goal-directed procedural reasoning with multi-step logical constraints.
  • vs. TiVi-Bench & V-ReasonBench: These earlier suites contain a substantial fraction of one-shot tasks solvable from the first frame and heavily utilize synthetic line-art inputs; VGI-Bench strictly enforces process sensitivity with photorealistic scenes and feasibility calibration.
  • vs. VBVR-Bench: Whereas VBVR focuses on synthetic adaptation benchmarks, VGI-Bench tests realistic transfer boundaries, proving that scaling abstract synthetic LoRA tuning yields highly constrained improvements that fail to generalize to non-aligned physical manipulation tasks.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ [Pioneering realistic, process-sensitive benchmark design with deep latent denoising trajectory analysis]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive multi-model video/image evaluations, 1100+ human preference alignments, and full ablation of judge mechanics]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Exceptionally rigorous formulation, precise definitions, and transparent diagnostic analysis]
  • Value: ⭐⭐⭐⭐⭐ [Crucial diagnostic foundation for advancing video generation models toward dependable world simulators and embodied reasoners]