title: >- [Paper Note] From Evaluation to Enhancement: Benchmarking and Improving Think-with-Video Reasoning for Video Generative Models description: >- [ECCV 2026][LLM Reasoning][Video Generation] Benchmarking think-with-video reasoning across 9 dimensions with VWG-Bench and boosting multi-generator reasoning accuracy via Vid-PRE's text-reward GRPO prompt rewriting. tags: - ECCV 2026 - LLM Reasoning - Video Generation - VWG-Bench - Prompt Enhancement - Reinforcement Learning date: 2026-09-19 content_hash: 4becbee3bec13d2a
From Evaluation to Enhancement: Benchmarking and Improving Think-with-Video Reasoning for Video Generative Models¶
Conference: ECCV 2026
Paper: ECCV Original
Code: https://huggingface.co/datasets/KlingTeam/VWG-Bench
Area: Multimodal VLM / Video Understanding / LLM Reasoning
Keywords: Video Generation, Visual Logical Reasoning, VWG-Bench, Prompt Enhancement, Reinforcement Learning Alignment
TL;DR¶
Addressing the core tension between video generative models' strong visual rendering and fragile physical/symbolic reasoning, this paper introduces VWG-Bench spanning 9 reasoning dimensions and a three-level decoupled evaluation protocol, alongside Vid-PRE—a model-agnostic prompt reasoner optimized via purely text-reward GRPO that substantially boosts downstream reasoning accuracy without architectural modifications.
Background & Motivation¶
Continuous spatiotemporal reasoning is a hallmark of physical intelligence that static images and text cannot fully capture. Following the evolution from "Thinking with Text" via Chain-of-Thought (CoT) prompting in language models to "Thinking with Images" via visual topological inference in Vision-Language Models, the frontier is advancing toward "Think with Video." Modern generative models—exemplified by Sora-2, Veo-3.1, and open-source systems such as the Wan family—are expected to operate as predictive physical simulators rather than mere visual rendering engines. However, whether these generative backbones genuinely comprehend symbolic rules, causal mechanics, and spatial-temporal constraints, or merely exploit shallow surface correlations to synthesize photorealistic yet physically impossible sequences, remains a critical unresolved question.
Existing evaluation benchmarks exhibit two fundamental limitations. First, task coverage is highly fragmented: existing suites such as VR-Bench, TiViBench, and V-ReasonBench focus narrowly on isolated domains like maze navigation, simple geometric shifts, or a small set of physical laws, lacking a holistic representation across symbolic logic, embodied action, digital interaction, and social commonsense. Second, existing evaluation protocols collapse visual smoothness, task-rule adherence, and final goal realization into a single holistic score or binary success metric. This entanglement conceals severe cognitive breakdowns: a model may generate a visually coherent video that blatantly violates rigid-body non-penetration or gravity, yet still receive a passable overall rating. Crucially, when downstream generators are queried with concise, ambiguous natural-language prompts, they lack explicit representations of spatial layout and rule hierarchies, inevitably drifting into severe cumulative reasoning errors.
The paper's core insight is that the primary bottleneck in video generation lies not in raw spatiotemporal rendering capacity, but in the cognitive disconnect between abstract instructions and physical-spatial execution plans. Addressing this challenge requires both a diagnostic suite that orthogonalizes visual quality, procedural constraints, and goal realization, and a mechanism that offloads the heavy cognitive burden from the downstream renderer. Core Idea: Construct VWG-Bench covering 9 core reasoning dimensions and 38 fine-grained tasks evaluated via a three-level decoupled VLM-as-Judge protocol, and introduce Vid-PRE, a plug-and-play prompt reasoner trained with rejection-sampled SFT and purely text-reward GRPO to translate implicit problems into constraint-aware, renderable instructions.
Method¶
Overall Architecture¶
The framework establishes a complete loop from diagnostic evaluation to reasoning enhancement. On the evaluation side, VWG-Bench defines a comprehensive taxonomy across 9 cognitive dimensions and 38 tasks, supported by rule dictionaries and automated data generation, where a multimodal LLM acts as an impartial judge executing independent evaluations across video quality, task constraints, and sample goals. On the enhancement side, Vid-PRE enforces a strict division of labor between reasoning and rendering: a dedicated multimodal LLM acts as the cognitive front-end, taking the initial frame \(I\) and raw user prompt \(P_{\text{user}}\), deriving explicit spatiotemporal constraints through a Chain-of-Thought (CoT), and distilling them into a concise, executable prompt \(P_{\text{opt}}\) that directly drives any frozen downstream video generator (e.g., Wan2.2, Wan2.6, Veo3.1).
The complete workflow of Vid-PRE prompt enhancement and VWG-Bench three-level evaluation is illustrated below:
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: Initial Frame I + Raw Instruction P_user"] --> B["Stage 1: Rejection-Sampled CoT SFT<br/>Qwen3-VL-8B learns structural formats & causal reasoning"]
B --> C["Stage 2: Purely Text-Based Reward GRPO Alignment<br/>Intent, spatial, correctness & format text rewards"]
C --> D["Output: Constraint-Aware Executable Prompt P_opt<br/>Decouples cognitive reasoning from visual rendering"]
D --> E["Downstream Video Generator Rendering<br/>Wan2.2 / Wan2.6 / Commercial models without fine-tuning"]
E --> F["Three-Level Decoupled Evaluation Protocol<br/>Video quality / Task rules / Sample goals independent diagnosis"]
Key Designs¶
1. Rejection-Sampled CoT SFT: Establishing Structured Causal and Formatting Foundations
Standard prompt rewriting for video generation predominantly amplifies aesthetic descriptions or scene embellishments, lacking the structured derivation necessary for spatial geometry and algorithmic puzzles. Directly mapping \((I, P_{\text{user}}) \to P_{\text{opt}}\) leads the rewriter to make ungrounded guesses about the terminal state. Vid-PRE models the autoregressive joint distribution \(P(\text{CoT}, P_{\text{opt}} \mid I, P_{\text{user}})\), inserting an explicit reasoning trajectory as an intermediate latent state. To curate roughly 20K high-fidelity training tuples \(\{I, P_{\text{user}}, \text{CoT}, P_{\text{opt}}\}\), an automated 4-step pipeline is deployed: first, define deterministic target ground truth \(G_{\text{truth}}\); second, prompt advanced proprietary APIs to analyze visual premises and produce candidate CoT and optimized prompts; third, apply an independent evaluator assessing visual alignment (penalizing hallucinated entities and spatial misidentifications) and logical sufficiency toward \(G_{\text{truth}}\), filtering candidates to only those scoring \(\ge 4\) on a 1–5 scale; fourth, incorporate "Difficulty Patching" on challenging tasks by providing \(G_{\text{truth}}\) as a corrective hint during CoT generation. Fine-tuning Qwen3-VL-8B on this corpus establishes a robust baseline for constraint-aware rewriting.
2. Purely Text-Based Multi-Dimensional Reward GRPO Alignment: Mitigating Hallucinations and Logical Drift
While SFT instills reasoning structures, complex long-horizon tasks still suffer from subtle spatial hallucinations and sequence step omissions. In the reinforcement learning stage, Vid-PRE makes a critical design choice: adopting a purely text-based reward system rather than backpropagating signals from rendered videos. This decision is driven by three factors: (1) eliminating the high variance and aesthetic noise of video generators where rendering artifacts suppress rewards for logically sound prompts; (2) cutting the massive computational cost of online video rendering during RL; and (3) maintaining generator agnosticism, ensuring Vid-PRE acts as a universal plug-and-play module. For each query \(x = (I, P_{\text{user}}, G_{\text{truth}})\), the policy samples a group of \(G\) rollouts \(\{o_1, \dots, o_G\}\) evaluated across four dimensions: - Intent Preservation (\(r_{\text{intent}}\)): confirms \(P_{\text{opt}}\) refines \(P_{\text{user}}\) without altering the underlying objective; - Visual-Spatial Consistency (\(r_{\text{image}}\)): strictly penalizes spatial descriptors that contradict initial frame \(I\); - Solution Correctness (\(r_{\text{correct}}\)): verifies that the proposed execution steps correctly reach \(G_{\text{truth}}\); - Format Compliance (\(r_{\text{format}}\)): ensures structural adherence to the required response template.
The combined reward \(r_k = w_1 r_{\text{intent}} + w_2 r_{\text{image}} + w_3 r_{\text{correct}} + w_4 r_{\text{format}}\) is group-normalized to compute advantages \(\hat{A}_k = (r_k - \mu_G) / \sigma_G\), optimizing the policy under KL regularization against the SFT baseline.
3. Three-Level Decoupled Evaluation Protocol: Orthogonalizing Perception and Cognitive Verification
Conventional binary or aesthetic metrics fail to localize why video models break down. VWG-Bench decouples evaluation into three orthogonal tiers on a calibrated 1–5 integer scale: - Video-Level Assessment: acts as coarse quality control, evaluating holistic visual fidelity and temporal smoothness while penalizing temporal jitter, flickering, and static freezes; - Task-Level Assessment: checks physical consistency against explicit rule dictionaries. Progress Consistency evaluates whether manipulated foreground objects evolve plausibly while background regions remain strictly static; Implicit Rule Following strictly penalizes physical violations such as solid mesh interpenetration, gravity inversion, or unnatural soft deformations of rigid bodies; - Sample-Level Assessment: assesses semantic instruction fulfillment. Progress Goal inspects intermediate keyframes to ensure the "Start \(\to\) Intermediate \(\to\) End" milestone sequence is respected; Last-Frame Goal (\(G_{\text{last}}\)) compares the terminal video frame directly against the target configuration (e.g., solved Sudoku numbers or correct arithmetic values).
Evaluated by Gemini-3.0-Pro acting as judge, this protocol achieves high human-expert correlation across 200 benchmark videos (Spearman's \(\rho = 0.69\text{--}0.80\), MAE \(\le 0.54\)).
Loss & Training¶
Vid-PRE is optimized through a two-stage paradigm:
-
Stage 1: Supervised Fine-Tuning
Optimized via standard autoregressive sequence negative log-likelihood: $\(L_{\text{SFT}}(\theta) = -\mathbb{E}_{(I, P_{\text{user}}, \text{CoT}, P_{\text{opt}}) \sim \mathcal{D}_{\text{SFT}}} \left[ \log P_\theta(\text{CoT}, P_{\text{opt}} \mid I, P_{\text{user}}) \right]\)$ Trained on 20K samples for 3 epochs with batch size 128 across 8 NVIDIA H100 GPUs using DeepSpeed ZeRO-3, with image token length capped at 1,024. -
Stage 2: GRPO Policy Optimization
Trained on 1.5K specialized queries selected from the 10 most challenging tasks post-SFT. For query \(x\), policy \(\pi_\theta\) generates rollouts \(o_k = (\text{CoT}_k, P_{\text{opt}, k})\). Using group-normalized advantage \(\hat{A}_k\), the objective maximizes: $\(J(\theta) = \mathbb{E}_{x \sim \mathcal{D}_{\text{RL}}, o \sim \pi_{\theta_{\text{old}}}} \left[ \frac{1}{G} \sum_{k=1}^G \frac{\pi_\theta(o_k \mid x)}{\pi_{\theta_{\text{old}}}(o_k \mid x)} \hat{A}_k - \beta D_{\text{KL}}(\pi_\theta \parallel \pi_{\text{ref}}) \right]\)$ where \(\pi_{\text{ref}}\) is the frozen SFT model. The optimization runs for 3K steps with \(\beta\) constraining policy drift.
Key Experimental Results¶
Main Results¶
Baseline diagnostics on VWG-Bench reveal widespread failure on rule adherence across 6 leading generators. Evaluated across VWG-Bench, V-ReasonBench (0–100 scale), and MME-CoF (1–5 scale), Vid-PRE consistently improves downstream generation across all models.
Table 1: Baseline diagnostic on VWG-Bench across commercial and open models (1–5 scale)
| Model | Video Quality | Prog. Consist. | Rule Follow. | Prog. Goal | Last Goal | Overall |
|---|---|---|---|---|---|---|
| Wan2.2 (Open-weight) | 2.32 | 2.33 | 1.56 | 2.45 | 1.98 | 2.13 |
| Wan2.5 | 2.70 | 2.86 | 3.14 | 2.48 | 2.56 | 2.75 |
| Wan2.6-Flash | 2.73 | 2.92 | 3.23 | 2.50 | 2.62 | 2.80 |
| Kling-2.5-Turbo-Pro | 2.95 | 3.12 | 3.38 | 2.62 | 2.78 | 2.97 |
| Sora2 | 3.45 | 3.68 | 3.79 | 3.32 | 2.98 | 3.44 |
| Veo3.1 | 3.54 | 3.78 | 3.95 | 3.45 | 3.05 | 3.55 |
Table 2: Performance enhancement of Vid-PRE on VWG-Bench and V-ReasonBench
| Base Generator | Prompt Setting | VWG Overall (1-5) | VWG Rule Follow. (1-5) | VWG Last Goal (1-5) | V-Reason Avg (0-100) | V-Reason Spatial Cognition |
|---|---|---|---|---|---|---|
| Wan2.2 | None | 2.13 | 1.56 | 1.98 | 26.65 | 1.32 |
| Wan2.2 | Vid-PRE (Ours) | 2.51 (+17.8%) | 2.12 (+35.9%) | 2.35 (+18.7%) | 44.48 (+66.9%) | 36.25 (+2646%) |
| Wan2.5 | None | 2.75 | 3.14 | 2.56 | 31.34 | 20.05 |
| Wan2.5 | Wan PE | 2.99 | 3.35 | 2.74 | 41.06 | 38.25 |
| Wan2.5 | Vid-PRE (Ours) | 3.19 (+16.0%) | 3.55 (+13.1%) | 2.93 (+14.5%) | 51.19 (+63.3%) | 42.35 (+111.2%) |
| Wan2.6-Flash | None | 2.80 | 3.23 | 2.62 | 32.10 | 20.71 |
| Wan2.6-Flash | Wan PE | 3.04 | 3.45 | 2.86 | 39.50 | 29.18 |
| Wan2.6-Flash | Vid-PRE (Ours) | 3.26 (+16.4%) | 3.62 (+12.1%) | 3.02 (+15.3%) | 52.57 (+63.8%) | 43.78 (+111.4%) |
Ablation Study¶
Ablation across training stages on external benchmarks isolates the independent contributions of SFT and GRPO:
Table 3: Ablation on training stages
| Base Generator | Variant | MME-CoF Overall (1-5) | V-ReasonBench Average (0-100) | Note |
|---|---|---|---|---|
| Wan2.2 | Base (None) | 1.72 | 26.65 | Severe spatial hallucinations and rule violations |
| Wan2.2 | + Vid-PRE (SFT only) | 2.44 | 40.26 | CoT provides basic step-by-step guidance |
| Wan2.2 | + Vid-PRE (SFT + RL) | 2.58 | 44.48 | Text rewards eliminate hallucinated coordinates |
| Wan2.6-Flash | Base (None) | 2.69 | 32.10 | Baseline reasoning is constrained |
| Wan2.6-Flash | + Vid-PRE (SFT only) | 2.97 | 45.96 | Substantial boost from explicit constraint derivation |
| Wan2.6-Flash | + Vid-PRE (SFT + RL) | 3.09 | 52.57 | Outperforms native prompt expanders by large margins |
Key Findings¶
- Superficial Fluency vs. Cognitive Collapse: Across all evaluated models, video quality scores consistently outshine reasoning metrics. For example, Wan2.2 attains 2.32 in video quality but drops to 1.56 in implicit rule adherence. Even top-tier commercial systems like Veo3.1 and Sora2 drop from ~3.5 in video quality to ~3.0 in terminal goal achievement, demonstrating that video generation models without external guidance function primarily as statistical surface interpolators.
- Generic Prompt Expanders Fail at Cognitive Tasks: Built-in prompt extensions (such as Wan PE) decorate prompts with aesthetic adjectives, camera angles, and ambient lighting, which frequently dilute key physical constraints. In contrast, Vid-PRE boosts Wan2.2's spatial cognition on V-ReasonBench from 1.32 to 36.25, proving that structured constraint formulation is essential for physical and algorithmic reasoning.
- Efficacy of Purely Text-Based Reinforcement Learning: The ablation validates that RL with text-based rewards successfully transfers to downstream video synthesis. Without synthesizing a single video frame during GRPO, the policy learns to penalize direction errors and phantom objects, directly boosting temporal rule tracking in downstream generation.
Highlights & Insights¶
- Decoupled Cognitive-Rendering Division of Labor: Rather than attempting end-to-end fine-tuning on massive diffusion backbones, Vid-PRE delegates cognitive planning to a dedicated multimodal LLM and leaves rendering to the video model, achieving large performance leaps at minimal training cost.
- Offline Text-Based Video RL Alignment: Optimizing video reasoning via purely text-grounded rewards avoids the severe training instability and GPU overhead of in-the-loop video generation while ensuring generator agnosticism.
- Multi-Level Orthogonal Diagnostics: VWG-Bench's decoupling into video quality, task constraints, and sample goals provides a standardized lens that isolates rendering fidelity from logical reasoning breakdowns.
Limitations & Future Work¶
- Open-Loop First-Frame Planning: Vid-PRE constructs the execution prompt solely from the initial frame \(I\) and prompt \(P_{\text{user}}\). If the downstream generator experiences physical drift or morphing mid-video, the static prompt cannot perform closed-loop dynamic replanning.
- Upper Bounds on Complex Symbolic Logic: On complex 9x9 Sudoku puzzles or intricate mazes, the underlying Qwen3-VL-8B may occasionally produce flawed CoT traces, necessitating the author's heuristic Difficulty Patching during dataset curation.
- Future Directions: Exploring closed-loop multi-chunk video generation with dynamic test-time monitoring, as well as incorporating lightweight keyframe latent guidance into the RL loop.
Related Work & Insights¶
- vs Perceptual Benchmarks (VBench / VBench++): VBench evaluates perceptual quality (e.g., motion smoothness, temporal flickering, aesthetic quality), whereas VWG-Bench systematically probes higher-order symbolic, embodied, and counterfactual reasoning across 9 dimensions.
- vs Specialized Reasoning Benchmarks (VR-Bench / RULER-Bench / V-ReasonBench): VR-Bench is limited to 2D mazes, RULER-Bench focuses on 6 narrow rule categories, and V-ReasonBench evaluates only last-frame terminal states. VWG-Bench unifies 38 diverse tasks with explicit progress and implicit rule verification.
- vs Generic Prompt Optimizers (Prompt-A-Video / VPO / RAPO): Existing video prompt optimizers focus on text-video aesthetic alignment and stylistic embellishment. Vid-PRE specifically targets physical constraint satisfaction and logical derivation, pioneering prompt engineering for reasoning-centric video generation.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Comprehensive introduction of the VWG-Bench think-with-video suite alongside a text-reward RL prompt reasoner.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across 6 leading models on 3 benchmarks, validated by human-judge correlation and multi-stage ablations.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally structured presentation with clear mathematical definitions, granular taxonomies, and sharp analytical insights.
- Value: ⭐⭐⭐⭐⭐ Offers both an authoritative diagnostic foundation and a highly practical, plug-and-play enhancement paradigm for future world simulators.