VGGRPO: Towards World-Consistent Video Generation with 4D Latent Reward

Conference: ECCV2026
Paper: ECCV Paper
Authors: Zhaochong An, Orest Kupyn, Théo Uscidda, Andrea Colaco, Karan Ahuja, Serge Belongie, Mar Gonzalez-Franco, Marta Tintore Gazulla
Affiliations: Google; University of Copenhagen; University of Oxford; Institut Polytechnique de Paris
Area: Video Generation
Keywords: geometric consistency, latent rewards, camera motion, 4D reconstruction, GRPO

TL;DR

VGGRPO connects video latents directly to a geometry foundation model and performs GRPO post-training with camera smoothness and reprojection consistency rewards, achieving a 66.00% motion-quality win rate on dynamic scenes with Wan2.2-5B while reducing reward computation time from 54.73 s to 41.33 s.

Background & Motivation

A video diffusion model can render convincing individual frames without making them observations of one stable world. Walls may deform, background structures may drift, and camera trajectories may suddenly jitter as the viewpoint changes. Point-cloud conditioning, depth branches, or additional geometric losses can alleviate these problems, but typically require architectural changes or substantial retraining. This paper instead seeks to improve an existing generator through post-training without rebuilding its generation architecture.

Another approach scores videos with an external geometry model and aligns the generator through preference optimization. Epipolar-DPO and VideoGPA use epipolar constraints and dense geometry, respectively, but static-scene assumptions do not readily accommodate moving objects and nonrigid deformation. There is also a representation mismatch: the generator operates in VAE latent space, whereas the reward model consumes RGB. Decoding every candidate group adds repeated VAE computation and memory costs to online GRPO sampling.

The paper transfers the spatial priors of a geometry foundation model to a latent-input scoring pathway, then uses Any4D's dynamic reconstruction capability to distinguish background from moving regions. Core Idea: train a geometry model that reads video latents directly, allowing online GRPO to reward both stable cameras and cross-view consistency of static structures in dynamic videos without repeatedly decoding RGB.

Method

Overall Architecture

VGGRPO is a post-training framework for text-to-video generation, not a system whose final output is a reconstructed point cloud. Training first establishes a Latent Geometry Model (LGM), then generates a group of fully denoised video latents for each prompt. LGM predicts cameras, depths, point maps, and scene flow; these predictions produce two rewards that guide updates to the generator's LoRA parameters.

The latent geometry model turns existing geometry priors into a scoring interface, the camera smoothness reward constrains camera trajectories, the reprojection consistency reward constrains static structure, and separately normalized GRPO converts the differently scaled scores into comparable group-relative advantages. RGB decoding is removed from post-training reward evaluation, not from the final process of producing a viewable video.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Prompt and a group<br/>of generated video latents"] --> LGM["Latent geometry model"]
    LGM --> Motion["Camera smoothness reward"]
    LGM --> Geometry["Reprojection consistency reward"]
    Motion --> Update["Separately normalized GRPO"]
    Geometry --> Update
    Update --> Output["Update generator LoRA<br/>and continue online sampling"]

Key Designs

1. Latent geometry model: bypass RGB input while retaining downstream geometry reasoning

The original geometry model consumes RGB images, whereas the generator supplies VAE latents, so the two cannot be connected directly. A lightweight 3D convolutional connector replaces the first several geometry-model layers, mapping latents into an intermediate feature space before the remaining Transformer layers and geometry prediction components. This creates a separate reward pathway rather than adding a new generator branch that must participate in sampling.

Connector training uses two paths for the same calibration video: the original geometry model produces intermediate features, while the VAE encoder and connector produce features to be aligned with them. The stitching layer and connector parameters are jointly selected using feature alignment error. The connector and downstream geometry layers are then fine-tuned together to align latent-input predictions with the original model's camera, depth, point-map, and other outputs. Equation (7) expresses the architecture as:

\[ \widehat{\Phi}_{\psi}=\Phi_{\hat{\ell}+1:L}\circ S_{\psi}. \]

Here \(S_{\psi}\) is the connector, \(\hat{\ell}\) is the stitching position, and \(\Phi_{\hat{\ell}+1:L}\) is the retained downstream geometry network. The main configuration uses Any4D, which provides scene flow in addition to cameras, depths, and world-frame point maps; the VGGT variant primarily serves static scenes. The cache does not include the supplement and does not specify the final stitching-layer index, calibration-set size, or output-alignment weights, so it cannot support a complete reproduction configuration.

2. Camera smoothness reward: penalize trajectory jitter rather than camera speed itself

LGM estimates camera poses from fully denoised latents. World-frame camera centers give consecutive displacements as discrete velocities, and differences between consecutive velocities give accelerations. The translational error divides the acceleration norm by the sum of neighboring velocity norms to reduce sensitivity to reconstruction scale. The following reformats Equations (11) and (13) using their accompanying prose; some mathematical layout is missing from the cached extraction.

\[ e_{\mathrm{trans}}=\frac{1}{N-2}\sum_{i=2}^{N-1}\frac{\|\mathbf a_i\|_2}{\|\mathbf v_i\|_2+\|\mathbf v_{i-1}\|_2}, \qquad r_{\mathrm{motion}}=\frac{1}{2}\left(\frac{1}{1+e_{\mathrm{trans}}}+\frac{1}{1+e_{\mathrm{rot}}}\right). \]

Here \(N\) denotes the number of frames to avoid confusion with the denoising-step count; \(\mathbf v_i=\mathbf c_{i+1}-\mathbf c_i\) and \(\mathbf a_i=\mathbf v_i-\mathbf v_{i-1}\). The rotational error \(e_{\mathrm{rot}}\) uses the same normalization, replacing velocity with \(\boldsymbol\omega_i=\log_{SO(3)}(R_i^\top R_{i+1})\) and acceleration with differences between consecutive angular velocities. Both errors are mapped through \(1/(1+e)\) and averaged, approaching 1 for smoother trajectories.

Because the reward targets changes in velocity, a constant-speed push-in is not directly penalized for traveling a large distance. However, it cannot determine whether a sudden turn is artistically appropriate for the prompt. The main text does not explain denominator stabilization for completely stationary cameras, so no numerical stabilizer should be presented as an author-specified setting. These are frame-difference scoring quantities, not physical accelerations calibrated by frame rate.

3. Reprojection consistency reward: expose local structural failures through the worst views

A smooth camera trajectory does not guarantee correct walls or road geometry. The second reward aggregates world-frame point maps into a scene point cloud, projects it into each view with the predicted cameras, and compares rendered depth with the depth directly predicted for that view. All points can be aggregated for a static video. For a dynamic video, predicted scene flow filters out moving regions, and only static points are aggregated, avoiding the interpretation of a moving object's different positions as contradictory background geometry.

Valid projected pixels form the set \(\Omega_i\). Equations (14) and (15), reformatted according to the prose describing depth differences and worst-view aggregation, are:

\[ e_{\mathrm{geo}}^{(i)}=\frac{1}{|\Omega_i|}\sum_{\mathbf p\in\Omega_i}\left|\widehat D_i(\mathbf p)-D_i(\mathbf p)\right|, \qquad r_{\mathrm{geo}}=-\frac{1}{3}\sum_{i\in\operatorname{top3}(e_{\mathrm{geo}})}e_{\mathrm{geo}}^{(i)}. \]

The top3 operator selects the 3 views with the largest errors, not the 3 highest-scoring candidate videos. Focusing on the worst views prevents brief local failures from being diluted by many normal frames. Depth errors retain the scale of predicted depths; the main text does not specify their physical unit or the dynamic-filtering threshold, so they are not labeled as meters here. In this formulation, a "4D reward" primarily uses dynamic reconstruction to identify reliable static structure, rather than directly constraining the full deformation trajectories of moving objects.

4. Separately normalized GRPO: compare candidates within a group before updating latent denoising

For each prompt, the current policy samples \(K\) complete denoising trajectories, and both rewards are computed from each trajectory's final clean latent. The motion reward is bounded, whereas the geometry reward is a negative depth error. Direct addition could let one dominate merely because of numerical scale. Equation (16) therefore standardizes each reward using its own group mean and standard deviation, then averages the standardized values:

\[ A^k=\frac{1}{2}\left(\frac{r_{\mathrm{motion}}^k-\mu_{\mathrm{motion}}}{\sigma_{\mathrm{motion}}}+\frac{r_{\mathrm{geo}}^k-\mu_{\mathrm{geo}}}{\sigma_{\mathrm{geo}}}\right). \]

This advantage drives GRPO updates across the trajectory: the algorithm computes the new-to-old policy ratio for latent transitions, takes the smaller of the unclipped and clipped advantage-weighted terms, and applies a KL penalty against a reference model. Stochastic exploration follows the ODE-to-SDE conversion used by flow-based GRPO. The central change is evaluating rewards and policy transitions in latent space, not introducing a new preference dataset.

Group-relative comparison removes the need for perfectly calibrated absolute rewards across prompts. Nevertheless, training depends on LGM reliably ranking candidate quality, and the main text does not describe handling zero within-group variance. Self-consistency under this scorer should not automatically be equated with correct real-world geometry.

A Worked Example

Consider the paper's illustrated tracking shot of a speeding car on a road. The same prompt produces 64 candidate latent videos, and LGM estimates camera trajectories, road point maps, and moving regions for each. One candidate might depict convincing car motion but gradually distort the road and background wall; another might preserve the background but introduce camera jumps. The two rewards identify these different problems separately.

Scene flow first helps exclude the moving car from background point aggregation. Reprojection then checks static areas such as the road and emphasizes the 3 views with the largest errors, while the camera reward independently checks changes in trajectory velocity. The two group-standardized scores jointly determine each trajectory's advantage, and LoRA updates increase the probability of candidates with better combined performance. This is a mechanism illustration, not a claim that the paper reports individual measured scores for this group of 64 candidates.

Loss & Training

LGM is trained for 20 epochs on a mixture of videos synthesized by the base generator and real videos from DL3DV, RealEstate10K, and MiraData. Feature stitching precedes geometry-output alignment; generator post-training uses prompts from the same three datasets.

The main text names the backbones Wan2.1-1B and Wan2.2-5B. This note preserves those names instead of substituting a different parameter scale. Post-training uses LoRA with rank 32 and scaling factor 64, groups of 64 samples, and AdamW with both learning rate and weight decay set to \(1\times10^{-4}\). Group size is distinct from the efficiency experiment's batch size of 4.

The paper also demonstrates training-free test-time gradient guidance: latent reward guidance is applied once every 20 steps during a total of 50 denoising steps. This additional use case is distinct from the LoRA-post-trained models in the main results. The cache lacks the corresponding supplementary implementation and does not provide a full specification of resolution, frame count, or hardware.

Key Experimental Results

Main Results

Evaluation uses 190 static-scene prompts and 200 dynamic-scene prompts. VQ/MQ are VideoReward evaluator win rates against the base model for visual/motion quality, expressed as percentages; they are not newly collected human votes in this paper. Sampson epipolar error is evaluated only on the static split. General VBench quality metrics are computed across all 390 prompts.

The following selection is from Table 1. Higher VQ/MQ and lower epipolar error are better; the main text does not specify the error's unit.

Backbone / post-training	Static VQ (%)	Static MQ (%)	Static epipolar error	Dynamic VQ (%)	Dynamic MQ (%)
Wan2.1-1B / Epipolar-DPO	54.21	55.79	0.098	45.50	43.00
Wan2.1-1B / VideoGPA	53.68	56.32	0.105	42.50	41.00
Wan2.1-1B / VGGRPO	59.47	66.84	0.102	57.00	63.00
Wan2.2-5B / Epipolar-DPO	52.11	58.95	0.101	38.00	54.50
Wan2.2-5B / VideoGPA	54.74	60.53	0.098	40.00	54.00
Wan2.2-5B / VGGRPO	62.63	68.42	0.093	56.50	66.00

For Wan2.2-5B, dynamic MQ exceeds VideoGPA by 12.00 percentage points, but this is the difference between their respective win rates against the base model, not a direct head-to-head win rate. Wan2.1-1B's epipolar error of 0.102 remains above Epipolar-DPO's 0.098, so the results do not support claiming the best score on every metric.

Ablation Study

Tables 2(a) and 2(b) use Wan2.2-5B. Their VQ/MQ values summarize the static and dynamic splits and must not be confused with the dynamic-only values in Table 1.

Config	VQ (%)	MQ (%)	Static epipolar error	Analysis
VGGT geometry model	54.96	60.61	0.090	Table 2(a), trained only on static prompts
Any4D, camera smoothness reward only	55.60	63.40	0.104	Table 2(b), no reprojection reward
Any4D, both rewards	59.57	67.21	0.093	Tables 2(a) and 2(b), full configuration

Adding reprojection raises MQ from 63.40% to 67.21% and reduces epipolar error from 0.104 to 0.093. VGGT yields lower static error, while Any4D yields higher aggregate VQ/MQ. This comparison changes both dynamic-scene support and the training-prompt scope, so the difference cannot be attributed solely to geometry-backbone architecture.

The following reward-computation measurements are from Table 2(e), with batch size 4. They do not measure total training time or end-to-end video generation time.

Reward computation	Time (s)	Peak GPU memory (GB)
RGB-based	54.73	76.80
Latent rewards	41.33	68.57

Key Findings

• Reward computation saves 13.40 s, a 24.5% time reduction, and peak memory decreases by 8.23 GB. These measurements do not include the upfront cost of training LGM separately.
• On standard VBench prompts in Table 2(d), motion smoothness improves from 0.9841 to 0.9895, while Dynamic Degree falls from 0.4237 to 0.3962. The authors attribute this to reduced camera jitter lowering RAFT optical-flow magnitude, but this metric alone cannot establish that all meaningful object motion is preserved.
• Training-free guidance in Table 2(c) reduces epipolar error from 0.142 to 0.136 while increasing runtime from 44.35 s to 62.60 s. It has its own inference overhead and should not be assigned Table 2(e)'s reward speedup percentage.

Highlights & Insights

• Geometry foundation models need not remain post-hoc RGB evaluators. Intermediate-feature stitching turns them into reward interfaces for a generator's native latent space and avoids repeatedly decoding candidate groups.
• Smooth trajectories and consistent structures are separate requirements. The ablation shows that stabilizing the camera alone can leave wall geometry errors, precisely the local failures targeted by worst-view reprojection.
• Dynamic reconstruction is used to identify points that should not be aggregated, not to force moving objects to remain stationary. This principle can transfer to other tasks that need reliable background constraints in dynamic videos.

Limitations & Future Work

• The main text has no dedicated limitations section. The following points are analysis based on the method and evaluation scope, not author-verified failure findings.
• Dynamic points are excluded from reprojection rewards, so the results do not establish physically correct articulated motion, collisions, or nonrigid deformation. Dynamic correspondences or trajectory constraints are possible extensions.
• Rewards depend on cameras, point maps, and depths predicted by LGM itself, allowing shared prediction biases or reward exploitation. Independent geometry evaluation and human inspection could distinguish internal consistency from actual correctness.
• Evaluation covers two Wan backbones and a limited held-out prompt set without reported confidence intervals. Handheld footage or deliberately abrupt camera turns also require separate tests of the smoothness preference.
• The cache contains the complete main paper and references but not the supplement. Stitching details, dynamic thresholds, denominator stabilization, and the complete training budget remain unverified, preventing a claim of full reproducibility.

Related Work & Insights

• vs Epipolar-DPO / VideoGPA: These methods primarily construct static-geometry preferences in RGB space. VGGRPO instead uses online group sampling, latent geometry rewards, and scene flow to handle dynamic regions, although its smaller backbone does not universally lead on static epipolar error.
• vs Flow-GRPO / DanceGRPO: VGGRPO builds on the existing formulation of generation as policy optimization. Its contribution centers on the latent geometry scoring pathway and complementary rewards rather than a redefinition of GRPO.
• vs Geometry Forcing / explicit geometry branches: VGGRPO retains the generator's core architecture and introduces geometric preferences through LoRA post-training. Geometry-modeling cost shifts to separately training and running LGM rather than disappearing entirely.

Rating

• Novelty: 4/5. Latent geometry stitching and dynamic-aware rewards form a clear combination, while policy optimization follows existing frameworks.
• Experimental Thoroughness: 4/5. Two backbones, two scene types, reward ablations, and efficiency studies are included, but independent dynamic-geometry ground truth is missing.
• Writing Quality: 4/5. The main argument and reward motivation are clear, but implementation details rely on a supplement absent from the cache.
• Value: 4/5. Directly useful for geometry-aware video post-training, without establishing a complete physical-world simulator.