Taming Camera-Controlled Video Generation with Verifiable Geometry Reward¶

Conference: ECCV2026
Paper: ECCV Paper
Area: Video Generation
Keywords: camera control, verifiable geometry reward, relative pose, GRPO, dynamic scenes

TL;DR¶

CamVerse converts the reconstructed 3D camera trajectories of generated and reference videos into segment-level relative pose rewards for online GRPO; Table 1 reduces T2V translation error from the SFT baseline's 0.0395 to 0.0293, but I2V perceptual quality does not improve alongside it.

Background & Motivation¶

Camera-controlled video generation requires a model to render the requested scene and translate a specified 3D camera path into frame-by-frame viewpoint changes. CameraCtrl, MotionCtrl, and AC3D already inject camera parameters or ray representations into generators, but supervised fine-tuning (SFT) remains the dominant training approach. SFT learns from fixed text-video-camera pairs without directly checking whether the model's own generated videos execute the instruction. A video can therefore appear realistic and roughly follow the intended direction while continuing straight at a turn or imitating camera motion through implausible scene deformation. Training sets dominated by static scenes can also weaken subject motion, making it difficult to handle camera movement alongside independently moving people or objects.

Online reinforcement learning (RL) can repeatedly sample the current model and increase the probability of outputs that execute the control successfully, but constructing the reward is difficult. Generic image or clip quality scores do not identify the short interval in which the camera deviates; trajectory estimation also introduces scale ambiguity and reconstruction noise. Direct absolute trajectory comparison may favor globally similar but locally inconsistent motion, leaving locally correct samples with accumulated drift without useful feedback. The question is consequently not how to add another camera-conditioning interface, but how to extract sufficiently reliable and detailed geometric feedback from generated outputs.

The authors use a large feed-forward 3D model to estimate camera poses, remove scale and coordinate-frame discrepancies, and compare relative motion within short intervals. This makes the reward more directly reflect the required local movement and allows correct and incorrect segments within the same video to receive different feedback. Core Idea: turn 3D trajectory consistency into confidence-filtered, segment-level relative pose rewards, then inject this geometric prior into a video generator through online GRPO.

Method¶

Overall Architecture¶

Inputs are a text prompt, a target camera trajectory, and the first frame in the I2V setting; the output is a video following that trajectory. CamVerse first performs supervised fine-tuning on diverse camera-annotated videos, then samples multiple candidate videos from the current generator under identical conditions. During training, Trajectory Reconstruction and Alignment, Segment-Level Relative Pose Reward, and Group-Relative Policy Update form a feedback loop; the reference video is not a mandatory inference input. The reference video supports trajectory comparison during training, whereas deployment retains the generator and camera conditioning without reward evaluation or policy updates.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Text, target trajectory,<br/>and optional first frame"] --> Generator["SFT-initialized generator"]
    Generator -->|Training sampling| Samples["Same-condition candidate videos"]
    Samples --> Align["Trajectory Reconstruction<br/>and Alignment"]
    Reference["Training reference video"] --> Align
    Align --> Reward["Segment-Level<br/>Relative Pose Reward"]
    Reward --> Update["Group-Relative<br/>Policy Update"]
    Update -.->|Training feedback| Generator
    Generator -->|Inference| Output["Camera-controlled video"]

The base generator is an internally pretrained diffusion Transformer with approximately 2B parameters, paired with a spatiotemporal VAE that compresses time by 4 and space by 8. Keyframe camera poses are interpolated to the video frame count and converted into per-pixel Plücker ray representations for a lightweight camera network. Each ray contains its direction and the cross product of camera center and direction, producing a 6-dimensional geometric condition per pixel instead of merely concatenating global extrinsic parameters. This representation connects camera orientation and position to pixel rays and is easier to combine with video features, but does not itself guarantee accurate trajectory execution.

The dataset contains 315k videos, with 314k used for training and 1k held out for testing, covering real footage and game environments. The authors emphasize large-amplitude camera movement and varied subject dynamics, using reconstruction and multistage filtering to remove failures and broaden the trajectory distribution. These data establish the generator's baseline capabilities; the geometry reward further selects more accurate execution patterns within its current output distribution. The supplied paper does not provide all filtering thresholds, so the complete data construction pipeline cannot be reproduced from this description alone.

Key Designs¶

1. Trajectory Reconstruction and Alignment: remove scale ambiguity before judging control errors

Feed-forward 3D reconstruction is applied separately to generated and reference videos to obtain per-frame camera-to-world extrinsics and pixel confidence. The paper's 3D model ecosystem includes π3; the evaluation section explicitly uses π3 for trajectories and geometric confidence. Even when two videos execute the same motion, their reconstructed camera positions may have different scales and coordinate frames, which direct subtraction would mistake for generation errors. The authors therefore estimate an Umeyama similarity transform from the two sets of camera centers, jointly recovering scale, rotation, and translation. The transform is applied consistently to the generated trajectory: camera centers receive scale, rotation, and translation, whereas orientation matrices receive only the alignment rotation. This preserves the meaning of rotation matrices and places subsequent comparisons in the reference trajectory's coordinate frame.

Here, verifiable means that geometry can be re-estimated from generated video and converted into an error, not that sensor-grade ground-truth poses are available. Reconstruction can still fail under blur, occlusion, or dominant moving objects, so confidence is also used to exclude unreliable segments. Alignment makes the evaluation insensitive to a uniform scale change; a small error cannot directly be interpreted as accurate recovery of motion distances in meters. The constraint consequently concerns aligned motion consistency, not uncalibrated absolute physical scale.

2. Segment-Level Relative Pose Reward: divide a trajectory into comparable local motions

After alignment, both trajectories are divided into non-overlapping short segments with matching temporal boundaries, and each segment's start-to-end relative transform is compared. The following retains the core relation from Equation (7): tildes denote generated poses, primes denote alignment, and hats denote the reference.

\[ \widetilde{T}_k=(\tilde{E}'_{n_k})^{-1}\tilde{E}'_{n_k+L},\qquad \hat{T}_k=(\hat{E}_{n_k})^{-1}\hat{E}_{n_k+L}. \]

Multiplying the inverse reference relative transform by the generated relative transform gives the segment's error transform. Its translation component determines translation error, while the trace of its rotation matrix determines angular distance; smaller errors indicate closer adherence to the desired camera movement. The paper clips translation error and numerically clips the argument of the inverse cosine used for rotation to limit outliers or invalid numerical values. The alignment score is the negative weighted sum of translation and rotation errors, corresponding to Equation (9):

\[ s_k=-\bigl(\lambda_t e_t(k)+\lambda_R e_R(k)\bigr). \]

A score closer to zero indicates less geometric deviation; this is not a binary reward in which only positive values indicate success. Generated-video pixel confidence is pooled into frame confidence, and a threshold masks unreliable segments so that reconstruction failures do not directly dominate policy updates. The text does not clearly specify frame-to-segment aggregation, segment length, or the actual reward weights; this note does not invent those hyperparameters. Local relative motion reduces the influence of accumulated deviations on individual segment scores, but an endpoint transform is not equivalent to checking every intermediate frame for smoothness. Dense therefore means finer feedback than a single clip-level score, not independent control supervision for every pixel or frame. The ablations compare relative/absolute error combinations and clip-level rewards, separating the roles of error definition and reward granularity.

3. Group-Relative Policy Update: reinforce successful current-policy samples while limiting drift

Once a group of candidate videos has been sampled under the same condition, each receives a sequence of segment scores. The authors use the mean and standard deviation of all reward values in the group for z-score normalization, converting scores into relative advantages. The key in Equation (10) is whether a sample is better or worse within its group, rather than whether all conditions clear one potentially unbalanced fixed success threshold:

\[ A_k^g=\frac{s_k^g-\mu}{\delta+\gamma}. \]

Here \(g\) indexes candidate videos, \(k\) indexes segments, \(\mu\) and \(\delta\) are group statistics, and \(\gamma\) provides numerical stability. The actual reward pipeline includes the confidence filtering described above; the paper does not use the notation for raw and masked scores entirely consistently. The update then uses new-to-old policy probability ratios for denoising transitions and a GRPO/PPO-style clipped objective to avoid excessive changes in a single update. The SFT model serves as a frozen reference policy, with KL regularization discouraging the generator from sacrificing its original capabilities to maximize geometric scores. This process does not require pixel-level gradients through the 3D reconstructor; it updates the policy through sampled-trajectory rewards and policy probabilities. The objective sums over candidates, denoising steps, and segments, but does not clearly expose finer spatial or temporal gradient assignment, so pixel-level credit assignment should not be inferred.

To make exploration practical, training mixes stochastic and deterministic sampling instead of making every denoising step incur stochastic exploration costs. Each group contains 16 generated samples, from which the highest-reward 4 and lowest-reward 4 are selected for optimization, preserving learning signals from successes and failures. This is how online post-training produces learning samples, not an inference algorithm that necessarily generates 16 candidates at deployment. Training cost and single-generation cost must therefore be distinguished; the duration of a GRPO optimization step is not an inference latency measurement.

A Worked Example¶

Consider a first frame showing a road and a target camera path that moves diagonally forward-right before changing viewing direction; this is an explanatory example, not an additional experiment. Two candidates may both render a clear road, but one initially moves only straight ahead while the other better executes the diagonal displacement. Reconstruction and alignment first remove possible overall scale discrepancies, after which the relative start-to-end poses of matching short segments are compared. The straight-moving candidate receives a larger translation error for the relevant segment, while an inaccurate turn is penalized through the rotation term. A segment with insufficient reconstruction confidence is masked instead of being treated as definite evidence of failure. The group-relative update increases the probability of better sampled trajectories while reference-policy regularization limits visual degradation; the failure in Figure 4 also shows that straight-motion bias remains.

Loss & Training¶

SFT uses a flow-matching velocity prediction objective and updates all model parameters for 10k iterations with a batch size of approximately 128. Videos are resized to a 512-pixel longer side with their aspect ratios preserved, last 2–10 seconds, and are processed by a packing dataloader supporting variable lengths. The task mixture is 30% T2V and 70% I2V; text conditioning is dropped with probability 0.3, while camera conditioning is always retained. AdamW uses a learning rate of \(5\times10^{-5}\), weight decay of 0.01, and a numerical stability term of \(10^{-15}\); the SFT timestep shift is 5. GRPO performs I2V post-training on approximately 3.2k training samples for 200 iterations, adding LoRA to all linear layers in Transformer blocks with rank 64 and scaling parameter 128. There are 14 denoising steps: the first 3 use SDE sampling with noise level 0.7, and the remainder use a first-order ODE sampler; timestep shift is 6 and CFG is 3.5. The KL weight is \(10^{-4}\) and experiments use 32 NVIDIA H200 GPUs; this does not establish lightweight single-GPU training. The paper explicitly describes GRPO as targeting I2V yet reports post-training T2V results without fully explaining the training/evaluation connection between tasks; a separate equally sized T2V RL run should not be assumed.

Key Experimental Results¶

Main Results¶

Table 1 appears on page 9; CamVerse* is the SFT baseline and CamVerse includes online RL post-training. Its caption specifies a mixed test set, described in Section 4.2 as 1k clips comprising 342 RealEstate10K clips and 658 curated clips, each lasting 4–8 seconds. However, Section 4.3 calls the Table 1 evaluation the RealEstate10K test set; this note follows the caption while explicitly retaining the conflict, rather than treating the two sets as verified equivalents.

Task	Method	Translation Error ↓	Rotation Error ↓	Geometric Consistency ↑	VQ ↑
T2V	CameraCtrl	0.0887	1.4586	0.7567	2.93
T2V	AC3D-2B	0.0476	1.0451	0.8156	3.82
T2V	AC3D-5B	0.0428	0.9120	0.8820	4.69
T2V	CamVerse*	0.0395	0.6506	0.9081	5.30
T2V	CamVerse	0.0293	0.5140	0.9173	5.91
I2V	CameraCtrl	0.1696	3.2809	0.6087	2.63
I2V	CamVerse*	0.0337	0.5613	0.9174	5.02
I2V	CamVerse	0.0286	0.4685	0.9226	4.87

Translation error is the average RMSE between corresponding camera positions after similarity alignment, and rotation error is the mean angular difference between orientations; the table does not explicitly state the angular unit, so neither degrees nor radians are added here. Geometric consistency is the fraction of pixels with π3 confidence above 0.1, not ground-truth point-cloud error or a manually annotated geometric correctness rate. VQ uniformly samples 8 frames per video and computes HPSv3 with a fixed high-quality, clear-frame prompt; it measures sampled-frame perceptual quality rather than complete temporal quality.

Ablation Study¶

Table 3 appears on page 12 and, following Section 4.4, evaluates I2V on a 100-clip RealEstate10K subset. The SFT result of 0.0280/0.4444 belongs to this ablation setting and must not be combined directly with Table 1's SFT values to calculate improvements.

Config	Translation Reward Error	Rotation Reward Error	Translation Error ↓	Rotation Error ↓
SFT baseline	Not applicable	Not applicable	0.0280	0.4444
Mixed reward	Relative	Absolute	0.0292	0.4537
Mixed reward	Absolute	Relative	0.0293	0.4281
Segment-level reward	Relative	Relative	0.0238	0.3286
Clip-level reward	Relative	Relative	0.0277	0.3739

Key Findings¶

In Table 1, T2V translation error falls from 0.0395 to 0.0293, a 25.8% relative reduction reported by the authors; this within-baseline comparison is central evidence for the benefit of post-training.
In Table 3, segment-level relative rewards achieve a rotation error of 0.3286 versus 0.3739 for clip-level rewards; neither mixed absolute/relative configuration matches the fully relative segment-level design.
Table 4 on page 12 reports that 3/14 SDE steps achieve translation/rotation errors of 0.0238/0.3286 at 343s per optimization step; 1/14 and 5/14 steps take 274s and 412s, respectively, so more exploration is not always better.
I2V VQ falls from 5.02 to 4.87, showing that a purely geometric reward does not automatically solve perceptual quality; the abstract's overall improvement claim should not be read as improvement on every metric.

Highlights & Insights¶

The reward does not require additional human preferences for every video: it converts a 3D model's existing capabilities into a computable training signal. Its value lies in checking control execution from outputs rather than merely strengthening input conditioning.
Aligning first and comparing local motion afterward separates reconstruction scale ambiguity from control deviations. This reduces irrelevant penalties while also defining which physical quantities can be verified.
Segment rewards and SDE exploration address different problems: one provides finer feedback and the other produces differing outputs to compare. The ablations suggest that neither can simply be replaced by a larger sampling budget.

Limitations & Future Work¶

Figure 4 and Section 4.3 identify a remaining forward-motion bias: the model may initially move straight when forward-right motion is requested, so broader trajectory data have not eliminated the bias.
The authors acknowledge the slight I2V perceptual quality decrease, suggest multi-objective rewards, and propose future extensions to world-model action control and embodied data generation.
From an evaluation perspective, training rewards and geometric metrics both rely on learned 3D estimation, so higher confidence does not independently establish more accurate geometry; independent pose sources or multiple evaluators would strengthen validation. This is a reader interpretation.
Reproducibility remains limited by the internal pretrained backbone, the 32-H200 training setup, and reward parameters that are not explicitly listed; the paper also provides no cross-seed error intervals.
Several equations in the cached extraction have formatting damage, and the prose near Equation (8) swaps the rotation and translation labels; this note retains only identifiable core relations and explains them by matrix meaning without inventing a complete loss or missing hyperparameters.

vs CameraCtrl / MotionCtrl: These methods provide camera or motion conditioning interfaces; CamVerse performs online post-training using output-derived geometry on top of such conditional generators, so the approaches are not mutually exclusive architecture choices.
vs AC3D: AC3D studies camera-control interfaces and training design; CamVerse's SFT also benefits from data and training recipes, but its additional contribution is best judged through its own SFT-to-RL comparison rather than cross-backbone rankings alone.
vs Flow-GRPO / DanceGRPO: These methods provide foundations for group-relative RL in generation; the task-specific contribution here is scale alignment, relative poses, and segment feedback, not a reinvention of GRPO.
vs π3 / VGGT: Reconstruction models supply geometric feedback rather than generate the video. A transferable direction is converting reliable task measurements into local rewards while auditing evaluator bias and reward hacking.

Rating¶

Novelty: 4/5. Segment-level verifiable geometry rewards specifically address camera control, while policy optimization largely follows established methods.
Experimental Thoroughness: 4/5. Main comparisons and multiple ablations cover the central design choices, but conflicting test-set descriptions, missing uncertainty intervals, and no independent geometry evaluation limit completeness.
Writing Quality: 3/5. The central pipeline is clear, but reward details, connections between task settings, and some notation require clarification.
Value: 4/5. The work demonstrates a practical post-training route from 3D priors to improved video control while exposing the need to manage perceptual quality trade-offs.