VERTIGO: Visual Preference Optimization for Cinematic Camera Generation

Conference: ECCV 2026
Paper: ECCV Official
Project: http://vertigo.magic-lab.tech/
Area: Alignment & RLHF (also relevant to Video Generation / Computational Cinematography)
Keywords: Camera Trajectory Generation, Visual Preference Optimization, Direct Preference Optimization (DPO), Computational Cinematography, Composition Alignment

TL;DR

VERTIGO introduces the first visual preference optimization post-training framework for camera trajectory generators by rendering 3D camera paths into 2D previews via a real-time graphics engine (Unity), scoring them using a cinematically adapted VLM via inverse captioning consistency, and applying DPO to align camera motions with shot composition, reducing target off-screen rates from 38.7% to 0.8%.

Background & Motivation

In traditional filmmaking, cinematography relies on an indispensable collaborative feedback loop between the cinematographer and the director: the cinematographer manipulates the camera according to the script, while the director continuously supervises on-screen framing, composition, and emotional resonance from the monitor to approve or veto shots. Recent advancements in generative camera systems—such as GenDoP, DIRECTOR, and CCD—have demonstrated strong capabilities in producing diverse 3D camera trajectories conditioned on textual prompts. However, these systems inherently operate in a purely geometric space and lack a "director behind the monitor." They generate motion patterns that follow learned training distributions but have no awareness of how the resulting shots actually appear on a 2D screen, leading to severe framing failures where main characters drift out of frame, compositions become off-balance, and cinematic aesthetics degrade.

Adapting reinforcement learning or preference alignment techniques (such as RLHF or DPO) to camera trajectory generators introduces two fundamental obstacles. First, raw camera trajectories are high-dimensional sequences of 3D extrinsic poses (translations and rotations) devoid of pixels or visual semantics, preventing standard vision-based reward models from evaluating them directly. Second, cinematographic quality—encompassing shot scale, viewing angles, headroom, and dynamic pacing—is highly subjective and resistant to rigid geometric heuristics, while generic Vision-Language Models (VLMs) directly assigned to output numerical scores suffer from severe score compression, lack of granularity, and mode collapse.

To bridge this chasm between geometric trajectory generation and visual quality assessment, this paper introduces a real-time graphics engine (Unity) to act as a virtual shooting stage, materializing sampled trajectories into rendered 2D preview sequences. Core idea: materialize raw camera paths into real-time rendered previews as visual rewards, evaluate shots via an inverse captioning consistency mechanism using a cinematically adapted VLM to construct preference pairs, and align the trajectory generator using DPO to achieve director-in-the-loop post-training.

Method

Overall Architecture

VERTIGO structures the computational cinematography post-training pipeline into three coordinated phases: candidate trajectory generation and engine-based preview synthesis, VLM-driven inverse semantic preference evaluation, and DPO post-training. Given a natural-language prompt \(p\) describing cinematographic intent and optional scene context, the generator samples multiple candidate camera paths. These paths are immediately rendered into 2D video sequences inside pre-built 3D scenes within Unity. A VLM adapted for cinematography acts as a virtual director, performing inverse reasoning to describe the realized camera behavior in natural language, which is compared with the original prompt in a fine-tuned embedding space to derive continuous preference ranks. Finally, Direct Preference Optimization (DPO) aligns the generator policy toward superior composition and prompt adherence while strictly preserving physical motion plausibility.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Cinematographic Prompt p + Scene Layout c"] --> B["Camera Generator π_θ<br/>Sample candidate paths {τ_i}"]
    B --> C["Unity Real-Time Graphics Engine<br/>Render 2D shot preview sequences I_i"]
    C --> D["Inverse Cinematic Reasoning & Consistency Scoring<br/>VLM captions previews and computes embedding similarity"]
    D --> E["Construct preference pairs (τ_w, τ_l)"]
    E --> F["DPO Post-Training Alignment<br/>Align visual composition and prevent distribution drift"]
    F -->|Update weights θ| B
    F --> G["High-Quality Cinematic Camera Trajectories<br/>Deploy to CG rendering or C2V diffusion generation"]

Key Designs

1. Render-in-the-Loop: Bridging Geometric Paths and Visual Space via Real-Time Engines

The primary reason previous trajectory generators failed in framing is that their loss functions only constrained coordinate errors relative to training samples without ever observing the rendered canvas. VERTIGO integrates a real-time Unity rendering pipeline that instantly translates candidate 3D camera matrices into 2D preview frames \(I_i = \{I^t_i\}_{t=1}^T\) across pre-constructed scenes. To ground this process with cinematic rigor, the authors procedurally developed the LenScript dataset in Unity, which includes diverse trajectories paired with shot dimensions (scale, angle, movement) and natural language descriptions in a subject-local coordinate frame compatible with RealEstate10k. Because graphics engine rendering is orders of magnitude faster than neural diffusion video synthesis, this design enables massive-scale online trajectory rollouts and visual evaluation at negligible compute cost.

2. Caption-Based Consistency Scoring: Escaping Discrete Regression Collapse into Latent Alignment

A critical question is how to reliably score rendered shot sequences using a foundation model. The authors compared three paradigms: tag-consistency scoring (counting matches across five canonical dimensions: shot scale, direction, angle, movement, and screen property), direct scalar regression (predicting a 0–9 score via RAFT token regression), and caption-based consistency scoring. Empirical evaluations showed that off-the-shelf and directly prompted VLMs suffer from mode collapse and low variance when outputting numerical grades. Drawing inspiration from cycle consistency, VERTIGO tasks a Qwen2.5-VL model (fine-tuned on ShotBench and adapted to Unity renders) with inverse cinematic reasoning to generate an explicit natural-language summary \(\hat{p}_i\) of the camera behavior visible in preview \(I_i\). An E5 text embedding model \(\phi(\cdot)\), fine-tuned via contrastive LoRA on LenScript film captions, encodes both the original prompt \(p\) and the inverse description \(\hat{p}_i\), computing latent cosine similarity as the continuous preference reward:

\[s_{\text{sem}}(I_i, p) = \frac{\phi(p) \cdot \phi(\hat{p}_i)}{\|\phi(p)\| \, \|\phi(\hat{p}_i)\|}\]

This strategy sidesteps calibration drift and uninformative numerical clustering, providing fine-grained, smooth preference discrimination that is robust to lighting and scene variations.

3. Reference-Regularized DPO: Policy Alignment Without an Explicit Reward Model

Applying standard reinforcement learning (like PPO) directly to continuous auto-regressive trajectory generators frequently causes policy instability, trajectory jerking, and geometric degeneration. VERTIGO sorts candidate trajectories for each prompt into pairwise preferences \((p, \tau_w, \tau_l)\) where \(s_w > s_l\) and applies Direct Preference Optimization (DPO). Defining the implicit relative reward as \(r_\theta(\tau | p) = \log \frac{\pi_\theta(\tau | p)}{\pi_{\text{ref}}(\tau | p)}\), the trajectory policy is optimized with:

\[\mathcal{L}_{\text{DPO}}(\theta) = -\mathbb{E}_{(p, \tau_w, \tau_l)} \left[ \log \sigma \left( \beta \left( \log \frac{\pi_\theta(\tau_w \mid p)}{\pi_{\text{ref}}(\tau_w \mid p)} - \log \frac{\pi_\theta(\tau_l \mid p)}{\pi_{\text{ref}}(\tau_l \mid p)} \right) \right) \right]\]

Here, \(\pi_{\text{ref}}\) is the frozen pre-trained reference generator and \(\beta\) regulates divergence from the reference policy. By increasing the relative likelihood of well-framed trajectories while implicitly regularizing against reference drift, this objective preserves natural physical camera dynamics and prevents reward-hacking behaviors without requiring an auxiliary reward network.

Loss & Training

While initial pre-training uses supervised imitation learning on LenScript, post-training optimization relies exclusively on the pairwise DPO loss. Setting \(\beta = 0.1\) delivers the optimal balance: smaller values (\(\beta = 0.01\)) under-regularize the policy and degrade framing stability, while larger values (\(\beta \ge 0.5\)) over-constrain the model and prevent effective absorption of the visual preference signals. The full preview rendering, scoring, and DPO training cycle is remarkably lightweight, converging efficiently on two NVIDIA RTX 5880 Ada GPUs.

Key Experimental Results

Main Results

Evaluation spans both geometric fidelity using CLaTr metrics (FCD, CS, Precision, Recall, Density, Coverage) on the full test set, and perceptual quality on 1,000 sampled trajectories via VBench across Unity rendering and Wan 2.2 VACE video generation. Crucially, the Missing Rate (MisR) tracks the fraction of frames where the target character is off-screen or inside the outer 20% margin.

The complete quantitative comparison from Table 1 of the paper is reported below:

Method	Trajectory FCD ↓	Trajectory CS ↑	Trajectory P ↑	Trajectory R ↑	Trajectory D ↑	Trajectory C ↑	Render MisR ↓	Render Cons. ↑	Render Aes. ↑	Video Cons. ↑	Video Aes. ↑
CCD [21]	104.60	30.03	0.58	0.01	0.41	0.10	0.999	0.820	0.460	0.901	0.296
DIRECTOR [2]	26.28	32.20	0.87	0.29	0.79	0.44	0.803	0.892	0.464	0.908	0.293
GenDoP [6]	4.17	67.98	0.92	0.72	0.93	0.78	0.387	0.904	0.515	0.967	0.722
VERTIGO (Ours)	4.22	68.40	0.92	0.73	0.93	0.78	0.008	0.908	0.518	0.969	0.735

Ablation Study

The ablations dissect the VLM preference scoring strategies, DPO temperature \(\beta\), and generalization to animated dynamic characters.

Ablation results from Table 2 of the paper (evaluated on Unity renders with VBench) are summarized below:

Experiment Group	Specific Configuration	Missing Rate MisR ↓	Consistency Cons. ↑	Aesthetic Aes. ↑	Note
Baseline	GenDoP Baseline	0.387	0.904	0.515	Pure geometric training without visual feedback
Scoring Strategy	Tag-consistency (5 dimensions)	0.327	0.902	0.516	Discrete tags fail to capture subtle framing nuances
	Direct Regression (RAFT scalar)	0.286	0.906	0.515	Score variance collapses into narrow range
	Caption Consistency (Ours)	0.008	0.908	0.518	Latent semantic matching eliminates framing failures
DPO \(\beta\) Tuning	\(\beta = 0.01\)	0.097	0.897	0.482	Insufficient reference regularization causes instability
	\(\beta = 0.1\) (Optimal)	0.008	0.908	0.518	Best trade-off between alignment and fidelity
	\(\beta = 0.5\)	0.132	0.905	0.517	Over-constrained policy limits preference learning
	\(\beta = 0.9\)	0.414	0.901	0.515	Fails to depart from baseline initialization
Dynamic Characters	GenDoP (Animated)	0.387	0.876	0.479	Baseline frequently loses moving characters
	VERTIGO (Animated)	0.008	0.902	0.488	Local frame formulation generalizes to subject motion

Key Findings

• Near-Elimination of Off-Screen Errors: GenDoP suffers from an off-screen missing rate of 38.7%, and CCD fails in 99.9% of cases. VERTIGO slashes this failure rate down to 0.8% (a 97.9% reduction) while preserving state-of-the-art geometric realism (FCD remains at 4.22 vs 4.17, CLaTr-Score improves to 68.40).
• Inverse Captioning Overcomes Numerical Degradation: Ablations confirm that discrete tag matching (MisR 0.327) and direct scalar regression (MisR 0.286) provide poor preference differentiation. Inverse natural-language captioning combined with domain-adapted embedding similarity offers smooth and highly discriminative reward signals.
• Broad Human User Preference: In a double-blind best-of-4 study involving 34 participants (11 cinematography professionals), VERTIGO achieved an overall 52.4% win rate across Unity renders and diffusion-based video transfers, excelling in composition (51.8%) and instruction adherence (53.9%).

Highlights & Insights

• Graphics Engine as a Zero-Cost Visual Critic: By utilizing a real-time graphics engine instead of slow diffusion generation inside the post-training loop, VERTIGO solves the non-differentiable visual rendering bottleneck with remarkable computational efficiency.
• Inverse Cinematic Reasoning: Rather than demanding that a VLM function as an arbitrary numerical judge, VERTIGO leverages the VLM's strength as a descriptive captioner, comparing prompt-caption pairs in a contrastively trained latent space to produce robust preference signals.
• Subject-Centric Local Trajectory Coordinate Invariance: Formulating trajectories in the subject's local frame allows the learned visual composition preferences to generalize seamlessly from static mannequins to complex animated moving characters.

Limitations & Future Work

• Upper Bound Imposed by VLM Perception: As an external evaluator, any perceptual failure or hallucination from the VLM directly contaminates the DPO training pairs and caps optimization quality.
• Scope Limited to Atomic Camera Motions: The training distribution primarily covers basic primitives (pan, tilt, dolly, zoom, orbit); complex multi-beat long-take camera blockings remain to be investigated.
• Trajectory-Only Distortion in Video Generation: Direct camera conditioning on diffusion video backbones occasionally causes subject stretching during aggressive rotation trajectories, reinforcing the current necessity of video-to-video (V2V) transfer protocols.

Related Work & Insights

• vs GenDoP [63] / DIRECTOR [2]: Both GenDoP and DIRECTOR generate trajectories from text in purely geometric coordinates without ever visualizing the camera's perspective, leading to severe character dropouts (38.7% and 80.3% MisR); VERTIGO preserves their smooth trajectory kinetics while instilling visual composition awareness.
• vs VideoDPO [33] / Control-A-Video [4]: Video diffusion DPO methods operate at the heavy 2D pixel tensor level and cannot directly output precise 3D camera matrices for cinematic previsualization; VERTIGO applies preference optimization directly to lightweight 3D camera policies.

Rating

• Novelty: ⭐⭐⭐⭐⭐ First framework to link graphics engine previews, VLM inverse captioning rewards, and camera trajectory DPO.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Exhaustive evaluation across CLaTr geometry metrics, VBench visual metrics, dynamic animated subjects, and a 34-participant user study.
• Writing Quality: ⭐⭐⭐⭐⭐ Clearly framed around cinematography principles with thorough exploration of alternative scoring designs.
• Value: ⭐⭐⭐⭐⭐ Highly practical bridge connecting generative camera planning with production-ready visual composition.

Related Papers

• [ECCV 2026] Don't Let the Video Speak: Audio-Contrastive Preference Optimization for Audio-visual Language Models
• [ACL 2025] Retrieval-Augmented Fine-Tuning With Preference Optimization For Visual Program Generation
• [ICML 2026] Autoregressive Direct Preference Optimization
• [ICML 2026] Boosting Direct Preference Optimization with Penalization
• [ACL 2025] Focused-DPO: Enhancing Code Generation Through Focused Preference Optimization on Error-Prone Points