Beyond Inpainting: Unleash 3D Understanding for Stable Camera-Controlled Video Re-rendering¶

Conference: ECCV2026
Paper: Official Page | PDF
Code: https://github.com/FREDZEL2020/DepthDirector
Area: Video Generation
Keywords: Camera control, video re-rendering, depth conditioning, dual-stream conditioning, identity preservation

TL;DR¶

DepthDirector controls novel viewpoints with warped depth and preserves appearance and motion through the source video, avoiding erroneous textures inherited from warped RGB; its full model achieves RE 1.010, TE 0.053, and CamMC 1.434 under the absolute-camera protocol.

Background & Motivation¶

Changing the camera trajectory of an existing video is more constrained than generating camera motion from text: the person must remain the same, and expressions and actions must occur at the original moments. Methods such as ReCamMaster use camera poses as implicit conditions, leaving the network to learn their relationship with scene geometry. Another family reconstructs per-frame geometry, warps the source video to the target camera, and asks a diffusion model to fill the holes. This provides direct control but turns geometric errors into visible textures within the generation condition.

The paper calls this the Inpainting Trap: when warped RGB already looks nearly complete, the model can reuse it and focus on missing regions. Distorted facial structures are then difficult to correct, while reflections become baked-in surface textures instead of changing with the viewpoint. Even an additional source-video input may be underused if copying the warped condition is the easier solution.

The aim is not to discard explicit geometry but to restrict what it communicates. Geometry should describe the viewpoint and layout, while appearance and motion come from the source video without target-view warping. Core idea: replace warped RGB with warped depth and inject viewpoint and content through separate streams, forcing the video diffusion model to jointly interpret geometry and source content when synthesizing novel views.

Method¶

Overall Architecture¶

The inputs are a monocular source video and a per-frame target camera trajectory; the output is a novel-view video preserving the original scene dynamics. At inference, the system builds per-frame dynamic meshes, renders target-view depth and occlusion masks, and injects them as view conditions into the target noise tokens. The source video independently supplies content tokens, which enter the video diffusion Transformer together with the target sequence.

Synchronized MultiCamWarp data provides actual novel-view supervision instead of merely teaching the model to fill holes in the source view. A lightweight adapter learns to combine the conditions on a frozen Wan backbone. Dataset construction and adapter training are training-side operations, not steps repeated for each inference request.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Source video and target trajectory"] --> Geometry["Warped-Depth Conditioning"]
    Input -->|Source video| Dual["View-Content Dual Stream"]
    Geometry -->|Depth and masks| Dual
    Data["MultiCamWarp<br/>Synchronized Multi-View Data"] -->|Training samples| Adapter["Lightweight Adapter Training"]
    Dual -->|Condition fusion| Adapter
    Adapter -->|Trained model| Output["Novel-view video"]

Key Designs¶

1. Warped-Depth Conditioning: supply layout without fixing appearance

The full model uses \(\pi^3\) to estimate source-camera extrinsics and per-frame point maps, then connects the three-dimensional points associated with neighboring pixels into meshes. Compared with directly projecting isolated points, this helps reduce noisy artifacts in rendered depth; it does not make the reconstruction exact. Rendering each mesh from the target camera produces depth and occlusion masks. Regions that were unobserved or outside the source image coverage receive a zero mask, informing the model about both the target layout and the absence of reliable source observations.

Depth does not contain facial color, clothing texture, or reflection patterns, so it cannot directly bake these appearances into the target condition. To reuse the pretrained video VAE, the method normalizes depth to \([0,1]\) in log space and maps it to RGB with a predefined colormap. This RGB representation encodes depth values, not warped scene colors. It still carries geometric errors but no longer tells the network to preserve erroneous appearance pixels. The cached main text does not identify the colormap, so a particular colormap cannot be specified from this evidence.

2. View-Content Dual Stream: establish correspondences through attention

The view stream separately VAE-encodes depth and masks, concatenates their latents along the channel dimension, and linearly projects them into view tokens. These are added to patchified target noise tokens. Rather than appending camera parameters to a text prompt, this introduces spatial guidance at target-aligned token locations. The following restates the operations described in Section 3.3; some equality signs and brackets are corrupted in the cached equations, so this is not a verbatim transcription:

\[ \hat{x}_{\mathrm{target}} =\operatorname{patchify}(x_t) +F_c\!\left([\mathcal E(D^r),\mathcal E(M^r)]_{\mathrm{channel}}\right). \]

Here \(D^r\) and \(M^r\) are rendered depth and masks, \(\mathcal E\) is the VAE encoder, and \(F_c\) is the linear projection. The content stream does not warp the source video. Instead, it tokenizes its VAE latent and concatenates it with target tokens along the frame dimension:

\[ \hat{x}_{\mathrm{input}} =\left[\operatorname{patchify}(x_s),\hat{x}_{\mathrm{target}}\right]_{\mathrm{frame}}. \]

Here \(x_s\) denotes the source-video latent. If the compressed source video has \(f\) temporal positions, the combined input has \(2f\) positions, doubling the input token count relative to ordinary target-video generation. This is neither pixelwise blending of source and target frames nor treating the source as extra color channels. Spatiotemporal self-attention reads across both sequences to retrieve corresponding content across viewpoints. The view stream specifies the geometry expected at the target location; the content stream specifies the object's appearance and current motion. Neither is sufficient by itself.

This also explains why adding source-video tokens beside warped RGB is insufficient: when the target condition already supplies copyable textures, attention can underuse the source. The paper visualizes more concentrated attention from target patches to corresponding source regions under depth conditioning. This supports the proposed explanation, but it is not a formal demonstration of complete three-dimensional reasoning.

3. MultiCamWarp Synchronized Multi-View Data: supervise re-rendering rather than hole filling

The dual-stream mechanism needs targets showing the same moment from a different camera to learn novel-view synthesis from source content. The authors arrange environments and animated characters in Unreal Engine 5, synchronize multiple cameras, and use 8 random trajectories per dynamic scene while rendering RGB and ground-truth depth. They then construct source-video, warped-depth, and target-video training pairs. Actual target views are not constrained by the source warp's artifacts, providing supervision for regenerating content rather than preserving the warp.

The dataset contains 40 three-dimensional environments, 1K dynamic scenes, and 8K videos, each with \(720\times1280\) resolution and 81 frames. During training, random scaling and shifting precede the depth-to-RGB mapping to reduce sensitivity to inconsistent monocular depth scales at inference. This augmentation improves scale robustness but cannot repair incorrect topology or missing objects. The comparison between 8K videos and ReCamMaster's 136K indicates reduced data requirements, not a measured compute ratio under matched hardware, training steps, and backbones.

4. Lightweight Adapter Training: retain generative priors while learning new conditions

The full model uses Wan2.2-TI2V-5B, freezes the pretrained video diffusion backbone, and adapts it with rank-32 LoRA. The VAE handles video latent encoding and decoding; text prompts still pass through T5 and enter via cross-attention. The contribution changes the geometry and video-content conditioning paths rather than replacing the underlying video generator.

Synchronized target videos provide generation supervision, while the dual-stream inputs teach the adapter when to use geometric layout and when to retrieve identity and motion from the source. LoRA reduces trainable parameter count, not necessarily inference cost: source-video concatenation still adds tokens, and each output requires multiple denoising steps. The main text does not separately report the projection layer's parameter count or optimizer groups, so those implementation details are not supplied here.

A Worked Example¶

Consider a source video containing changing facial expressions, with a request to move the camera from a frontal to a side view. This illustrates the mechanism rather than introducing another experiment. Per-frame meshes first produce target-side depth and visibility; masks identify unobserved facial regions, while the source video retains expression details at each moment. The view stream constrains target facial layout, the content stream supplies identity and dynamics, and diffusion jointly generates the side-view video.

Removing the content stream leaves geometry and first-frame appearance insufficient to recover later subtle expressions. Replacing depth with warped RGB can reintroduce distorted facial textures as a copying shortcut. Models (c) and (b), respectively, test these failure paths rather than relying solely on subjective inspection of the final output.

Loss & Training¶

The method retains flow-matching velocity prediction, using the difference between noise and target data as the velocity target and a timestep-dependent weight in the training objective. Cached Equations (1), (2), and (6) contain conspicuous character corruption. Consequently, the exact interpolation expression, norm formatting, and weighting function are not reconstructed here. The main text does not introduce an additional identity loss or geometric reprojection loss.

Inputs are resized to \(704\times1280\) with 81 frames. Training uses AdamW at \(1\times10^{-4}\) on 8 NVIDIA A100 GPUs for 4 days. Inference takes 50 denoising steps and approximately 4 minutes per video. The main text does not explicitly identify the inference GPU count associated with that timing, so it should not be described as single-GPU latency.

Key Experimental Results¶

Main Results¶

Evaluation uses 150 in-the-wild web videos sampled from Koala, with 10 trajectories per video, covering basic translations and rotations plus orbit, helix, S-curve, and parabola. ViPE estimates camera parameters from generated videos, after which RE, TE, and CamMC are computed following CamI2V; lower is better. The cache does not provide their complete formulas or units, so RE must not be arbitrarily labeled in degrees.

RS measures ArcFace embedding similarity between the source video's first frame and generated frames. IFS measures identity embedding similarity between consecutive generated frames. Both are evaluated on the face-containing subset and are better when higher. Img.Q denotes VBench imaging quality. The following extracts Table 1 while retaining the two camera protocols; cross-protocol values are not strictly matched comparisons.

Protocol	Method	RE lower	TE lower	CamMC lower	RS higher	IFS higher	Img.Q higher
Relative camera	TrajectoryCrafter	1.464	0.133	2.084	0.567	0.916	71.59
Relative camera	GEN3C	1.397	0.076	1.982	0.618	0.943	72.04
Relative camera	EX-4D	1.186	0.101	1.687	0.653	0.925	70.65
Relative camera	Ours(DC)	1.382	0.096	1.962	0.669	0.959	72.88
Absolute camera	ReCamMaster	3.576	0.180	5.063	0.576	0.932	67.27
Absolute camera	Ours(1.3B)	1.349	0.073	1.913	0.653	0.942	70.00
Absolute camera	Ours(Full)	1.010	0.053	1.434	0.663	0.954	72.90

Relative-camera trajectories reference each source frame's pose; Ours(DC) and the warping baselines share DepthCrafter depth and the fixed-focal-length setup. Absolute-camera trajectories reference the first frame, with Ours(Full) using \(\pi^3\). Under the relative protocol, Ours(DC) improves identity preservation and imaging quality, but EX-4D has lower RE and CamMC, and GEN3C has lower TE. This is a trade-off between geometric adherence and appearance reconstruction, not dominance on every metric.

Ablation Study¶

The configurations in original Table 3 are explained in Section 4.3, and all use MultiCamWarp. Cs.S. and Cs.B. denote VBench subject and background consistency, respectively; higher is better. The full model's RS of 0.6887 here must not be mixed with the main table's 0.663: the main text does not fully explain the evaluation difference between these tables.

Configuration	View condition	Source-video content stream	RS higher	IFS higher	Cs.S. higher	Cs.B. higher
Model (a)	Warped RGB	No	0.5954	0.9606	94.80	94.57
Model (b)	Warped RGB	Yes	0.6463	0.9621	95.20	94.43
Model (c)	Warped depth	No	0.5804	0.9628	95.17	94.66
Full model	Warped depth	Yes	0.6887	0.9661	95.29	94.66

Adding the content stream with RGB conditioning increases RS from 0.5954 to 0.6463; retaining that stream and switching to depth raises it to 0.6887. Conversely, removing the content stream under depth conditioning lowers RS to 0.5804, demonstrating that geometry alone does not ensure identity preservation. Model (c) still reaches IFS 0.9628, illustrating that a person can look consistent across frames without matching the person in the source video.

Key Findings¶

The extreme-view test selects videos with fixed source cameras to remove the difference between relative and absolute trajectory definitions. The following extracts the \(0^\circ\rightarrow\pm90^\circ\) setting from Table 2. GEN3C and the full model use \(\pi^3\), whereas EX-4D still uses DepthCrafter, so not every method shares the same geometry estimator.

Method	Cs.S. higher	Cs.B. higher	Img.Q higher
EX-4D	86.03	89.02	67.30
GEN3C	87.73	90.07	67.64
ReCamMaster	83.54	86.31	58.42
CamCloneMaster	90.04	90.46	69.32
Ours(Full)	89.92	90.91	71.83

The full model leads on extreme-view background consistency and imaging quality, but its subject consistency is slightly below CamCloneMaster. The authors attribute the latter's higher score to failing to reach the requested camera rotation. That explanation cannot be verified from this table alone, and high consistency should not automatically be interpreted as accurate camera control.

Highlights & Insights¶

Richer conditions are not always more useful. Removing actual scene colors from geometry guidance weakens the shortcut of copying corrupted textures and encourages the model to consult source content.
Synchronized multi-view supervision changes the task. The model must generate what another camera actually observes rather than restore a masked version of the source image.
Identity matching and temporal stability require separate measurements. Model (c)'s high IFS and low RS show that a stable but incorrect identity can still produce attractive temporal scores.

Limitations & Future Work¶

The authors explicitly discuss inaccurate monocular point maps and the trade-off between geometric adherence and visual quality, but the main text has no dedicated limitations section. Generative correction does not guarantee strict agreement with the real three-dimensional scene.
Note assessment: 150 web videos and learned pose, identity, and perceptual metrics do not replace evaluation against actual multi-camera target videos. The cache also provides no error bars or multi-seed stability analysis.
The 8K synthetic videos come from 40 environments, and depth-scale augmentation cannot cover every real-world failure. Transparent, reflective, and heavily occluded scenes warrant dedicated tests; they should not be presented as already quantified failures of this method.
The 81-frame input, doubled token count, and approximately 4-minute generation time constrain real-time use. Continuous long-video camera motion, authenticity of unobserved regions, and faster inference remain open issues without complete validation in the main text.

Versus EX-4D, GEN3C, and TrajectoryCrafter: these methods anchor target videos with warped RGB. DepthDirector retains explicit geometry but restricts it to depth layout, reducing inherited appearance artifacts while requiring more content to be generated.
Versus ReCamMaster: the latter learns control through implicit camera conditions, whereas this method supplies target layout through depth. Ours(1.3B) still outperforms ReCamMaster in the table, but this is not a single-factor experiment with every other training condition fixed.
Versus CamCloneMaster: that method uses reference videos to guide camera motion, whereas DepthDirector accepts target trajectories directly. Comparisons should verify whether the desired viewpoint is reached rather than relying exclusively on VBench consistency.
Transferable idea: geometry-guided generation can separate layout and appearance into distinct conditioning paths and supervise their fusion with actual paired targets. Transfer to other tasks still requires experimental validation.

Rating¶

Novelty: 4/5. Separating condition information directly addresses the warped-RGB shortcut; the main novelty lies in conditioning and training-data organization.
Experimental Thoroughness: 4/5. Two camera protocols, component ablations, and extreme-view tests are included, but error bars and more direct real multi-view validation are missing.
Writing Quality: 4/5. The problem and ablation logic are clear, although some claims of universal superiority are stronger than the tables support.
Value: 4/5. A reusable approach to camera-controlled video re-rendering, with practical value still constrained by geometry errors, latency, and long-video capabilities.