DualCamCtrl: Dual-Branch Diffusion Model for Geometry-Aware Camera-Controlled Video Generation

Conference: ECCV2026
Paper: ECCV paper page
PDF: Full paper
Project: DualCamCtrl
Area: Video Generation
Keywords: camera control, dual-branch diffusion, depth generation, semantic-guided alignment, spatiotemporal fusion
Paper ID: 3378
Citations: 9 (task-provided count, 2026-09-17)

TL;DR

DualCamCtrl treats RGB and depth as separately modeled but jointly evolving generative modalities, improving camera adherence through semantic-first bidirectional interaction and 3D fusion; on RealEstate10K text-to-video generation, it reduces AC3D's rotation error from 2.67 to 1.23 while lowering FVD from 415.6 to 408.1.

Background & Motivation

Camera-controlled video generation requires more than visible motion: objects should exhibit plausible perspective and occlusion changes along a specified camera trajectory. MotionCtrl, CameraCtrl, and Wan-based control methods feed camera poses or Plucker ray embeddings into diffusion models, but rays describe where the camera looks from and toward, not the distances of scene objects. A single RGB representation must therefore infer appearance and 3D layout together, allowing plausible images that nevertheless follow the requested camera motion inaccurately.

Depth supplies missing geometric information, but concatenating it does not automatically solve the problem. A single depth map cannot describe geometry throughout later frames, while forcing RGB and depth through one generative branch introduces modality interference. External reconstruction or reprojection approaches such as ReconX, ViewCrafter, and GEN3C offer another route, but require explicit 3D representations and multi-stage workflows that make appearance and geometry difficult to adjust jointly during generation.

The paper instead generates an entire depth video rather than using depth as a static control map. Separate branches preserve appearance and geometry modeling capacity, but must exchange information at appropriate locations; otherwise, even a shared camera trajectory does not prevent their contents from drifting apart. Core Idea: first teach RGB and depth to generate independently, then let RGB semantics lead and depth feedback follow, so that both modalities evolve together within one diffusion process.

Method

Overall Architecture

For image-to-video generation, the inputs are a reference RGB image and a target camera trajectory, with a depth estimator supplying reference-image depth. Shared camera conditioning drives two separate DiT branches that generate RGB and depth video latents. SIGMA orders their interactions, 3D fusion incorporates spatiotemporal neighborhoods, and the RGB video is decoded at the end.

The depth branch is not a preprocessing module that generates a fixed depth sequence before RGB synthesis begins. Both branches continue updating during denoising: depth receives appearance-based semantic constraints and feeds geometric corrections back to RGB. Training first disables interaction and subsequently enables fusion, preventing an untrained depth generator from immediately disrupting RGB generation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    INPUT["Reference image + camera trajectory"] --> PREP["Estimate reference depth<br/>VAE and ray encoding"]
    PREP --> DUAL["Dual-branch joint generation<br/>RGB and depth latents"]
    DUAL --> ALIGN["SIGMA semantic-guided alignment<br/>RGB first, depth feedback later"]
    ALIGN --> FUSE["3D fusion and frame gating<br/>Spatiotemporal interaction"]
    FUSE --> OUTPUT["Joint denoising and RGB decoding"]
    TRAIN["Two-stage training<br/>Independent learning, then fusion"] -.-> DUAL
    TRAIN -.-> FUSE

Key Designs

1. Dual-branch joint generation: update geometry throughout the video instead of relying only on first-frame depth

Section 3.2 replicates reference depth across channels to match RGB, then encodes the two inputs separately with a pretrained encoder. Reference latents are zero-padded along the frame dimension to match the noise sequence length, each receives the same camera Plucker embedding by element-wise addition, and each is concatenated with the same Gaussian noise before entering its respective DiT branch. Identical camera conditioning makes the branches follow the same viewing trajectory; separate generative representations avoid forcing texture and distance through a single feature stream.

Both branches predict full video latents, so later depth frames are not repetitions of the initial depth map. The depth stream must generate scene structure under changing viewpoints, while the RGB stream generates the corresponding appearance, allowing geometry to constrain the entire sequence. Supervision comes from RGB videos and estimated per-frame depth rather than perfect sensor-measured RGB-D data. At inference, RGB is the desired video output: depth is primarily an internal participant in generation, not an additional per-frame control signal required from the user.

2. SIGMA semantic-guided alignment: establish content before using geometric feedback to refine structure

Separate modeling creates another problem: shared camera conditioning does not guarantee corresponding cross-modal content. Figure 3 illustrates local semantic misalignment from independent evolution. One-way transfer cannot correct deviations that originate in the receiving branch, while depth-first interaction can make geometric motion evolve ahead of RGB content. SIGMA therefore uses ordered bidirectional interaction: RGB features first anchor semantic structure, and depth feedback subsequently refines geometry, instead of indiscriminately mixing both streams from the shallowest layers.

The motivation includes linear centered kernel alignment (CKA) analysis in Figure 4. Using the model after decoupled training, the authors sample 500 RealEstate10K validation sequences and compare RGB features with depth and Plucker embeddings at early, middle, and late denoising stages, across 3 random seeds. Shallow RGB features are more closely associated with camera embeddings, motivating asymmetric treatment of semantics and geometry. CKA provides correlational evidence rather than establishing causality by itself; the injection-order ablations in Table 4 supply the direct performance evidence.

Two meanings of "stage" must be distinguished. The narrative discusses early and late denoising, but Table 4 directly manipulates network layer ranges, such as RGB injection at layers 1-5 and depth injection at layers 6-15. These are not sampling timesteps, and the cache does not provide a temporal threshold schedule that could be reconstructed from them. The supported implementation principle is RGB-first interaction in shallower layers followed by depth feedback, not switching control signals at sampling step 5.

3. 3D fusion and frame gating: exchange neighborhood-aware features instead of pointwise projections

SIGMA specifies interaction direction and placement, but the form of the exchanged information still matters. Pointwise linear projection processes each latent position independently, ignoring cross-frame motion and spatial boundaries and potentially causing flicker or discontinuity. Section 3.3 introduces 3D convolutional fusion to aggregate across time and space, so that RGB and depth exchange signals containing local spatiotemporal context rather than isolated values.

The fusion block also uses a bottleneck to control parameter and computational overhead, and frame-wise gating adjusts fusion strength according to temporal dynamics. Figure 2 includes a zero-initialized convolutional layer, preventing fusion from immediately making large changes to branch behavior. The main text does not fully specify convolution kernels, bottleneck widths, or the gating equation, so these hyperparameters are not invented here. Table 3 directly supports the additional benefit of frame gating over 3D fusion alone.

4. Two-stage training: make depth informative before allowing it to provide feedback

Both branches start from the same pretrained Wan weights, rather than randomly initializing an entirely empty depth network. Those weights nevertheless originate from appearance generation and do not already provide depth-generation competence. Figure 6 shows that depth inputs can initially be interpreted as low-light or hazy images. Immediately enabling joint fusion lets unreliable geometric signals interfere with RGB while the depth stream itself may fail to converge.

The first stage disables all cross-branch fusion and lets RGB and depth learn their respective generation tasks independently. The second activates zero-initialized fusion blocks so the branches gradually learn complementary feedback. Because depth supervision comes from an estimator, the authors also early-stop the decoupled stage to retain coarse geometry and randomly drop the depth loss to avoid overfitting pseudo-label errors. Dropping a training loss does not mean randomly disabling the depth branch at inference; the exact probability and training-step counts are not provided in the supplied main text.

A Worked Example

Consider a kitchen image and a sideways camera trajectory as a conceptual example. Reference depth describes the relative distances of the counter, walls, and distant objects. Both branches receive the same trajectory and noise: RGB generates newly visible textures, while depth generates the corresponding distance layout, without an external renderer projecting each target frame.

Shallower RGB semantics help the depth stream preserve the identity and correspondence of the counter, while later depth feedback constrains its structural motion relative to the distant wall. 3D fusion incorporates motion context from neighboring frames, and frame gating adjusts its influence. Newly revealed regions still depend on generative priors, so the procedure does not guarantee exact 3D reconstruction and can fail under large movements. This example explains the mechanism and is not an additional reported experiment.

Loss & Training

Section 3.4 supervises the predictions of the RGB and depth branches separately, with cross-branch fusion disabled in the decoupled stage and enabled in the fusion stage. The combined objective in the paper's Equation (4) is:

\[ \mathcal{L}_{\mathrm{Overall}}=\mathcal{L}_{\mathrm{RGB}}+\mathcal{L}_{\mathrm{D}}. \]

The full mathematical formatting of Equations (2)-(3) is corrupted in the text cache, so their individual terms are not reconstructed here. Section 4.1 explicitly states that actual training uses rectified flow; the generic noise-prediction objective in the preliminaries should not be mistaken for the complete implementation. The backbone is Wan-V2.1, trained on RealEstate10K using 8 H100 GPUs with a reported training timestep setting of 1000. This does not establish that inference requires 1000 network evaluations.

Text-to-video generation is also evaluated, but Section 3.2 delegates its conditioning details to supplementary material. The supplied cache contains the main paper and references, not that supplementary implementation, so this note does not assume a reference image in T2V or transplant the I2V reference-depth input procedure into it.

Key Experimental Results

Main Results

The following excerpts come from Table 1 (page 10, I2V) and Table 2 (page 11, T2V). FVD and FID measure distribution-level generation quality; RE and TE are rotation and translation errors. All are lower-is-better. No units absent from the main text are assigned to RE or TE.

Source and setting	Dataset	Method	FVD	RE	TE
Table 1, I2V	RealEstate10K	Seva	104.2	2.62	0.32
Table 1, I2V	RealEstate10K	Wan	109.2	2.08	0.32
Table 1, I2V	RealEstate10K	DualCamCtrl	80.38	1.25	0.23
Table 1, I2V	DL3DV	Wan	127.9	1.01	1.64
Table 1, I2V	DL3DV	DualCamCtrl	92.2	0.88	1.39
Table 2, T2V	RealEstate10K	AC3D	415.6	2.67	0.38
Table 2, T2V	RealEstate10K	DualCamCtrl	408.1	1.23	0.25
Table 2, T2V	DL3DV	AC3D	452.8	1.06	2.11
Table 2, T2V	DL3DV	DualCamCtrl	427.4	0.83	1.53

The "over 40%" rotation-error reduction does not apply to every dataset and setting. On RealEstate10K T2V, the change from AC3D's 2.67 to 1.23 is approximately 53.9%, calculated from the table values. On DL3DV I2V, the change from Wan's 1.01 to 0.88 is approximately 12.9%.

RE and TE are computed using VGGSfM fine tracking, excluding approximately 5% tracking failures and near-static cases. They therefore describe a filtered evaluation subset rather than every generated video. The main text uses both Motion Strength and Motion Smoothness for MS; this note retains the abbreviation rather than merging those names into an unsupported definition.

Ablation Study

Table 3 (page 12) compares fusion blocks on RealEstate10K. Only the two rows with unambiguous column alignment in the cache are reproduced here; the first two rows are merged by text extraction and are not reconstructed by guessing.

Table 3 configuration	FVD, lower	FID, lower	MS, higher	FC, higher
Spatial-temporal 3D fusion	85.4	52.7	0.9861	0.9648
3D fusion + frame gating	80.4	49.9	0.9886	0.9677

Frame gating reduces FVD from 85.4 to 80.4, suggesting that spatiotemporal convolution alone does not fully exploit differences in geometric reliability across frames. FC measures frame consistency and rises from 0.9648 to 0.9677.

Table 4 (page 13) additionally controls the layer ranges for cross-branch feature injection. The following table preserves the original column meaning: these are RGB/depth feature injection layers, not video frame indices or sampling steps.

Table 4 alignment	RGB injection layers	Depth injection layers	FVD, lower	RE, lower	TE, lower
One-way alignment	None	1-15	83.2	1.55	0.35
Geometry-first	6-15	1-5	84.6	1.55	0.34
SIGMA semantic-first	1-5	6-15	80.4	1.25	0.25

The final row of the original Table 4 uses another SIGMA configuration, RGB 1-10 and depth 11-15, with FVD 80.3, RE 1.27, and TE 0.23, indicating trade-offs across fusion ranges. The full model in Table 1 has TE 0.23, whereas the RGB 1-5/depth 6-15 configuration in Table 4 has TE 0.25. These values must retain their respective table contexts rather than being combined into one row.

Key Findings

• Geometric-control gains exceed some distribution-quality gains: RealEstate10K T2V RE improves substantially, whereas FVD changes only from 415.6 to 408.1. The metrics are not interchangeable.
• Not every metric leads: RealEstate10K I2V CLIPSIM in Table 1 is 0.2998, below Seva's 0.3059; DL3DV T2V MS in Table 2 is 0.9807, below AC3D's 0.9869.
• Table 5 supports reliance on coarse geometry: fourfold depth downsampling changes RE from 1.25 to 1.28 and FVD from 80.4 to 82.9; simulated Kinect depth produces 1.37 and 84.7. These are bounded degradation tests, not guarantees against arbitrary depth errors.

Highlights & Insights

• Depth is both an output modality and an internal generative constraint. It turns geometry that might otherwise require external per-frame provision into a dynamically maintained model state, while retaining a path for geometric information to influence RGB.
• Branch separation and cooperation are compatible. Establishing modality-specific competence before learning directional, layer-specific interaction matches the difficulty of adapting a pretrained RGB generator to depth more closely than immediate unrestricted fusion.
• Gating adjusts fusion strength across video frames rather than imposing only a fixed global weight. The transferable idea is to let auxiliary modalities contribute according to spatiotemporal reliability, although the paper does not directly validate the same design for optical-flow or normal generation.

Limitations & Future Work

• Figure 11 shows the authors' large-motion failure case: substantial inter-frame motion still produces severe artifacts. Jointly generating depth does not establish a strictly reprojectable 3D world model.
• Duplicating the backbone increases capacity and latency. Table 7 reports 2.7B parameters and 49 seconds for DualCamCtrl versus 2.4B and 40 seconds for CameraCtrl at a shape of 14x576x320; these timings should not be extrapolated to other resolutions or larger video backbones.
• Depth supervision uses pseudo-labels. Training regularization and Table 5's degradation tests reduce dependence, but do not establish reliability under scale distortion, systematic semantic errors, or extreme scenes. The authors suggest lightweight or distilled depth branches as a future direction.
• Evaluation excludes tracking failures and near-static samples, and the main text omits some T2V and fusion implementation details. Full reproduction still requires supplementary material or implementation rather than inferring those hyperparameters from this note.

Related Work & Insights

• vs CameraCtrl / Wan: These baselines primarily control generation through camera or ray conditioning. DualCamCtrl adds a co-evolving depth modality under shared camera conditioning, changing how geometric information is modeled internally.
• vs ReconX / ViewCrafter / GEN3C: Related approaches rely on external reconstruction, caches, or reprojection. This paper moves geometric reasoning into the generator, reducing explicit multi-stage processing while forgoing some determinism available from explicit geometry constraints.
• vs AC3D: AC3D analyzes camera control in video DiTs and leverages dynamic-scene data; this paper emphasizes structural separation and controlled interaction of RGB and depth. Table 2 supports lower control errors, but is not a single-variable comparison with identical architectures and training budgets.

Rating

• Novelty: 4/5. The contribution is the combination of dual branches, semantic-first interaction, and training order, rather than simply adding depth.
• Experimental Thoroughness: 4/5. Covers I2V/T2V, two datasets, fusion ablations, and degraded depth, but filtered evaluation and implementation gaps remain.
• Writing Quality: 3/5. The motivation is clear, but layer depth versus denoising stage, MS naming, and some cross-table configurations require careful reading.
• Value: 4/5. Offers a reusable joint geometry-generation approach to camera control, subject to large-motion failures and dual-backbone costs.