Skip to content

MoCam: Unified Novel View Synthesis via Structured Denoising Dynamics

Conference: ECCV 2026
Paper: ECCV 2026 · Project Page
Area: 3D Vision
Keywords: novel view synthesis, video re-camera, structured denoising, geometry-appearance disentanglement, diffusion conditioning

TL;DR

MoCam hands the two conditioning signals — a point-cloud scaffold rendered along the target trajectory and the source-view video — to a video diffusion model on a relay along the denoising timeline: the scaffold anchors geometry and motion early, then a switch hands over to the source video for error correction and detail refinement, unifying single-image 3D reconstruction and video 4D re-camera in one pipeline.

Background & Motivation

Since diffusion models entered novel view synthesis (NVS), the field has split into two camps. One encodes camera pose directly into the model's conditioning pathway (ReCamMaster, VD3D, CameraCtrl and relatives): end-to-end generation with no explicit geometry, but complex or large trajectories are hard to follow and the output quietly drifts away from the target view. The other camp renders first and inpaints afterwards — a 3D scaffold (point cloud or depth map) is reconstructed from the monocular input, rendered along the target trajectory into a coarse video, and handed to a video inpainting model. Gen3C's spatiotemporal 3D cache, TrajectoryCrafter's Ref-DiT and ViewCrafter's multi-view diffusion all belong here. This second camp clearly gets the target-view geometry better, but it rests the entire pipeline on something inherently deficient: a point cloud reconstructed from monocular input is necessarily sparse, full of holes and depth-inaccurate under large view changes, and the inpainting stage merely treats these flaws as given facts to prettify — it has no capacity to undo them.

The real difficulty is that the two signal families are complementary yet cannot be used at the same time. The rendered scaffold is spatially aligned with the target trajectory but riddled with disocclusion holes and depth-induced distortion; the source image/video has clean texture and faithful appearance but was shot from a camera that is not the one you want. Worse, adding them together is not a simple sum of capabilities. Bring strong appearance cues in early and they overwhelm the scaffold's spatial constraints, dragging the result toward the source viewpoint (geometric drift). Let the flawed geometry stay in the condition until the last step instead, and those structural errors get permanently baked into the output by the high-fidelity synthesis stage — a cleaner texture only paints a wrong shape more beautifully. The paper pushes this observation to the level of conditioning mechanisms in general: ControlNet and IP-Adapter style work uses one static control signal throughout, while TSM and DMP have shown that making the control signal vary over time improves generation quality considerably. Nobody had worked out, for the re-camera setting specifically, what should vary and when.

This paper's angle is to restate the conflict as a scheduling problem rather than a fusion problem. Different denoising phases have genuinely different needs: early on the latent is almost pure noise and the task is to fix global structure and motion, so the condition needs little spatial precision and enjoys a high error tolerance; later the low-frequency structure is already in place and correctly positioned, the task turns to high-frequency detail, and what is needed is a clean texture source. MoCam therefore staggers the two conditions along the timeline, each in charge of one stretch: first the scaffold nails down "where the objects are and where the camera goes," then the scaffold is withdrawn and the source video takes over, so that the late high-fidelity synthesis can repair the geometric errors inherited from the early stage. Core idea: rewrite the condition as a time-dependent \(c(t)\) — early stages use the rendered scaffold to anchor geometry and motion, and past \(T_{\text{switch}}\) the model switches to the source video for active error correction and appearance refinement, letting one video diffusion model unify 3D static reconstruction and 4D dynamic re-camera.

Method

Overall Architecture

MoCam takes a monocular source video \(x^{src}\) (for the single-image 3D reconstruction setting, the frame is replicated into \(N\) frames to form a "stationary video") and a user-specified target camera trajectory \(\psi^{tgt}\), and outputs a target-view video \(x^{tgt}\) shot along that trajectory. The pipeline has only three steps: estimate depth and camera to build a dynamic point cloud, render it along the target trajectory into a spatially aligned but badly damaged scaffold video; encode the scaffold video and the source video into latents and feed them, together with the initial noise, into a pretrained video diffusion model; then switch conditions over the course of denoising — the scaffold alone holds the structure early on, the source video later repairs both detail and geometry — and decode the final video. There is no per-scene optimization and no auxiliary 3D representation: geometric consistency is guaranteed by when a condition is active, not by the representation itself.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    A["Source video / single image replicated to N frames<br/>+ target camera trajectory"] --> B["Dynamic point-cloud scaffold<br/>depth → point cloud → render along trajectory"]
    B --> C["Frame-dimension dual-condition injection<br/>scaffold and source video concatenated on the frame axis"]
    subgraph S["Structured denoising dynamics"]
        direction TB
        D["Early: scaffold anchors geometry and motion"] --> E["Late: source video corrects and refines appearance"]
    end
    C --> D
    E --> F["Decode into the target-view video"]

Key Designs

1. Dynamic point-cloud scaffold: giving camera control a spatially aligned but deliberately imperfect draft

For camera control, MoCam takes the least glamorous but most robust route: no Plücker rays, no pose tokens or pose encodings entering the model at all — the camera pose exists only inside the rendering operator. Concretely, per-frame depth \(d^{src}\) and camera intrinsics \(K\) are estimated for the source video, and the pixels are back-projected through inverse perspective projection into a time-varying dynamic point cloud \(p=\{P_i\}_{i=1}^{N}\); given the target trajectory \(\psi^{tgt}\), the point cloud is rendered to the target viewpoint by perspective projection, yielding the scaffold video \(x^{tgt}_{ren}\):

\[p=\Phi^{-1}\left(x^{src},d^{src},K\right),\qquad x^{tgt}_{ren}=\Phi\left(p,\psi^{tgt},K\right)\]

This pair of equations carries the paper's answer to "where does the camera condition come from": the target trajectory is spent once inside rendering, and the model only ever sees an image that already stands at the target viewpoint — except that this image is riddled with disocclusion holes and depth-induced distortion, and the further the viewpoint is from the source camera, the worse it gets. Reducing the pose problem to an image-conditioning problem means the model never has to understand cameras, only to align an approximately co-located image; arbitrary trajectories (orbit, translation, zoom) are handled uniformly without any dedicated positional encoding. The price is that geometric accuracy is staked entirely on the depth estimate, which we return to below.

2. Frame-dimension dual-condition injection: turning both conditions into "extra frames"

Both conditions are injected the same way, deliberately simply. The scaffold video and the source video are encoded separately by the VAE encoder into \(c^{ren}=E(x^{tgt}_{ren})\) and \(c^{src}=E(x^{src})\), and each is concatenated with the initial noise \(z_0\) along the frame dimension to form the video model's input (following ReCamMaster's frame-dimension conditioning, precisely to preserve temporal synchronization). Choosing the frame axis costs something and pays something back: the conditions are themselves videos, so concatenating on the frame axis means condition frames and generated frames share one timeline, and spatiotemporal attention can directly establish the correspondence "frame \(i\) should look like condition frame \(i\)" without any extra alignment machinery guessing how the tokens map to frames. This also explains why a condition as badly misaligned as the source video is still useful — what it supplies is not spatial alignment but clean texture and consistent appearance; the model learns to carry over its style, not its composition.

The design space here is explicitly closed off by a comparison figure in the paper: the most intuitive alternative is to concatenate \(c^{ren}\), \(c^{src}\) and \(z_0\) all at once and let the model learn the combination. But the two carry different camera motions, so they give contradictory information about "which camera motion to follow"; static simultaneous conditioning feeds the model conflicting gradients — softening the scaffold's spatial constraint (structure goes mushy) while also injecting appearance contamination. This conflict is not an implementation detail; it is where the paper's problem statement begins.

3. Structured denoising dynamics: geometry first, appearance second, handed over in relay

The actual novelty is letting the condition change hands over time. The model no longer predicts with a fixed \(c\) in \(f_\theta(z_t,t,c)\) but with a time-dependent context \(c(t)\), switched at a pre-defined threshold \(T_{\text{switch}}\):

\[c(t)=\begin{cases} c^{ren}, & t>T_{\text{switch}}\\ c^{src}, & t\le T_{\text{switch}}\end{cases}\]

The property the paper exploits is that the timestep \(t\) is not merely a noise level but a progression from global structure to local detail, so each signal is placed on the stretch where it is most effective. In the early stretch (\(t>T_{\text{switch}}\)) the latent is still dominated by noise and the model's job is to establish the global structure and motion of the video; the scaffold's error tolerance is at its highest here — even with holes and smeared edges it suffices to specify where the objects are and where the camera travels. This stage therefore deliberately tolerates geometric incompleteness and uses \(c^{ren}\) alone to force the generation onto the target trajectory. In the later stretch (\(t\le T_{\text{switch}}\)) the low-frequency structure has already formed and sits in the right place, and the task turns to high-frequency detail; the condition switches to \(c^{src}\), and because the coarse structure is anchored, the model can use this clean reference to "inpaint" and "correct" the structure inherited from stage one instead of being continuously poisoned by the scaffold's errors. This is the paper's most important semantic leap — the source video stops being a mere texture courier and becomes a geometry-correcting signal, which is exactly what a static pipeline cannot do.

The switch position is worth noting: \(T_{\text{switch}}=0.85\) means geometric anchoring occupies only the very beginning of the denoising trajectory (under the paper's convention that a larger \(t\) means more noise, i.e. roughly the first 15% of steps use the scaffold), and the vast majority of steps are left to the appearance condition. This matches the intuition that geometry merely sets an initial value while late appearance synthesis determines the final quality. ⚠️ The paper's prose about the direction of \(t\) conflicts literally with the flow-matching definition it states (with \(z_1\) as the clean latent): under the latter, \(t\) near 1 should be the cleanest, which contradicts "the latent is mostly noise in the early stage." The description here follows the paper's own semantics (\(t\) large = noisy = early denoising); for reproduction, defer to the original paper and official code for the timeline convention.

A Worked Example

Take a 49-frame monocular video where the user wants the camera to orbit roughly 60° to the rear left. First, ViPE estimates per-frame depth and camera pose, and back-projection yields a 49-frame dynamic point cloud; rendering it to the new viewpoint along the target trajectory gives a scaffold video — the subject is roughly in place, but its edges are smeared, the background originally occluded by the subject is a field of black holes, and distant regions show obvious stretching. Second, the scaffold and the source video are encoded into latents and concatenated with the initial noise along the frame axis, then fed to Wan2.2. Third, taking 50 sampling steps as an illustration (the paper does not report the sampling-step count), roughly the first 8 steps (\(t>T_{\text{switch}}\)) use only the scaffold condition: the subject's volume in frame, the direction the camera sweeps, and the depth ordering of the background are all fixed in this stretch, and the scaffold's holes do not matter yet. Past \(T_{\text{switch}}\) the condition switches to the source video, and over the remaining 40-odd steps the previously hole-punched background is filled with clean texture from the source video, the smeared edges are redrawn straight, and the mild misalignment caused by depth error is corrected. The output 49 frames follow the target trajectory while showing no trace that a scaffold ever existed.

Loss & Training

MoCam reuses the training objective of the latent video diffusion model it is built on, with the single change that the condition term goes from a fixed \(c\) to \(c(t)\). In the flow-matching form stated in the paper:

\[\mathcal{L}=\mathbb{E}_{z_0,z_1,t,c(t)}\left\|f_\theta\!\left(z_t,t,c(t)\right)-v_t\right\|_2^2,\qquad v_t=z_1-z_0\]

⚠️ The two objective equations in the original paper are damaged by typesetting; the form above is rewritten from its prose description (\(z_1\) the latent of a real video, \(z_0\) Gaussian noise, \(v_t=z_1-z_0\)) — refer to the original paper. Implementation details: the backbone is the pretrained Wan2.2 video diffusion model (the ablation also swaps in Wan2.1 to test backbone independence); training uses 20,000 paired samples from MultiCamVideo, each pair consisting of a reference video, the resulting scaffold video and the ground-truth target video; 20,000 steps on eight GPUs, learning rate \(1\times10^{-5}\), batch size 8; \(T_{\text{switch}}=0.85\) set empirically; depth and camera from ViPE. Training is end-to-end, and inference performs no per-scene optimization.

Key Experimental Results

Main Results

Evaluation spans three settings: 4D re-camera uses 100 monocular videos collected from OpenVid-1M with 9 distinct trajectories each (orbit, translation, zoom); single-view 3D reconstruction randomly samples one frame per video and replicates it to \(N\) frames; a further set of pixel-wise metrics is reported on the iPhone multi-view dataset following TrajectoryCrafter's setup. Metrics are VBench background consistency (BC), subject consistency (SC) and imaging quality (IQ), cross-view FVD-V and CLIP-V, and the CameraCtrl pose-accuracy pair (rotation error RotErr, translation error TransErr). In the tables, ReCam = ReCamMaster and TrajCrafter = TrajectoryCrafter, both direct competitors.

Method (4D re-camera, OpenVid) BC ↑ SC ↑ IQ ↑ FVD-V ↓ CLIP-V ↑ RotErr ↓ TransErr ↓
Gen3C 0.9270 0.9067 0.6908 291.13 0.79 1.36 5.13
TrajectoryCrafter 0.9235 0.9062 0.6697 317.08 0.80 1.38 5.12
ReCamMaster 0.8977 0.8801 0.5837 361.98 0.76 2.15 5.82
MoCam 0.9332 0.9247 0.6932 260.05 0.85 1.36 5.12

Single-view 3D reconstruction points the same way with larger gaps: MoCam reaches BC/SC/IQ of 0.9334/0.9250/0.6961, FVD-V 255.16, CLIP-V 0.87 and RotErr 1.35, against Gen3C at 289.37/0.80/1.36, TrajectoryCrafter at 313.65/0.79, ReCamMaster at 355.26/0.77/2.13 and the optimization-based ViewCrafter at 308.24/0.79/1.39 (the MoCam row is identical to the 4D table because the paper's Tab. 1 and Tab. 4 reuse the same evaluation setup). On the iPhone dataset, switching to pixel-wise metrics:

Method (iPhone) PSNR ↑ SSIM ↑ LPIPS ↓ FVD ↓
Gen3C 12.36 0.4028 0.5112 260.15
TrajectoryCrafter 13.74 0.4555 0.4819 273.36
ReCamMaster 11.44 0.3768 0.5622 301.41
MoCam 14.60 0.4581 0.4213 180.35

Ablation Study

The paper dismantles the two components of the mechanism in the 4D re-camera setting, producing three variants: Scaffold-Only (scaffold conditioning for all timesteps), Scaffold-Early (scaffold only for early timesteps \(t>T_{\text{switch}}\), no explicit conditioning later), and Static-Both (both scaffold and source video throughout denoising — the static fusion the paper argues against). A backbone swap to Wan2.1 tests whether the gains come from the mechanism or from the model.

Config (OpenVid, 4D re-camera) IQ ↑ FVD-V ↓ CLIP-V ↑ RotErr ↓ TransErr ↓ Note
Scaffold-Only 0.4807 359.38 0.76 1.37 5.10 imaging quality collapses; geometric error baked in throughout
Scaffold-Early 0.6172 273.19 0.83 1.37 5.13 worst artifacts avoided, but no reference later; inpainted regions go blurry
Static-Both 0.6740 242.81 0.87 2.71 11.01 strong perceptual scores, but geometry is completely destabilized
MoCam (Wan2.1 backbone) 0.6931 253.13 0.84 1.37 5.11 backbone swap barely moves the numbers
MoCam (full) 0.6932 260.05 0.85 1.36 5.12 full model

Key Findings

  • Removing the late appearance stage hurts more than removing the early geometry stage. Scaffold-Only is the most extreme case of static conditioning failing: IQ falls to 0.4807 and SC to 0.8448, showing that persistent exposure to scaffold error does not wash out as generation proceeds — it reliably paints the error into the final result. Scaffold-Early recovers substantially (IQ 0.6172) once the scaffold is withdrawn, confirming that "don't let bad geometry stay until the end" is indeed necessary; yet it stays clearly below the full model, because with no clean reference later the disoccluded regions can only be invented, and come out blurry and generic. Together the two controls say: removing the bad signal is not enough, you must also supply a good replacement — which is exactly the line between MoCam and the naive "stop using the scaffold early" approach.
  • Static-Both is the most instructive set of numbers — and a metric trap. Its FVD-V (242.81) and CLIP-V (0.87) are both better than the full model, so perceptual metrics alone would conclude that static fusion is fine. But its rotation error is 2.71 and translation error 11.01, roughly twice MoCam's 1.36 and more than twice its 5.12. The consequence of signal conflict is not an uglier picture; it is a picture that looks self-consistent while failing to follow the target trajectory at all — something frame-consistency metrics cannot see. Accordingly, the fact that MoCam's FVD-V is not the best in the table (260.05 vs Static-Both's 242.81) should be reported honestly: evaluation of camera-controllable tasks must read perceptual and pose metrics together.
  • The gains come from the mechanism, not the backbone. Wan2.1 and Wan2.2 results nearly coincide (IQ 0.6931 vs 0.6932, FVD-V 253.13 vs 260.05), ruling out "a stronger video model was used."
  • The motion-magnitude robustness study is the most convincing experiment in the paper. Scaling an orbit from 30° up to 90°, scaffold-based methods see progressively denser holes in the point cloud and implicit methods drift progressively further; every baseline degrades while MoCam stays stable. Early anchoring addresses the implicit methods' drift, late appearance correction addresses the scaffold methods' sparsity — two failure modes each neutralized by the same mechanism.
  • Depth perturbation does not break it (qualitative). With perturbed depth input, structures bent by depth error in the scaffold (e.g. lights) are still corrected in the final output, indicating the late appearance stage really is compensating actively rather than relying on sufficiently accurate geometry.
  • The distribution of gains on iPhone is telling. The margins on LPIPS (0.4213 vs 0.4819) and FVD (180.35 vs 260.15) are clearly larger than on PSNR (14.60 vs 13.74) — the advantage lies in "looking right," not in pixel alignment. Meanwhile all methods sit at only 12-15 dB PSNR, so this task remains hard at the pixel level and absolute PSNR should not be the sole criterion for a method.

Highlights & Insights

  • Restating condition conflict as "who goes on stage when," swapping one if-else for a large gain. This is the paper's aha moment: when two conditions interfere, the instinct is to design a smarter fusion module (attention, gating, weighting), whereas MoCam does exactly one thing — stagger them in time. The idea transfers well: any setting with two complementary but mutually interfering conditions (text vs reference image, semantic map vs scribble, depth vs normals) can consider scheduling them by denoising phase instead of building a fusion operator.
  • Camera control by "rendering into a frame" rather than pose encoding. Pose never enters the model; the model only sees a damaged image already standing at the target viewpoint, which degrades camera control into an image-conditioning problem. The benefit is no pose-encoding design and uniform handling of arbitrary trajectories; the cost is that geometric accuracy is staked entirely on depth estimation — the main risk exposure relative to implicit pose methods.
  • The late appearance condition is given a corrective, not textural, semantics. Because the low-frequency structure is already anchored, clean texture from the source video can be used to inpaint and repair the inherited structure. The argument "structure is already right, therefore an appearance signal can move structure without drifting" elevates the necessity of staging from an empirical trick to an explicable mechanism.
  • One mechanism unifies static and dynamic NVS. For the single-image case, replicating one frame into a stationary \(N\)-frame video leaves the point-cloud construction, condition injection and switching strategy untouched: 3D reconstruction and 4D re-camera share a single pipeline.
  • Methodological value in the ablation design. The Scaffold-Early variant is key: it demonstrates that "removing the bad signal" and "replacing it with a good one" are two different things. Many papers' ablations only test the former, which inflates the apparent contribution of their mechanism.

Limitations & Future Work

  • Depth and pose estimation are the ceiling. The scaffold comes from ViPE's monocular depth and camera estimation; when depth is grossly wrong over a large region, the early anchoring stage anchors structure in the wrong place and late correction cannot recover it. The paper shows only one qualitative depth-perturbation figure (Fig. 10) with no quantitative perturbation-strength curve and no failure-rate reporting.
  • \(T_{\text{switch}}=0.85\) is a hand-set global constant. No sensitivity sweep is given, and it is unclear whether the value remains appropriate for different trajectory lengths or motion difficulties. By the paper's own logic, a more principled choice would be to make the handover adaptive — for example switching when the predicted latent's uncertainty, or the agreement between scaffold and current estimate, crosses a threshold. That looks like a natural follow-up.
  • The 3D/4D "unification" is at the interface level, not the capability level. Replicating a frame into \(N\) frames leaves no temporal information on the input side for the single-image case, so the video model's temporal prior can only act on the generation side — plausibly why the single-view setting gains less than the 4D one.
  • Training data is small and narrow. All 20,000 pairs come from MultiCamVideo and training runs only 20,000 steps; OpenVid is used for evaluation only, so cross-domain generalization (indoor to outdoor, real to synthetic) is untested.
  • Extreme trajectories still degrade. The paper's own Fig. 7 shows MoCam under pressure at a 90° orbit, only better than the baselines — the mechanism mitigates rather than eliminates geometric degradation under large view changes.
  • Inference cost and failure mode. Every video requires a depth/pose estimation pass plus point-cloud construction and rendering, a substantial preprocessing overhead compared with purely implicit pose methods, and a failed depth estimate shows up directly in the scaffold. The authors' stated future work is joint scaffold-video refinement — letting the generated result correct the point cloud in an iterative loop, which if closed would address both the geometry ceiling and the fixed switch point.
  • vs Gen3C / TrajectoryCrafter (render-then-inpaint): they also use a point-cloud scaffold, but it conditions or references the generation throughout, so monocular reconstruction errors are inherited permanently and the inpainting stage only cares about looking good, not about being correctly shaped. MoCam's difference is not a better scaffold but a changed lifetime for the scaffold: active early, withdrawn later in favor of the source video. At equal scaffold quality, when a condition is active decides whether the error is inherited or corrected.
  • vs ReCamMaster (implicit pose conditioning): end-to-end, no explicit geometry needed, but without spatial constraints it drifts wholesale under large trajectories and has no mechanism pulling generation back to the target view. MoCam supplies that anchor via the early scaffold — conversely, it reuses ReCamMaster's frame-dimension injection verbatim, so the two agree on how to inject and differ only on when to inject what.
  • vs ControlNet / IP-Adapter / T2I-Adapter (static conditioning): these use one control signal throughout, with control strength as a hyper-parameter rather than a function of time. Making conditioning time-dependent is not new in itself (TSM and DMP already show dynamic conditioning improves quality); MoCam's contribution is to concretize "dynamic" as a handover between complementary signals and to define, for the re-camera setting, both the content of the handover (scaffold ↔ source video) and its timing (after geometry anchoring).
  • vs ViewCrafter and other video-diffusion single-view NVS: also video-diffusion-based single-view view synthesis, but ViewCrafter does no timeline-level treatment of scaffold error; in the paper's qualitative comparison, structures such as tires and text are visibly distorted under ViewCrafter while MoCam preserves their shape — the difference comes from the structured denoising dynamics.
  • vs optimization-based NeRF / 3DGS: these obtain explicit representations through per-scene optimization, so geometric consistency is guaranteed by the representation, at the cost of dense multi-view input and expensive optimization. MoCam does no per-scene optimization and delegates geometric consistency to the temporal scheduling of conditions, working from monocular input; the price is that geometric accuracy depends on depth estimation and cannot be inspected or edited the way an explicit representation can. The point where the two meet — using generated results to refine the point cloud — is precisely the authors' stated future direction.

Rating

  • Novelty: ⭐⭐⭐⭐ All ingredients are off the shelf (frame-dimension conditioning from ReCamMaster, the point-cloud scaffold from the render-then-inpaint line), but reframing "condition conflict" as "condition scheduling" and delivering a minimal yet effective switch is a clean and transferable insight.
  • Experimental Thoroughness: ⭐⭐⭐ Three evaluation settings, clean ablations and a backbone-swap check, but no sensitivity analysis for \(T_{\text{switch}}\), only a qualitative depth-perturbation study, a single training data source, and residual degradation at extreme 90° trajectories.
  • Writing Quality: ⭐⭐⭐ The argument is clear, but the direction of \(t\) contradicts the stated flow-matching definition, the two objective equations are typeset-damaged, and Tab. 1 and Tab. 4 are partly redundant.
  • Value: ⭐⭐⭐⭐ The mechanism adds essentially no cost and is trivial to implement, and it transfers to any dual-condition generation task; it offers a convincing answer to how geometry and appearance should coexist in generative novel view synthesis.