Geometry-Aware Single-Image 4D Synthesis via Dense Trajectory Generation¶

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/Zhangyr2022/MoGe4D
Area: 3D Vision
Keywords: Single-image 4D generation, dense point trajectories, depth normalization, motion perception, novel view synthesis

TL;DR¶

MoGe4D estimates geometry from one image, generates geometry-conditioned dense motion trajectories with a diffusion model, and completes novel-view videos, improving imaging quality and trajectory error under complex camera motion without leading every video consistency or dynamics metric.

Background & Motivation¶

Generating a video from a photograph and generating a dynamic 3D scene that can be inspected with a moving camera are different problems. A video model can produce convincing adjacent frames without knowing that a pixel should remain attached to the same 3D surface point. When the viewing direction changes, that missing correspondence can manifest as texture drift, changing object shape, or backgrounds moving with the subject. Generate-then-reconstruct approaches such as 4Real, DimensionX, and Free4D leave reconstruction models to interpret these appearance changes, potentially turning generation errors into incorrect geometry and motion.

Reconstructing a static 3D structure before introducing dynamics provides a stable spatial reference, but often separates shape from motion. The authors argue that this paradigm handles externally constrained oscillations better than self-initiated motion, large deformations, and scene-level interactions inferred from the image itself. The paper does not claim that a single image reveals a unique true future, nor does it repeatedly optimize a complete static mesh during generation. Instead, initial depth conditions a generator that directly predicts future positions of the same surface points, reducing opportunities for appearance and geometry to disagree at the representation level.

This choice introduces two training challenges: real videos lack inexpensive human-annotated 3D trajectories, and pretrained video generators do not naturally output 3D displacement. The former requires quality-filtered pseudo-trajectory data; the latter requires an intermediate representation that can exploit video priors while remaining decodable into displacement. Core Idea: anchor dense relative motion generation to first-frame geometry, separating 3D dynamics prediction from subsequent appearance completion instead of letting an unconstrained video determine the entire dynamic scene first.

Method¶

Overall Architecture¶

The input is, strictly speaking, one RGB image and a textual description; novel-view output additionally requires camera poses or a camera path. The first output is a sequence of 3D point clouds with persistent point correspondence, followed by a video along the requested camera path. TrajScene-60K supplies training supervision, the 4D Scene Trajectory Generator (4D-STraG) generates normalized trajectories, and the Motion Perception Module (MPM) injects local motion priors during denoising. The 4D View Synthesis Module (4D-ViSM) projects predicted point clouds into video and fills regions the projections cannot cover.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    DATA["TrajScene-60K<br/>Trajectory Supervision"] -.->|Training only: trajectories| TRAG["Normalized Trajectory Generation<br/>4D-STraG"]
    INPUT["Image + Text<br/>Initial Depth"] --> TRAG
    INPUT --> MPM["Local Motion Perception<br/>MPM"]
    MPM -->|Token-wise denoising modulation| TRAG
    TRAG --> POINTS["Denormalize<br/>Recover Dynamic Point Clouds"]
    POINTS --> VISM["Geometry-Guided View Completion<br/>4D-ViSM"]
    CAMERA["Target Camera Path"] --> VISM
    DATA -.->|Training only: renders, masks, targets| VISM
    VISM --> OUTPUT["Novel-View Dynamic Video"]

MPM is a conditioning branch inside the trajectory generator, not a motion correction applied after point-cloud generation. The training-data branch supplies no real future frames at inference time; future motion is sampled from the model. The following designs cover supervision, the trajectory backbone, its motion-conditioning branch, and final view completion in that order.

Key Designs¶

1. TrajScene-60K Trajectory Supervision: extract learnable scene dynamics from real videos

Dataset construction starts with approximately 200,000 candidate videos from WebVid-10M rather than sending the entire collection into a 3D tracker. CogVLM2 describes scene content and motion, and DeepSeek-V3 checks for countable entities and self-initiated, non-rigid, or articulated motion. Clips dominated by camera motion, background jitter, wind-driven oscillation, or chaotic crowds are excluded. This filtering directly serves the representation: if camera motion is mistaken for object motion, the generator may learn whole-scene drift as a valid dynamic pattern. Retained videos are processed by DELTA with monocular depth estimation to obtain dense, occlusion-aware trajectories. Each first-frame pixel is associated with later image coordinates, depth, and occlusion status, rather than only matching a sparse set of salient points.

The four-component vector in dataset construction contains horizontal and vertical image coordinates, depth, and an occlusion flag; it is not four spatial coordinates. The generated "4D scene" instead means 3D points evolving over time, so these uses of dimensionality should not be conflated. Further filtering rejects invalid or anomalous depth, excessive scene-depth standard deviation, and samples failing rendering consistency under uniform scaling from the original camera. The final dataset contains 60,000 samples at \(596\times336\); the authors report over 3 million frames and approximately 12 billion 3D point annotations. These annotations are extracted and cleaned pseudo-ground truth, not precisely measured 3D dynamics from sensors. Gaussian Splatting renders and hole masks are also stored, providing paired supervision for the later view-completion model.

2. Normalized Trajectory Generation: predict relative 3D displacement on the first-frame pixel grid

UniDepthv2 first estimates depth for the input image, which is backprojected into an initial 3D point cloud. Matching the depth-estimation setup at training and inference reduces systematic differences between trajectory supervision and inference inputs. The representation contains \(N=H\times W\) points whose indices stay attached to first-frame pixels; subsequent time steps change only their 3D coordinates. Rather than repeatedly predicting absolute positions, 4D-STraG predicts displacement relative to each initial point and reconstructs positions through:

\[ \Delta P_t=P_t-P_0,\qquad P_t=P_0+\Delta P_t. \]

Here \(t\) denotes scene time, not the interpolation-time variable used in flow-matching training. Fixed indices make the dense field record how the same point moves at each grid location, avoiding correspondence estimation after generation. However, an identical 3D displacement produces different screen-space motion at different depths, so direct regression of raw displacement introduces substantial scale variation. The authors therefore normalize lateral displacement by the viewing-frustum width and height at the initial depth, and depth displacement by that initial depth. Frustum dimensions depend on focal length, image dimensions, and initial point depth, encouraging motion relative to viewing scale instead of a preference for particular absolute depth ranges. This neither corrects erroneous depth nor establishes true physical scale; inference must still denormalize using the same initial geometry.

A shallow ResNet trajectory encoder maps normalized displacement into RGB motion maps, which an adapted VAE compresses into latent space. RGB here carries motion values in an intermediate representation, rather than the final color video shown to the viewer. Temporal color changes at a fixed pixel index encode motion of its corresponding 3D point, while the spatial layout remains aligned with the first frame. Object displacement thus becomes color evolution on a fixed grid, matching a video-like data format without sacrificing point identity. A trajectory decoder after the VAE decoder recovers displacement from predicted motion maps instead of treating color directly as 3D coordinates. The DiT concatenates image, noise, and initial-depth latents along the feature dimension and learns future trajectory distributions under image-text conditioning. Depth latents constrain scene layout, whereas normalization addresses numerical scale; these roles are not interchangeable.

3. Local Motion Perception: translate video priors into fixed-grid motion conditioning

Wan2.1 is pretrained to move objects through images, whereas the motion-map representation requires a stationary layout with temporally changing colors. Changing the training target alone may not transfer this temporal knowledge effectively. MPM extracts patch-level OmniMAE features from the static input image, supplying priors about semantic regions that could plausibly move. These features are neither observed future trajectories nor conditions extracted from a real future video. The model aligns features with the DiT token sequence, then uses linear layers to produce scaling and bias parameters for each token.

Motion-aware Adaptive Normalization (MAdaNorm) applies these local parameters to normalized DiT features, together with learnable global gates in attention and feed-forward computation. Compared with one global conditioning vector, this lets different spatial regions receive different motion cues. It therefore informs regional trajectory variation instead of assigning a single motion-strength label to the whole image. An ablation replaces patch features with a global [CLS] token, reducing dynamics and aesthetic quality while slightly increasing consistency. The benefits of local conditioning concern motion expressiveness and the overall quality profile, not guaranteed improvement on every metric. The text extraction of original Equations (4)–(6) is incomplete, so this account retains the verifiable mechanism without reconstructing exact gating equations.

4. Geometry-Guided View Completion: constrain observed regions with points and fill holes with generative priors

Predicted trajectories are denormalized and added to the initial point cloud to obtain point positions at each frame. For a target camera, each frame's point cloud is rasterized as a set of 3D Gaussians, with these per-frame Gaussians linked over time. This is a lightweight dynamic renderer, not expensive per-scene optimization of a complete dynamic 4D Gaussian Splatting field. Points provide explicit projection constraints, but a single image offers limited surface coverage, leaving disocclusions and projection holes in new views. 4D-ViSM therefore receives rendered video and masks, using a video-generation prior to recover missing appearance.

Also based on Wan2.1, 4D-ViSM trains on TrajScene-60K renders, occlusion masks, ground-truth videos, and descriptions. Following the chosen Wan2.1 mask convention, regions without projected points receive a value of 0.5; this should not be casually replaced by a conventional binary hole mask. Training targets supervise complete appearance, whereas inference only has renders and masks produced from predicted point clouds. Geometry and completion thus have distinct responsibilities: the former determines projections of represented surfaces, and the latter generates pixels unsupported by the single observation. Producing videos along different camera paths does not establish a unique, verifiably true 3D surface in unseen regions. This distinction also limits how strongly the visual-consistency scores below should be interpreted.

A Worked Example¶

Consider the windsurfing scene in Figure 4: the input is an image of a windsurfer at sea and an accompanying motion description. Initial depth places the windsurfer and distant water at different 3D positions, while MPM supplies local motion cues from the static image. The model generates 49 frames of relative displacement on the fixed first-frame grid, animating the points instead of first producing an RGB video for DELTA to reconstruct. After denormalization, the same dynamic point-cloud sequence can be projected along two user-specified camera paths. When camera motion reveals previously unseen regions, 4D-ViSM completes the corresponding video pixels. This example explains the pipeline illustrated in Figure 4 without inventing actual speed, numerical motion directions, or camera-path parameters.

Loss & Training¶

The trajectory backbone uses Wan2.1-14B and staged adaptation, avoiding simultaneous learning of a new trajectory representation and the complete generative distribution from the outset. Trajectory-related components and the VAE decoder are trained for 5k steps, followed by 2k steps for the DiT with OmniMAE motion conditioning. The DiT uses flow matching to align its predicted vector field with the true field along a noise-to-data interpolation path; the trajectory backbone receives full-parameter training. The view-completion module is separately adapted with LoRA for 10k steps, so the final system is not one model trained jointly end to end in a single stage. Training uses AdamW with a learning rate of \(2\times10^{-5}\), and experiments run on 4 NVIDIA H20 GPUs. Outputs contain 49 frames at \(512\times368\), distinct from the original dataset-construction resolution. The cached text also loses symbols in Equations (1) and (3), so reconstructed normalization or flow-matching formulas are not presented as exact original equations.

Key Experimental Results¶

Main Results¶

The paper separates evaluation into three groups; MoGe4D scores across these groups should not be merged into a single leaderboard. Group I uses official demonstration videos from closed-source 4Real and simple camera paths; Group II compares GenXD and DimensionX's S-Director with moderate 90° left rotations. Group III compares Free4D and Gen3C under complex trajectories including upward, forward, leftward, rightward, and downward 90° movements. Groups II and III use 200 held-out WebVid-10M samples disjoint from the training subset; this does not guarantee generalization to arbitrary real scenes.

The following excerpt is from Table 1, page 11, Group III only; all entries are VBench scores, higher being better.

Method	Subject Consistency	Background Consistency	Motion Smoothness	Dynamic Degree	Aesthetic Quality	Imaging Quality
Free4D	0.7899	0.8883	0.9797	1.0000	0.3607	0.3562
Gen3C	0.8112	0.8871	0.9845	0.9940	0.3812	0.4814
MoGe4D	0.8339	0.9065	0.9773	0.9000	0.4820	0.5939

Imaging quality increases from Gen3C's 0.4814 to MoGe4D's 0.5939, an absolute gain of 0.1125; subject consistency, background consistency, and aesthetic quality also improve. However, motion smoothness and dynamic degree fall below both baselines, so the result is not "more dynamic and better on every metric." Table 2, page 12, uses Qwen2.5-VL-72B-Instruct to score 8 uniformly sampled frames per video on a 1–5 scale across geometry, texture, identity, motion-geometry coupling, and background stability. Group III averages are 3.76 for MoGe4D, 2.22 for Gen3C, and 1.23 for Free4D; these are VLM perceptual assessments, not direct 3D measurements.

Ablation Study¶

The following results come from Table 4, page 13, under the Group III VBench setting; the original aggregate name Consistency is retained because the main text does not explicitly define its aggregation formula for this table. The global [CLS] row corresponds to "w/o Patch Feat."; small-scale data means a 1k subset, and low-quality data means unfiltered 60k samples.

Config	Consistency ↑	Dynamic ↑	Aesthetic ↑
Without depth normalization	0.8604	0.8850	0.4672
Without depth latents	0.8567	0.8500	0.4738
Without MPM	0.8650	0.8500	0.4806
MPM with global [CLS]	0.8743	0.8840	0.4754
Small-scale training data	0.8567	0.8920	0.4791
Low-quality training data	0.8549	0.8850	0.4654
Random noise during training	0.8685	0.8967	0.4771
MoGe4D full model	0.8702	0.9000	0.4820

Key Findings¶

Removing MPM lowers Dynamic from 0.9000 to 0.8500, an absolute drop of 0.0500; removing depth latents also produces 0.8500, supporting the importance of both motion and geometry conditioning.
Global [CLS] obtains Consistency of 0.8743, above the full model's 0.8702. The claim that the full model is best on every metric does not fully agree with the table; individual entries should take precedence.
In Table 3a, page 13, the complex-trajectory block containing Free4D and Gen3C reports trajectory errors of 0.252, 0.197, and 0.042 (MoGe4D), respectively, and reprojection errors of 0.652, 0.755, and 0.639.
Trajectory error is an L2 distance between DELTA tracks extracted from generated videos and ground-truth tracks; reprojection error uses DROID-SLAM on co-visible pixels in consecutive frames. Physical units are not specified in the table, so these should not be described as accuracy in meters or pixels.
In the other block of Table 3a, DimensionX's reprojection error of 0.602 is slightly better than MoGe4D's 0.614, further showing that geometric benefits are not universal wins.
Table 3b, page 13, reports 6 min for 49 frames on one A100 with MoGe4D; Free4D takes 30 min for 16 frames at the same \(512\times368\) resolution. The claimed 5-fold speedup compares total times, not strictly matched frame-count and quality throughput.

Highlights & Insights¶

Fixing first-frame point indices is a central representation choice: the model always knows which initial surface point a trajectory belongs to. This reduces ambiguity from changing appearance first and estimating correspondence afterward.
Depth enters both numerical normalization and latent conditioning. The former handles scale distribution while the latter constrains scene structure, explaining why their separate ablations both matter.
Encoding motion as color variation lets a video-generation backbone learn a non-appearance signal. The transferable lesson is to design a pretrained-model-compatible representation that preserves task correspondence, rather than treating video generation as a general geometry solver.

Limitations & Future Work¶

On pages 14–15, the authors explicitly identify harsh backlighting and chaotic crowds as failure cases: monocular-depth degradation contaminates geometric conditioning and makes motion less reliable.
Filtering deliberately removes many unstructured and camera-dominated videos, so limited long-tail dynamics reflect the training distribution as well as model capacity.
Reader assessment: first-frame visible points cannot fully cover back surfaces or newly revealed areas; realistic generative completion does not substitute for validation of true geometry.
Reader assessment: VLM ratings and geometry metrics based on DELTA or DROID-SLAM inherit estimator biases; DELTA also generates training pseudo-ground truth, so these results are not independent sensor-ground-truth validation.
The authors propose stronger visual foundation models as future work; evaluating uncertain depth, unobserved regions, and longer temporal dynamics are further suggestions motivated by current limits, not experiments already completed in this paper.

Versus 4Real, Free4D, and DimensionX: generate-then-reconstruct approaches exploit powerful video priors but pass video inconsistencies into 3D reconstruction; MoGe4D moves geometry conditioning into trajectory generation at the cost of stronger dependence on initial depth quality.
Versus reconstruct-then-generate approaches such as Gen3C: the paper retains geometry-first reasoning but represents motion as dense relative trajectories; it neither abandons static geometry entirely nor merely adds preset animation to a static asset.
Versus Wan2.1-I2V + DELTA / VGGT: Figure 8 on pages 14–15 shows background drift and fragmentation in sequential pipelines, motivating direct geometry-conditioned trajectory generation; this is qualitative evidence in the main text, not an unreported numerical gain.

Rating¶

Novelty: 4/5. Dense trajectories, depth-scale handling, and local motion conditioning form a coherent single-image 4D generation approach.
Experimental Thoroughness: 4/5. Video quality, VLM consistency, geometric errors, and component ablations are covered, but group protocols differ and pseudo-ground truth and evaluators have limitations.
Writing Quality: 3/5. The method is logically clear, but the all-metrics-best ablation claim overstates the evidence, and some metrics and equations require consulting the fully typeset original.
Value: 4/5. The approach offers useful ideas for geometry-conditioned dynamic scene generation, while remaining far from a reliable open-world physical scene model.