One4D: Unified 4D Generation and Reconstruction via Decoupled LoRA Control¶

Conference: ECCV2026
Paper: ECCV Paper
Project: https://mizhenxing.github.io/One4D
Area: 3D Vision
Keywords: 4D generation and reconstruction, pointmaps, video diffusion model, decoupled LoRA, unified masked conditioning

TL;DR¶

One4D encodes varying numbers of observed frames through unified masked conditioning and jointly generates appearance and geometry using separately computed RGB/XYZ LoRA branches with shared frozen video weights and sparse bidirectional control links, supporting single-image 4D generation, sparse-frame completion, and full-video reconstruction in one model.

Background & Motivation¶

Video generation models can synthesize complex motion, but their RGB outputs do not directly specify where each pixel lies in 3D or how the camera moves. Meanwhile, methods such as DUSt3R and MonST3R represent geometry as pointmaps: each image pixel has an XYZ coordinate, giving appearance and geometry a natural pixel-level correspondence. Extending a video model to generate both RGB and XYZ may look like adding another output, but it requires preserving appearance priors while learning a very different coordinate distribution. This also motivates video-level pointmaps instead of depth alone: XYZ in a shared coordinate system constrains subsequent camera recovery.

Existing geometry reconstruction methods often concatenate clean RGB and noisy geometry along channels because observed images provide stable conditions. During joint generation, both modalities are noisy, so premature shared computation can let poorly adapted geometry disrupt video priors; the paper observes channel-wise concatenation collapsing under a limited training budget. Spatial-wise concatenation alleviates this problem but relies on attention to discover cross-modal positional correspondence, still producing misaligned RGB and geometry boundaries. Moreover, designing separate interfaces for single-image generation and full-video reconstruction splits the continuously varying question of observation density across different models.

One4D therefore addresses both computational coupling and task conditioning: it protects modality-specific adaptation before allowing controlled pixel-wise information exchange. It does not train a video foundation model from scratch, but adapts an existing video inpainting model to jointly output appearance and coordinate videos. Core Idea: encode observation density with a unified mask, preserve separate modality computation through decoupled LoRA, and learn appearance-geometry consistency through sparse bidirectional links initialized to zero.

Method¶

Overall Architecture¶

The inputs are a text prompt and observed images at specified temporal positions, ranging from a single frame to a complete video. The outputs are synchronized RGB and XYZ pointmap videos, both shaped \(3\times F\times H\times W\), storing colors and 3D coordinates respectively. Training pointmaps use the first-frame camera as the global reference and are normalized to \([-1,1]\); different frames are therefore not independent local depth maps. A video VAE separately encodes the two videos into spatiotemporal latents suitable for the same type of video DiT. Unified masked conditioning first enters the RGB path, decoupled modality branches predict their own denoising velocities, and sparse bidirectional control links exchange alignment information. Decoded pointmaps already constitute geometry; the final camera-depth recovery fits cameras and depths to those observations rather than reconstructing geometry from RGB anew.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Observed images and text"] --> UMC["Unified Masked Conditioning"]
    UMC -->|Image conditions enter RGB only| Branch["Decoupled Modality Branches"]
    Noise["RGB and XYZ noise"] --> Branch
    Branch --> Links["Sparse Bidirectional Control Links"]
    Links -->|Separate computation continues in later layers| Decode["VAE decoding<br/>RGB and XYZ"]
    Decode --> Recover["Camera-Depth Recovery"]
    Recover --> Output["Dynamic point clouds,<br/>cameras and depths"]
    Supervision["Training RGB and pointmaps"] -.->|Dual-modality velocity supervision| Branch

Key Designs¶

1. Unified Masked Conditioning: express generation and reconstruction through one interface

The model constructs a conditioning video as long as the target sequence, inserting images at observed positions and zeros elsewhere. It also builds a binary observation mask, distinguishing an unknown frame from a frame that genuinely contains black pixels. The conditioning video is VAE-encoded and the mask reshaped to latent resolution; both are concatenated with noisy RGB latents along channels. Equation (8) is legible in the source; the notation below uses semantic subscripts to identify the RGB denoising input:

\[ z_{\mathrm{input}}=\operatorname{Concat}(z_{\mathrm{rgb}},z_{\mathrm{c}},M_{\mathrm{c}}). \]

Single-image conditioning mainly relies on priors to generate subsequent content, full-video conditioning recovers geometry at observed times, and sparse frames combine completion with reconstruction. Unification does not make the information available in these tasks identical: the network stays fixed while the conditioning video and mask change. The XYZ branch always generates all pointmaps and does not directly receive the conditioning video or observation mask; the corresponding input slots in the architecture diagram are zero-padded. Observation information reaches geometry indirectly through the RGB branch and control links, avoiding direct propagation of the observation mask pattern into geometry artifacts. Thus, avoiding direct channel concatenation of two noisy output modalities does not conflict with concatenating RGB and clean conditions: these serve different purposes.

2. Decoupled Modality Branches: share foundation weights, not modality computation

The backbone is Wan2.1-Fun-V1.1-14B-InP, a video-inpainting adaptation of Wan2.1-I2V-14B. Duplicating an entire 14B backbone for geometry and attaching separate adapters would incur substantial parameter storage costs. One4D retains one set of frozen base parameters and attaches separate RGB and XYZ LoRA adapters to all DiT linear layers. Each base submodule runs once per modality with its own low-rank update, rather than processing both token sets in a single mixed computation. Consequently, shared weights do not mean shared activations, nor does dual-modality computation cost the same as a single RGB model.

The RGB path continues adapting within the pretrained video distribution, while the XYZ path learns to apply video priors to coordinate sequences. Separating modality updates reduces interference from the initially large geometry distribution shift into appearance generation. Unlike spatial-wise concatenation, each attention operation need not jointly model the modalities in a doubled-length mixed sequence. The claimed memory and compute advantages are relative to full parameter duplication or mixed long sequences, not a claim of zero added cost. Both LoRA adapters use rank 64 and together contain 685M trainable parameters; the base video model remains frozen.

3. Sparse Bidirectional Control Links: progressively learn cross-modal constraints at corresponding positions

Fully independent RGB and XYZ outputs can each look plausible while describing different object boundaries or motion. DLC therefore adds bidirectional linear control links between a few corresponding DiT layers, mapping one modality's features and adding them pixel-wise to the other. The two directions let geometry influence appearance and appearance influence geometry, unlike ControlNet's one-way conditioning. The linear links are initialized to zero, so they initially introduce no cross-branch perturbation and gradually learn useful cross-modal information. Zero initialization guarantees that the control path has no initial effect, not that an untrained XYZ branch already predicts correct geometry.

The actual model links only 5 DiT layers, adding 250.7M parameters rather than enforcing dense fusion at every layer. Sparse links limit where interference occurs, while pixel-wise addition gives a more direct correspondence than cross-spatial attention. These links operate inside iterative denoising, not as a final alignment step after separately generating RGB and pointmaps. Observed images also influence XYZ through these links, so they carry geometry conditioning as well as modality consistency information. Spatial-concatenation comparisons support DLC as a whole, but the supplied main text does not separately quantify the effects of zero initialization, bidirectionality, and link count.

4. Camera-Depth Recovery: interpret pointmaps as camera observations

After generation, every time step already has an XYZ map in a shared coordinate system, which can be converted into cameras and depth for downstream use. Post-processing fits per-frame intrinsics, world-to-camera rotation, global camera center, and a depth map. Intuitively, back-projecting a pixel with its intrinsics, scaling its ray by depth, and transforming it into world coordinates should reproduce its generated XYZ point. The optimization uses L1 pointmap alignment as its data term and constrains changes in adjacent camera rotations and translations to reduce trajectory jitter. This follows global-optimization ideas from MonST3R and Geo4D rather than introducing a new explicit scene representation.

Because this fitting starts from existing geometry, the authors report approximately 30 seconds for 500 post-optimization iterations. It extracts cameras and depths compatible with the pointmaps, but does not establish that all generated motion obeys physical laws. Smoothing camera trajectories cannot automatically remove incorrect structures already present in the predicted pointmaps. Equations (1), (2), (7), (9), and (10) contain clearly missing characters or operators in the cached extraction; this note does not present guessed repairs as exact author formulas. The velocity training, pixel-wise residual links, and camera fitting described above follow the legible prose; exact damaged equations should be checked in the paper PDF.

A Worked Example¶

Consider an input containing only the first and last images of an 81-frame output; this illustrates the interface and is not an additional experiment. The conditioning video contains observations at its endpoints and zeros elsewhere, with the mask marking the two known times and text describing content and motion. Sampling starts from RGB and XYZ noise; the RGB path reads the conditions while the XYZ path receives indirect constraints through control links. The model fills the intermediate RGB frames and generates all 81 pointmaps, including the endpoints that have RGB observations but no observed geometry. After decoding colors and coordinates, camera trajectories and depths are fitted so the dynamic point cloud can be inspected over time. Providing all 81 frames only changes the conditions and mask; the network still performs joint prediction, but the task becomes full-video geometry reconstruction.

Loss & Training¶

Training separately encodes RGB and pointmaps, constructs both noisy inputs at the same sampled timestep, and supervises their respective velocity predictions. The prose specifies mean-squared error between predicted and target velocities; because the extraction is damaged, this note does not add unverified modality weights or time-direction formulas. The data comprises approximately 17k synthetic and 17k real clips, totaling about 2M frames, with Geo4D producing pseudo geometry for real SpatialVID videos. Synthetic data comes from OmniWorld-Game, BEDLAM, PointOdyssey, and TarTanAir, providing relatively accurate geometric supervision. Long videos are segmented into roughly 81-frame clips and captioned with Gemini-2.0-Flash; real data broadens appearance and motion coverage. Single-image, sparse-frame, and full-video tasks are sampled with probabilities 0.35, 0.30, and 0.35, respectively, by changing masks. The model has 935.7M trainable parameters and trains for 5,500 steps on 8 H800 GPUs, with batch size 1 per GPU, gradient accumulation 4, and learning rate \(1\times10^{-4}\). Maximum training length is 81 frames at \(352\times624\); inference uses 50 flow-matching steps and default CFG 6.0. Generating a video of this size takes approximately 12 minutes on one H800 with 57.7GB peak VRAM, followed by approximately 30 seconds of post-optimization. Thus, a modest training budget is relative to training or heavily adapting large generators, not evidence of consumer-grade real-time deployment.

Key Experimental Results¶

Main Results¶

The following results come from Table 2, page 11, comparing generated videos with VBench; all three metrics are higher-is-better and none measures geometry accuracy.

Method	Dynamic	I2V consistency	Aesthetic
4DNeX	25.6%	98.7%	61.9%
One4D	55.7%	97.8%	63.8%

Dynamics improve by 30.1 percentage points and aesthetics by 1.9 points, but I2V consistency drops by 0.9 points; not every video metric improves. The user studies in Table 1 recruited separate pools of 11, 6, and 11 participants for the respective opponents, each evaluating 10 cases using anonymous randomized pairs with the same image and prompt. Against 4DNeX, 4D preference was 90.0% for One4D and 10.0% for 4DNeX; these are small-sample preferences, not objective geometry accuracy.

The next table excerpts Table 3, page 12, under full-video conditioning; predicted depth is aligned to ground truth, with lower Abs Rel and higher threshold inlier percentages preferred. Abs Rel averages absolute depth error relative to ground truth; an inlier has a maximum of the predicted-to-ground-truth depth ratio and its reciprocal below 1.25. R denotes reconstruction only and G&R unified generation and reconstruction; Geo4D-ref also supplies pseudo labels for the real training data, so it is not an unrelated reference.

Method	Task	Sintel Abs Rel	Sintel \(\delta<1.25\) (%)	Bonn Abs Rel	Bonn \(\delta<1.25\) (%)
MonST3R	R	0.335	58.5	0.063	96.4
CUT3R	R	0.311	62.0	0.070	96.7
Geo4D-ref	R	0.205	73.5	0.059	97.2
One4D	G&R	0.273	70.4	0.092	93.7

One4D's strength is retaining substantial geometry capability within a unified model: it outperforms MonST3R and CUT3R on Sintel but trails both on the two Bonn metrics. Table 4, page 13, also reports TUM-dynamics ATE of 0.129 versus 0.073 for Geo4D-ref and 0.063 for MonST3R; the word comparable does not establish leading camera estimation.

Ablation Study¶

The following results come from Table 8, page 15: all models train for 1,000 steps on the same data and settings and are evaluated on Bonn; \(s\) is the observed-frame fraction, not a spatial-resolution ratio.

Method	Full Abs Rel	Full inliers (%)	\(s=0.05\) Abs Rel	\(s=0.05\) inliers (%)	\(s=0.04\) Abs Rel	\(s=0.04\) inliers (%)
Spatial-1k	0.121	87.4	0.186	76.8	0.227	73.2
One4D-1k	0.114	88.9	0.167	83.0	0.208	76.9

All inlier columns use \(\delta<1.25\); DLC improves both metrics at all three observation densities. At \(s=0.05\) specifically, Abs Rel decreases from 0.186 to 0.167 and inliers increase from 76.8% to 83.0%, a gain of 6.2 percentage points. This is more direct architectural evidence than cross-paper comparisons with substantially different training budgets, although it tests DLC as a complete package.

Key Findings¶

Table 5, page 13: reducing Sintel observations from the full video to 0.25 only changes inliers from 70.4% to 67.7%, but raises Abs Rel from 0.273 to 0.443, revealing different metric sensitivities to sparsity.
Table 7, page 15: synthetic-only training for 1,000 steps gives Sintel Abs Rel 0.281 versus 0.331 with mixed data; adding pseudo-labeled real videos does not monotonically improve reconstruction metrics.
Table 6, page 14: CFG=4 gives Sintel Abs Rel 0.257, better than the default's 0.273; default CFG=6.0 is not optimal for depth accuracy.

Highlights & Insights¶

Parameter sharing is explicitly distinguished from computation sharing. Foundation weights can be reused while modality activations and low-rank updates remain separate, suggesting applications to other appearance-plus-dense-prediction tasks.
Conditioning density acts as the task switch. Generation and reconstruction no longer require different networks, but training must still cover the observation patterns; a unified interface does not imply equal accuracy across them.
Zero-initialized links place prior preservation and pixel alignment within the same training process. They can be understood as progressively learned information exchange rather than an additional hand-designed geometry-consistency loss.

Limitations & Future Work¶

The authors acknowledge dependence on backbone generalization to target motion and camera patterns, with ambiguous geometry and complex non-rigid motion remaining difficult; broader data and physical priors are future directions.
Reader assessment: sparse evaluation retains first and last frames and samples uniformly; intermediate occlusion, missing boundaries, and strong distribution shifts are not fully covered by this protocol.
Reader assessment: real videos use Geo4D pseudo labels and synthetic-only training outperforms mixed data in the reconstruction ablation, motivating separation of supervision noise from visual-diversity effects.
Approximately 12 minutes of sampling and 57.7GB peak VRAM remain substantial resource requirements; fewer training steps do not establish low inference cost.
Some equations are damaged in the available extraction while table values and methodological prose are legible; this note does not transcribe damaged equations exactly or verify supplementary material that was not provided.

Compared with Geo4D: Geo4D reconstructs geometry from observed video and provides pseudo supervision here; One4D must also generate unknown RGB, so strong coupling designed for clean RGB conditioning cannot simply be reused.
Compared with 4DNeX: both target dynamic RGB+XYZ, but One4D replaces spatial concatenation with separate computation and control links, while unifying full-video and sparse-condition tasks.
Compared with WVD: WVD uses RGB+XYZ and channel concatenation for static scene generation; One4D emphasizes dynamic modeling under limited adaptation, but differences in data, hardware, and tasks preclude a strict speed comparison.
Compared with ControlNet and JointNet: One4D applies control to sparse bidirectional communication between video RGB and shared-coordinate pointmaps, rather than only one-way injection or dense coupling of image-level branches.

Rating¶

Novelty: 4/5. Combining a unified observation interface with shared parameters and decoupled computation is well targeted, although pointmaps and zero-initialized control have established foundations.
Experimental Thoroughness: 4/5. Generation, full reconstruction, sparse inputs, and matched-budget architecture ablations are covered, but control-link details lack separate attribution.
Writing Quality: 4/5. The main argument is clear, but claims of near-full performance under sparsity and comparable cameras require metric-specific interpretation.
Value: 4/5. The approach offers a reusable adaptation route for explicit geometry in video foundation models, with deployment cost and complex-motion robustness still needing improvement.