4DGS360: 360° Gaussian Reconstruction of Dynamic Objects from a Single Video

Conference: ECCV2026
Paper: Official page · PDF
Project: 4DGS360
Area: 3D Vision
Keywords: Dynamic Gaussian reconstruction, occluded geometry, anchor-guided tracking, hierarchical motion, novel view synthesis

TL;DR

4DGS360 uses confident 2D tracks as spatiotemporal anchors for 3D tracking, initializes motion with geometry that includes occluded regions, and then optimizes dynamic Gaussians without diffusion in its base pipeline, reducing mean LPIPS on five iPhone scenes from HiMoR's 0.4639 to 0.3877.

Background & Motivation

Monocular dynamic reconstruction is difficult not simply because views are sparse, but because both the camera and the object move: each instant supplies only one image, preventing the direct combination of simultaneous geometric evidence available in static multi-view reconstruction. Methods such as HiMoR and Shape of Motion establish cross-frame correspondences with 2D tracking and lift them into 3D using depth maps. However, a depth map describes only the currently visible surface. Once a tracked point becomes occluded, the depth at its projected position can belong to an occluder rather than the point itself. Even a backside observed elsewhere in the video can therefore be incorrectly pulled toward the visible surface at the current time.

This error enters initialization, not just the final image. Local rigidity can preserve an existing structure, but it cannot easily recover the correct structure from already corrupted initial geometry. Simply switching to TAPIP3D is insufficient: 3D tracking captures spatial continuity but is sensitive to depth and camera calibration errors, and long propagation can accumulate drift. The paper therefore combines reliable 2D correspondence with 3D trajectory inference, rather than lifting every 2D prediction indiscriminately or abandoning 2D tracking entirely.

Core idea: use visible, high-confidence 2D correspondences as spatiotemporal anchors for 3D tracking, let the 3D model infer occluded trajectory segments, and pass the resulting more complete geometry to an existing hierarchical motion representation and rigidity-based optimization.

Method

Overall Architecture

Inputs are a single video, per-frame depth, and camera parameters. Depth and camera information may come from sensors or pretrained estimators, so "single video" does not mean RGB-only reconstruction without additional priors. The output is a time-varying 3D Gaussian representation that can render the dynamic object at a selected time from a new camera position.

The pipeline first estimates visible and occluded trajectories through "Anchor-Guided 3D Tracking," then builds canonical Gaussians and hierarchical motion nodes through "Trajectory-Driven Hierarchical Initialization," and finally applies "Joint Geometry and Rendering Optimization." The central contribution is initialization: the hierarchical motion structure follows HiMoR, and the paper argues that better initial geometry allows subsequent optimization to work effectively.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Video, depth<br/>and camera parameters"] --> B["2D tracking<br/>Confident correspondences"]
    A --> C["Anchor-Guided<br/>3D Tracking"]
    B --> C
    C --> D["Trajectory-Driven<br/>Hierarchical Initialization"]
    D --> E["Joint Geometry and<br/>Rendering Optimization"]
    E --> F["Dynamic Gaussians<br/>Full-circle novel-view rendering"]

Key Designs

1. Anchor-Guided 3D Tracking: constrain occluded inference with trustworthy observations

AnchorTAP3D starts from a query point in a frame and uses a 2D tracker to predict correspondences and confidence in other frames. Only correspondences above a confidence threshold are unprojected into 3D anchors using the target frame's depth and camera parameters. Low-confidence points often involve occlusion, so the visible surface depth at their predicted image positions cannot safely be treated as their actual depth. This selection does not delete an entire trajectory; it determines which observations qualify as constraints for the 3D model.

The 3D tracker jointly processes frames within a sliding temporal window and conditions trajectory and binary visibility predictions on multiple anchors in that window. The main text gives an example with a window length of 16 and an overlap of 8 frames. These anchors provide reliable locations across time, reducing the drift associated with propagating a single query. Unlike independently unprojecting and concatenating 2D tracks, the model infers an occluded point's position in 3D instead of assigning it the wrong visible depth during occlusion.

Restating Eqs. (8)-(11) with consistent notation, the window's anchor set is:

\[ \mathcal{A}_{w}=\left\{\pi_s^{-1}(p_s,D_s,\theta_s)\;\middle|\;s\in w,\ c_s>\tau\right\}. \]

Here, \(p_s\) is the 2D correspondence, \(D_s\) and \(\theta_s\) are the frame's depth and camera parameters, \(c_s\) is 2D tracking confidence, and \(\tau\) is the selection threshold. This set specifies which trustworthy observations condition inference; it does not imply that anchors have error-free depth. The main text neither specifies a numerical threshold nor details the exact anchor injection mechanism inside the Transformer, so an attention-layer design or coordinate replacement rule cannot be supplied as fact. The tracking modification requires no additional training, but scene reconstruction still requires optimization.

2. Trajectory-Driven Hierarchical Initialization: include occluded positions in the motion structure

After obtaining 3D tracks from multiple query times, the method randomly samples trajectories to initialize Gaussians and chooses the frame containing the most visible Gaussians as the canonical frame. It clusters trajectories by temporal velocity using k-means and estimates frame-to-frame rigid transformations within each cluster through Procrustes alignment, producing initial motion bases. Gaussian motion is additionally weighted by spatial distance to the corresponding cluster center. Whereas 2D lifting methods often discard invisible points during motion estimation, the occluded positions inferred by AnchorTAP3D allow initialization to retain more of the object's global motion structure.

The motion representation uses HiMoR's node hierarchy: upper levels describe shared global motion, deeper levels refine local deformation, and nearby leaf-node motions are interpolated to obtain each Gaussian's time-dependent transformation. Node initialization uses weighted sampling based on both motion magnitude and spatial density, allocating capacity to highly dynamic areas while preserving spatial coverage. The representation does not fit an unrelated Gaussian cloud to every frame. The same canonical Gaussians follow hierarchically organized trajectories, allowing geometry at different times to constrain one another.

3. Joint Geometry and Rendering Optimization: establish reliable structure before propagating rigidity

After initialization, optimization updates motion nodes and Gaussians together. Generalized ARAP selects node pairs within the same locally rigid cluster and encourages their distances and relative local transformations to remain consistent between arbitrary times. Because the constraint is not limited to adjacent frames, it can propagate structural information from a visible instant to a temporally distant occluded instant. Its justification is approximate local rigidity, not an assumption that the entire dynamic object is rigid.

Rendering supervision corrects the representation from the image side: the RGB term includes D-SSIM and LPIPS, the mask term encourages object compactness, depth consistency constrains geometric alignment, and 2D tracking constrains temporal correspondence. These signals complement ARAP but cannot create observations that were never available. The paper's argument is that ARAP may merely preserve mistakes under corrupted initialization; after anchors improve the starting structure, the same type of constraint is better positioned to preserve correct backside geometry.

A Worked Example

Consider the occlusion behavior illustrated by the paper's walk-around scene: a surface point is visible in earlier frames, becomes occluded during rotation, and later reappears. Direct 2D unprojection can read another surface's depth during the occlusion, whereas unanchored 3D tracking may drift over a long sequence. In the example setting with 16-frame windows and 8-frame overlap, AnchorTAP3D uses confident visible correspondences as anchors, lets the 3D model connect the occluded trajectory segment, and passes those tracks to motion clustering and Gaussian initialization.

Hierarchical nodes then describe global motion and local changes, ARAP preserves local structure, and image losses correct visible regions before rendering viewpoints away from the training camera. This is an explanation of the mechanism, not a new tracking measurement: the walk-around ablation reports qualitative reconstruction differences only.

Loss & Training

The overall objective from Eq. (13) is:

\[ \mathcal{L}_{\mathrm{total}}= \lambda_{\mathrm{rgb}}\mathcal{L}_{\mathrm{rgb}}+ \lambda_{\mathrm{mask}}\mathcal{L}_{\mathrm{mask}}+ \lambda_{\mathrm{depth}}\mathcal{L}_{\mathrm{depth}}+ \lambda_{\mathrm{2dtrack}}\mathcal{L}_{\mathrm{2dtrack}}+ \lambda_{\mathrm{arap}}\mathcal{L}_{\mathrm{arap}}. \]

The weights control the relative contributions of image, mask, depth, tracking, and rigidity terms. The cache contains the main paper and references, but not the supplementary implementation details, so loss weights, iteration counts, runtime, and memory usage are not supplied here. iPhone360 is captured with multiple synchronized iPhones, but reconstruction training uses video from only one camera. LiDAR metric depth corrects the scale and shift of estimated depth, and sensor camera parameters serve as ground truth. Additional test cameras are for evaluation, not multi-view training input.

Key Experimental Results

Main Results

iPhone360 contains 6 dynamic scenes with training-to-test view disparities of 70°-135°. Evaluation constructs and expands a bounding box from the ground-truth dynamic object mask, applying the same region to ground truth and foreground-only renderings so that empty extreme-view backgrounds do not dominate comparisons. CLIP-I measures CLIP similarity between ground-truth and rendered images; CLIP-T measures CLIP similarity between rendered frames separated by 5 frames. Higher is better for both, while lower is better for LPIPS. Direction arrows in the cache are corrupted by text extraction; directions below follow the metric definitions.

The following representative scenes come from Table 2. They are individual scene results, not dataset-wide averages.

Scene	Method	CLIP-I (higher)	CLIP-T (higher)	LPIPS (lower)
Block2	MoSca	0.7499	0.9513	0.5556
Block2	HiMoR	0.8422	0.9549	0.3122
Block2	4DGS360	0.9021	0.9633	0.2569
Goat	MoSca	0.7379	0.9522	0.6332
Goat	HiMoR	0.8357	0.9426	0.3260
Goat	4DGS360	0.8706	0.9345	0.2244
Jelly	MoSca	0.6239	0.9649	0.7127
Jelly	HiMoR	0.6839	0.9357	0.3539
Jelly	4DGS360	0.8359	0.9229	0.3538

The five-scene iPhone averages from Table 3 provide another quantitative comparison and must not be pooled with the individual scene results above.

Method	Mean CLIP-I (higher)	Mean CLIP-T (higher)	Mean LPIPS (lower)
HyperNeRF	0.8750	0.9714	0.5261
Shape of Motion	0.8603	0.9712	0.4827
HiMoR	0.8907	0.9750	0.4639
4DGS360	0.9015	0.9754	0.3877

Ablation Study

Section 4.2 and Figure 8 contain qualitative ablations only, with no numerical ablation table. The following table summarizes the written observations without turning visual evidence into invented quantitative gains.

Configuration	Initialization	walk-around observation	jelly observation
w/o 3D init	Directly unprojected 2D tracks	Fails to preserve occluded geometry	Fails to preserve occluded geometry
w/o Anchor	Original TAPIP3D tracking	Partially recovers occluded geometry, better than 2D initialization	Fails to reconstruct the occluded region
4DGS360	AnchorTAP3D	Preserves overall shape	Preserves overall shape

Key Findings

• Across all six iPhone360 scenes, CLIP-I and LPIPS outperform the listed MoSca and HiMoR results, but CLIP-T does not improve uniformly. Goat's CLIP-T is 0.9345 versus HiMoR's 0.9426; Jelly's LPIPS changes only from 0.3539 to 0.3538.
• Mean iPhone LPIPS decreases by an absolute 0.0762, while mean CLIP-T increases by only 0.0004. Better perceptual image similarity and better temporal similarity are distinct claims.
• In Table 4, iPhone360 PSNR increases from 10.9883 to 11.3364 and SSIM from 0.6854 to 0.7012. However, iPhone SSIM decreases from 0.6329 to 0.6309, so not every pixel-level metric improves.
• DAVIS and the backpack scene without test ground truth provide qualitative evidence; the cache provides no numerical gains for these cases. Ablations support the role of anchors but do not quantify the independent contribution of each loss term.

Highlights & Insights

• The method separates reliable localization when visible from 3D inference when invisible. Instead of asking 2D tracking to resolve an ambiguity that the depth image cannot answer, it uses 2D tracks to constrain a geometry-aware predictor.
• Initialization changes the effectiveness of subsequent regularization. A transferable lesson is to inspect initial correspondences and structure when optimization persistently preserves incorrect geometry, rather than simply adding more losses.
• iPhone360 explicitly evaluates large-angle extrapolation. Attractive renderings near the training camera do not establish that backside and occluded geometry have been recovered.

Limitations & Future Work

• The authors acknowledge dependence on pretrained tracking and geometry estimators. Confident 2D correspondence does not guarantee correct depth or camera calibration; joint confidence estimation and robust treatment of anchor errors are possible extensions, not demonstrated capabilities.
• The pipeline cannot synthesize extreme-view background regions never observed in the input. Qualitative combinations with DIFIX3D+ do not establish unconditional completion in the base method.
• Pixel-level accuracy remains limited under large extrapolation. As an assessment of evaluation scope, neither CLIP semantic similarity nor temporal similarity independently verifies accurate 3D surfaces.
• iPhone360 has only 6 scenes, and the cached ablations lack numerical metrics, threshold sensitivity, and efficiency comparisons. More diverse motion, prolonged occlusion, and calibration-noise tests would clarify the method's operating limits.

Related Work & Insights

• Compared with HiMoR: the hierarchical motion representation is inherited, while trajectory initialization is the main change. Improvements cannot all be attributed to a new representation; the key question is whether occluded positions enter a reliable initial motion structure.
• Compared with TAPIP3D: 3D trajectory inference is retained, with confident 2D correspondences supplying multiple spatiotemporal anchors to address long-term drift and occlusion stability.
• Compared with Shape of Motion and MoSca: all organize dynamic reconstruction using motion trajectories. This paper emphasizes occluded-depth ambiguity in 2D-native initialization, suggesting a reusable initialization direction rather than proving a plug-and-play replacement for every tracker-based method.
• Compared with DIFIX3D+: diffusion priors can improve images from unobserved views, whereas the base 4DGS360 pipeline avoids diffusion and first seeks more consistent geometry from cross-frame evidence already present in the video.

Rating

• Novelty: 4/5. Combining reliable 2D anchors with 3D occlusion inference directly addresses a specific failure, although the hierarchy and major optimization components follow prior work.
• Experimental Thoroughness: 3/5. A new benchmark, multiple quantitative comparisons, and qualitative ablations are provided, but quantitative component ablations and efficiency reporting are missing.
• Writing Quality: 4/5. The argument from initialization failure to geometry recovery is clear; anchor injection details and some hyperparameters require supplementary material for reproduction.
• Value: 4/5. The method is practically relevant to large-viewpoint monocular dynamic object reconstruction and cautions against treating nearby-view quality as evidence of complete geometry.