D\(^2\)USt3R: Enhancing 3D Reconstruction for Dynamic Scenes

Conference: NEURIPS2025 arXiv: 2504.06264 Code: cvlab-kaist/DDUSt3R Area: 3D Vision Keywords: dynamic 3D reconstruction, pointmap regression, dense correspondence, optical flow, DUSt3R

TL;DR

This paper proposes the Static-Dynamic Aligned Pointmap (SDAP) representation, which unifies 3D alignment of static and dynamic regions into a single framework, enabling DUSt3R-based methods to achieve accurate dense 3D reconstruction and correspondence estimation in dynamic scenes.

Background & Motivation

DUSt3R achieves elegant feed-forward dense stereo reconstruction via direct pointmap regression, demonstrating strong performance in static scenes. However, real-world scenes frequently contain moving objects. Since DUSt3R aligns pointmaps solely based on camera pose, dynamic object regions suffer from corrupted correspondences and erroneous depth estimates, which in turn degrade reconstruction quality even in static regions.

MonST3R attempts to mitigate this issue by fine-tuning DUSt3R on dynamic video, but fundamentally still models all pointmaps with a single rigid transformation, lacking explicit cross-frame correspondence constraints for dynamic objects. Through visualization of cross-attention maps, the authors find that DUSt3R produces sharply focused attention patterns in static regions but diffuse, poorly localized attention in dynamic regions — a deficiency inherited by MonST3R. This observation directly motivates the need for explicit dynamic alignment.

Core Problem

How can the DUSt3R pointmap regression framework simultaneously capture static scene structure and dynamic object motion, such that every pixel obtains correct 3D alignment within a unified coordinate system?

Method

1. Static-Dynamic Aligned Pointmap (SDAP)

Core Idea: The scene is decomposed into static and dynamic components, each aligned to a unified coordinate system via a distinct strategy.

• Static regions: Warped using camera pose transformations, following DUSt3R's convention.
• Dynamic regions: Warped using optical flow to establish cross-frame 3D correspondences for dynamic pixels.

2. Occlusion Mask

Forward flow \(\mathbf{f}\) and backward flow \(\mathbf{b}\) are obtained from an optical flow estimator. A forward-backward consistency check is applied to compute the occlusion mask \(M_{\text{occ}}\). Occluded pixels are excluded from loss computation to prevent noisy supervision.

3. Dynamic Mask

The estimated optical flow is compared against the camera-motion-induced flow \(\mathbf{f}_{\text{cam}}\). Pixels where the discrepancy exceeds a threshold \(\tau\) are labeled as dynamic regions \(M_{\text{dyn}}\), thereby separating static and dynamic supervision signals.

4. Training Objectives

Static alignment loss \(\mathcal{L}_{\text{static}}\): The standard DUSt3R regression loss is applied exclusively to non-dynamic pixels in the second view, with confidence-aware weighting.

Dynamic alignment loss \(\mathcal{L}_{\text{dyn}}\): For dynamic, non-occluded pixels, optical flow is used to warp the second-view pointmap and compute alignment error against the first-view GT pointmap. A symmetric constraint is introduced by swapping the roles of the two views. Confidence-aware weighting is also applied.

Total loss: \(\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{static}} + \mathcal{L}_{\text{dyn}}\)

5. Additional Downstream Heads

• Dynamic Mask Head: A DPT head regresses the dynamic mask under binary cross-entropy supervision.
• Optical Flow Head: Built on the RAFT architecture, leveraging cross-attention maps (rather than conventional 4D correlation volumes) for optical flow estimation.

6. Training Details

• The encoder is frozen; only the decoder and DPT head are fine-tuned (ablation studies confirm this outperforms full fine-tuning).
• Training data: BlinkVision (Indoor/Outdoor), PointOdyssey, TartanAir, and Spring — all synthetic.
• 20,000 image pairs are randomly sampled per epoch; training runs for 50 epochs.
• AdamW optimizer, learning rate \(5 \times 10^{-5}\), on \(4 \times\) RTX 6000 GPUs.

Key Experimental Results

Multi-Frame Depth Estimation (Core Results)

Dataset	Metric	MonST3R	D2USt3R
TUM-Dynamics (All)	AbsRel↓	0.145	0.142
TUM-Dynamics (Dynamic)	AbsRel↓	0.152	0.148
Bonn (All)	AbsRel↓	0.068	0.060
Bonn (Dynamic)	AbsRel↓	0.066	0.059
Sintel (All)	AbsRel↓	0.345	0.324

Dynamic Region Pointmap Alignment (EPE↓)

Dataset	DUSt3R	MonST3R	D2USt3R	D2USt3R+Flow
Sintel-Clean	30.96	38.47	16.19	9.25
Sintel-Final	35.11	41.92	25.31	12.77
KITTI	14.19	14.91	8.91	3.57

Dynamic alignment accuracy substantially surpasses all baselines; with the flow head, the model even outperforms the dedicated optical flow method SEA-RAFT.

Robustness to Frame Interval (Bonn Dataset)

Across all intervals \(\Delta t \in \{1,3,5,7,9\}\), D2USt3R consistently outperforms MonST3R*, achieving AbsRel of approximately 0.058–0.061 versus 0.072–0.078.

Highlights & Insights

1. Elegant representation design: SDAP unifies static and dynamic alignment within the same pointmap framework, yielding a clean and efficient formulation.
2. Comprehensive loss design: Occlusion masking, dynamic masking, symmetric constraints, and confidence-aware weighting jointly ensure stable training.
3. Significant dynamic alignment gains: EPE is reduced by approximately 50% relative to DUSt3R/MonST3R, with qualitatively more accurate correspondence visualizations.
4. Diagnostic-driven methodology: Attention map visualization is used to identify the root cause of failure before designing targeted solutions — a methodology worth emulating.
5. Extensible auxiliary heads: The flow head and dynamic mask head enable the model to directly output optical flow and dynamic segmentation.

Limitations & Future Work

1. Entirely synthetic training data: All five training datasets are synthetic; generalization to real-world scenes remains to be validated.
2. Limited performance on KITTI: The absence of autonomous driving scenarios in training data leads to underperformance relative to MASt3R on KITTI.
3. Dependence on precomputed optical flow: An external flow estimator (e.g., SEA-RAFT) is required to construct training labels, increasing data preparation complexity.
4. Two-frame setting only: Extension to multi-frame global optimization (i.e., handling dynamics within DUSt3R's multi-view optimization) is not discussed.
5. Sensitivity of the dynamic mask threshold \(\tau\): The paper does not sufficiently analyze the impact of threshold selection on results.

Related Work & Insights

Method	Static Reconstruction	Dynamic Correspondence	Training Strategy	Key Distinction
DUSt3R	Excellent	None	Static data	Camera-pose alignment only
MonST3R	Good	Implicit	Fine-tuning on dynamic video	Still uses a single rigid transformation; no explicit dynamic constraints
D2USt3R	Excellent	Explicit	Separated static/dynamic supervision	SDAP + dual losses + occlusion/dynamic masks

The key distinction from MonST3R is that D2USt3R explicitly incorporates cross-frame correspondences of dynamic objects into the training objective, rather than expecting the network to implicitly learn such correspondences from data.

Broader insights: - The visualization-driven design paradigm is worth adopting: attention map analysis is used to localize the failure mode (diffuse attention in dynamic regions) before a targeted solution is devised. - The static/dynamic decomposition in SDAP generalizes naturally to other tasks requiring dynamic scene handling, such as dynamic SLAM and video depth estimation. - The flow head's use of cross-attention maps in place of 4D correlation volumes, inspired by ZeroCo, suggests that internal Transformer representations encode rich geometric information. - D2USt3R is complementary to multi-frame extensions such as Fast3R: the former addresses scene dynamics while the latter targets multi-frame efficiency; combining both could be highly valuable.

Rating

• Novelty: ★★★★☆ — The SDAP representation and separated loss design are novel, though the overall framework constitutes an incremental extension of DUSt3R.
• Experimental Thoroughness: ★★★★☆ — Comprehensive multi-task, multi-dataset evaluation with robustness analysis and ablation studies; comparisons with real-data training are lacking.
• Writing Quality: ★★★★☆ — Visualization analysis is thorough, method motivation is clear, and formulations are complete.
• Value: ★★★★☆ — Dynamic 3D reconstruction addresses a practical need; the method is applicable and yields meaningful improvements.

Related Papers

• [ICCV 2025] Dynamic Point Maps: A Versatile Representation for Dynamic 3D Reconstruction
• [NeurIPS 2025] EAG3R: Event-Augmented 3D Geometry Estimation for Dynamic and Extreme-Lighting Scenes
• [NeurIPS 2025] RGB-Only Supervised Camera Parameter Optimization in Dynamic Scenes
• [NeurIPS 2025] Styl3R: Instant 3D Stylized Reconstruction for Arbitrary Scenes and Styles
• [ICCV 2025] InstaScene: Towards Complete 3D Instance Decomposition and Reconstruction from Cluttered Scenes