UniFusion: Sparse-View 4D Reconstruction via Unified Spatio-temporal Depth Alignment¶

Conference: ECCV 2026
Paper: ECCV Official Link
Code: https://yongzhelyu.github.io/UniFusion/
Area: 3D Vision
Keywords: 4D reconstruction, sparse views, depth alignment, spatio-temporal neural fields, Gaussian splatting

TL;DR¶

UniFusion introduces a unified spatio-temporal depth alignment framework that models depth deformation via low-rank residual neural fields, jointly resolving cross-view and cross-temporal inconsistencies without foreground segmentation or point tracking to provide robust geometric priors for 4D Gaussian splatting.

Background & Motivation¶

Reconstructing dynamic 3D scenes from multi-view video streams is essential for spatial computing, augmented reality, and robotics. While dense multi-view capture systems achieve remarkable reconstruction fidelity, they demand expensive, studio-confined camera arrays that are impractical for daily in-the-wild captures. This limitation has motivated the use of casual, sparse-view capture setups—such as the Ego-Exo4D benchmark—where merely three to four cameras record human activities. In this setting, adjacent cameras are separated by approximately 90 degrees, offering full scene coverage but minimal cross-view visual overlap. To compensate for the scarcity of multi-view constraints, modern pipelines leverage monocular depth foundation models (e.g., Depth Anything, Metric3D) to supply dense geometric priors.

However, monocular depth estimators inherently produce predictions with unknown per-view scale and shift ambiguities, and processing video frames independently induces severe inconsistencies across viewpoints and time. Crucially, in dynamic settings, genuine physical scene motion is heavily entangled with per-frame monocular prediction errors. Existing approaches, most notably MonoFusion, tackle this problem through disjoint, multi-stage pipelines: spatial alignment is performed strictly on static backgrounds, necessitating brittle foreground masks from external segmentation foundation models (e.g., SAM series); temporal alignment subsequently relies on 2D/3D point tracking models (e.g., TAP series), which suffer from substantial error accumulation and prohibitive preprocessing runtimes. Furthermore, decoupling space and time prevents temporal continuity from assisting cross-view alignment, and vice versa.

The fundamental insight of this work is that foreground segmentation and explicit point tracking can be completely bypassed by representing multi-view dynamic depth deformations within a unified spatio-temporal neural field. Core idea: model monocular depth deformation across all views and timesteps via a shared spatial representation modulated by low-rank temporal residual weights, leveraging the intrinsic smoothness bias of neural fields and a multi-view depth-order constraint to achieve unified spatio-temporal depth alignment without segmentation masks or tracking models.

Method¶

Overall Architecture¶

UniFusion operates in two main stages: Stage 1 performs joint spatio-temporal depth alignment across all views and frames, and Stage 2 utilizes the aligned, geometrically consistent depth maps to initialize and supervise 4D Gaussian splatting reconstruction backends.

In Stage 1, given $N$ sparse synchronized video streams across $T$ timesteps, monocular depths predicted by Depth Anything V2 are first coarsely aligned to sparse SfM points from MASt3R via weighted least squares to solve initial per-view affine scales and shifts. For each view $k$, UniFusion constructs a deformation MLP driven by time-independent spatial feature grids and depth feature grids. Temporal variations are modeled by parameterizing hidden layer weights through low-rank residual weight conditioning. Static regions naturally see their temporal residuals vanish, while dynamic elements are smoothly accommodated. The entire framework is optimized end-to-end using a combination of a novel multi-view depth-order loss, scale-and-shift-invariant loss, and multi-view geometric preservation losses. In Stage 2, the refined 4D depths initialize canonical Gaussian primitives and supervise dynamic Gaussian splatting (Def-2DGS, Def-3DGS, MonoFusion) during volume rendering training.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: N Views × T Frames Sparse Videos & SfM Points"] --> B["Time-Independent Encodings<br/>Spatial Grid Ek and Depth Grid zk(D)"]
    B --> C["Low-Rank Spatio-Temporal Residual Weight Network<br/>Shared Base Weights with Rank-8 VM Residuals"]
    C --> D["Joint Spatio-Temporal Geometric Optimization<br/>Multi-View Depth-Order Loss & SSI Loss"]
    D --> E["4D Gaussian Splatting Initialization & Supervision<br/>Supervising Def-2DGS / MonoFusion for Novel View/Time Synthesis"]

Key Designs¶

1. Low-Rank Spatio-Temporal Residual Weight Network: Uniform Static-Dynamic Modeling Without Segmentation or Tracking Prior methods either optimize per-frame deformation fields independently—which lacks temporal coherence and incurs prohibitive computational overhead—or inject a scalar time embedding into the MLP, which leads to severe overfitting and high-frequency distortions. UniFusion resolves this dilemma by decomposing deformation network weights. For each viewpoint $k$, the network maintains time-independent features combining a bilinearly interpolated 2D spatial grid $\mathbf{E}_k(\mathbf{u})$ and a 1D quantile-normalized depth grid $\mathbf{z}_k(D_k^t(\mathbf{u}))$: $$\mathbf{f}_k^t(\mathbf{u}) = \mathbf{E}_k(\mathbf{u}) + \mathbf{z}_k(D_k^t(\mathbf{u}))$$ To capture dynamic motions while retaining spatial compactness, the weight matrix $\mathbf{W}_k^{t,(l)}$ at layer $l$ and view $k$ is decomposed into a shared base weight $\mathbf{W}_k^{(l)}$ and a time-dependent residual $\delta\mathbf{W}_k^{t,(l)}$ using vector-matrix (VM) factorization: $$\mathbf{W}_k^{t,(l)} = \mathbf{W}_k^{(l)} + \delta\mathbf{W}_k^{t,(l)}$$ $$\delta\mathbf{W}_k^{t,(l)} = (\mathbf{c}_k^{t,(l)})^\top \mathbf{M}_k^{(l)}$$ where $\mathbf{c}_k^{t,(l)} \in \mathbb{R}^R$ represents per-frame temporal coefficients, and $\mathbf{M}_k^{(l)} \in \mathbb{R}^{R \times (F_{\text{out}} \cdot F_{\text{in}})}$ is a shared basis matrix with rank $R=8$. This formulation adds only $T \cdot R + R \cdot F_{\text{out}} \cdot F_{\text{in}}$ parameters per layer. For static background regions, the optimization naturally drives the residual to zero ($\delta\mathbf{W} \approx \mathbf{0}$), ensuring rigid cross-frame consistency. For moving foregrounds, non-zero residuals activate smoothly. This implicit separation eliminates the need for SAM-based foreground masks or TAP-based point tracking, while reducing optimization runtime from half a day down to one hour.

2. Multi-View Depth-Order Loss: Correspondence-Free Occlusion Geometry Constraints In extreme sparse-view settings with wide baselines (~90 degrees), finding accurate cross-view dense correspondence matches is often impossible in textureless or occluded regions. However, projective geometry dictates that a 3D surface point from view $k$, when unprojected and reprojected into view $j$, must not appear in front of the recorded surface of view $j$. UniFusion introduces a multi-view depth-order loss that penalizes such physical penetrations: $$\mathcal{L}_{\text{order}} = \frac{1}{|\mathcal{N}|} \sum_{(k,j) \in \mathcal{N}} \frac{1}{|\mathcal{U}|} \sum_{\mathbf{u} \in \mathcal{U}} \max\!\left(0, \frac{\hat{d}_j(\mathbf{u}') - z_{j \leftarrow k}(\mathbf{u})}{\hat{d}_j(\mathbf{u}')}\right)$$ where $z_{j \leftarrow k}(\mathbf{u})$ is the reprojected depth of view $k$'s deformed point in view $j$, and $\hat{d}_j(\mathbf{u}')$ is the surface depth read directly from view $j$'s depth map at pixel $\mathbf{u}' = \pi_j(\Pi_k^{-1}(\mathbf{u}))$. This loss acts as a self-supervised, non-local geometric barrier that resolves inter-view scale and position contradictions without relying on external appearance descriptors.

3. Scale-and-Shift-Invariant Shape Preservation: Complementary Dense Monocular Regularization While geometric reprojection losses enforce multi-view and temporal alignment, they risk distorting the fine-grained surface geometry predicted by foundation models. To preserve high-frequency details, UniFusion incorporates the scale-and-shift-invariant (SSI) loss. Each depth map is normalized by subtracting its median and dividing by its mean absolute deviation (MAD): $$\widetilde{d} = \frac{d - \text{median}(d)}{\text{MAD}(d)}, \quad \text{MAD}(d) = \text{mean}(|d - \text{median}(d)|)$$ $$\mathcal{L}_{\text{ssi}} = \|\widetilde{d}_{\text{pred}} - \widetilde{d}_{\text{ref}}\|_1$$ Together with sparse SfM alignment $\mathcal{L}_{\text{fit}}$, surface normal/gradient preservation $\mathcal{L}_{\text{struct}}$, and cross-view photometric reprojection $\mathcal{L}_{\text{match}}$, the total alignment objective guarantees both global multi-view consistency and local geometric fidelity.

4. 4D Gaussian Splatting Initialization & Supervision: Simplifying Downstream Architectures Conventional 4D reconstruction pipelines often rely on heavily engineered dynamic representations to handle under-constrained sparse views. UniFusion demonstrates that providing highly consistent geometric depth maps is far more impactful than downstream architecture complexity. The aligned depths directly initialize canonical 3D/2D Gaussian primitives and serve as explicit depth supervision targets, enabling straightforward deformable 2DGS (Def-2DGS) models to synthesize photorealistic novel views across wide extrapolation angles (30°–45°) without floaters or hollow surfaces.

Key Experimental Results¶

Main Results¶

Quantitative evaluations are conducted on Ego-Exo4D (ExoRecon subset covering dance, sports, bike repair, cooking, music, and healthcare across four ~90° cameras) and EgoHuman (fencing and legoassemble sequences with held-out cameras evaluating true novel-view synthesis).

Dataset	Method / Config	Full PSNR ↑	Full SSIM ↑	Full LPIPS ↓	AbsRel ↓	Dyn PSNR ↑	Dyn SSIM ↑
Ego-Exo4D	SOM	14.73	0.535	0.482	0.843	15.63	0.559
Ego-Exo4D	Dyn3D-GS	24.28	0.692	0.539	0.612	24.61	0.673
Ego-Exo4D	MV-SOM-DS	28.37	0.906	0.079	0.398	28.23	0.931
Ego-Exo4D	MonoFusion	30.41	0.944	0.079	0.277	29.48	0.958
Ego-Exo4D	MonoFusion + Ours	31.63	0.955	0.064	0.238	30.44	0.972
Ego-Exo4D	Def-3DGS	34.15	0.952	0.062	0.330	37.24	0.982
Ego-Exo4D	Def-2DGS	34.15	0.964	0.058	0.376	38.64	0.984
Ego-Exo4D	Def-2DGS + Ours	34.66	0.963	0.049	0.222	38.70	0.984

On the challenging EgoHuman novel-view synthesis benchmark with held-out cameras:

Dataset	Method	PSNR ↑	SSIM ↑	LPIPS ↓
EgoHuman	SOM	13.97	0.652	0.645
EgoHuman	MV-SOM	13.36	0.588	0.564
EgoHuman	SplatFields	17.74	0.675	0.571
EgoHuman	MonoFusion	14.15	0.436	0.628
EgoHuman	MonoFusion + Ours	20.41 (+6.26)	0.612 (+0.176)	0.492 (-0.136)
EgoHuman	Def-2DGS	18.88	0.657	0.431
EgoHuman	Def-2DGS + Ours	22.76 (+3.88)	0.856 (+0.199)	0.277 (-0.154)

Ablation Study¶

The progressive contributions of alignment components on Ego-Exo4D are evaluated across both MonoFusion and Def-GS backends:

Configuration	MonoFusion PSNR ↑	MonoFusion SSIM ↑	MonoFusion LPIPS ↓	Def-GS PSNR ↑	Def-GS SSIM ↑	Def-GS LPIPS ↓	Note
(a) Per-frame MAtCha	30.14	0.943	0.070	32.60	0.956	0.071	Static chart alignment per frame; slow and temporal jitter
(b) + Time Embedding	28.53	0.935	0.088	32.60	0.947	0.071	Naive MLP time conditioning causes severe degradation
(c) + Residual Weights	30.90	0.948	0.067	33.61	0.960	0.055	Largest single gain; captures smooth temporal correlations
(d) + $\mathcal{L}_{\text{order}}$	31.52	0.953	0.065	33.90	0.961	0.055	Penalizes cross-view depth penetration
(e) + $\mathcal{L}_{\text{ssi}}$ (Full)	31.63	0.955	0.064	34.66	0.963	0.049	Complete model with optimal geometric and visual fidelity

Key Findings¶

Residual weights provide essential inductive bias: Moving from independent per-frame alignment (a) to residual weights (c) boosts Def-GS PSNR from 32.60 dB to 33.61 dB, whereas injecting naive time embeddings (b) severely degrades performance (MonoFusion drops from 30.14 to 28.53 dB).
Substantial depth error reduction: UniFusion reduces Absolute Relative Error (AbsRel) by 14.1% for MonoFusion (0.277 to 0.238) and by 40.9% for Def-2DGS (0.376 to 0.222), successfully eliminating floating artifacts and surface ghosting.
Over an order-of-magnitude speedup: Aligning 100 frames across 4 views takes approximately 12 hours with per-frame MAtCha, while UniFusion completes the optimization in just 1 hour (~10× faster).

Highlights & Insights¶

Implicit dynamic-static disentanglement: Factoring neural field weights into shared base parameters and low-rank temporal residuals allows the network to naturally determine dynamic variations without explicit foreground masks (SAM) or point trackers (TAP).
Non-local depth order geometry: The depth-order loss leverages one-sided geometric inequalities to eliminate inter-camera depth conflicts without needing explicit keypoint matches.
Priors surpass downstream complexity: Equipping a vanilla deformable 2D Gaussian splatting model with aligned geometric priors consistently outperforms complex, multi-stage pipelines like MonoFusion.

Limitations & Future Work¶

Dependency on initial monocular depth and SfM: Reconstruction fidelity remains bounded by the initial monocular depth maps (Depth Anything V2) and sparse camera pose estimates from MASt3R; extreme initial topological errors cannot be recovered.
Downstream representation bottleneck under violent motion: In scenarios featuring rapid, highly non-rigid deformations, the basic Def-2DGS backend can struggle to represent complex dynamics, where sparse track guidance could still provide complementary value.
Future directions: Distilling the iterative optimization process into a feed-forward spatio-temporal foundation model for real-time 4D reconstruction, and integrating multi-modal physical priors for occlusion reasoning.

vs MonoFusion: MonoFusion isolates spatial and temporal alignment into separate stages, relying heavily on SAM segmentation and TAP tracking. UniFusion operates as a single unified neural field, improving efficiency tenfold while surpassing MonoFusion by 6.26 dB PSNR on EgoHuman.
vs MAtCha: MAtCha pioneered neural deformation fields for static multi-view depth alignment but lacks temporal modeling. UniFusion generalizes chart alignment into the 4D domain via temporal residual conditioning and depth-order regularization.
vs SplatFields: SplatFields focuses primarily on foreground object modeling under sparse views. UniFusion reconstructs complete dynamic indoor/outdoor scenes, achieving superior full-image fidelity and novel-view rendering.

Rating¶

Novelty: ⭐⭐⭐⭐☆ Elegant application of residual weight decomposition to multi-view 4D depth alignment combined with a principled depth-order loss.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive cross-dataset evaluations on Ego-Exo4D and EgoHuman with extensive ablations and multi-backend validation.
Writing Quality: ⭐⭐⭐⭐⭐ Rigorous methodology, crisp motivation, and clear figures detailing both successes and failure modes.
Value: ⭐⭐⭐⭐⭐ Sets a practical and reproducible geometric alignment standard for sparse-view dynamic 4D reconstruction.