Sparfels: Fast Reconstruction from Sparse Unposed Imagery

Conference: ICCV 2025 arXiv: 2505.02178 Code: Available Area: 3D Vision Keywords: Sparse-view reconstruction, unposed reconstruction, 2D Gaussian splatting, MASt3R, color variance regularization

TL;DR

Sparfels integrates a 3D foundation model (MASt3R) with efficient test-time optimization (2DGS). MASt3R provides an initial point cloud, camera poses, and dense correspondences to guide optimization. A novel splat color variance loss is introduced, enabling state-of-the-art geometric reconstruction from sparse unposed images in under three minutes.

Background & Motivation

Reconstructing 3D geometry from sparse, uncalibrated images is both highly challenging and practically valuable.

Limitations of classical pipelines: SfM/MVS pipelines require dense image overlap and precise calibration; 3DGS/2DGS methods depend on accurate poses and initial point clouds from COLMAP, and fail entirely under sparse, uncalibrated settings.

Opportunities and limitations of foundation models: Large models such as DUSt3R/MASt3R can infer camera poses and coarse 3D structure from a handful of images, but lack the precision and detail needed for novel view synthesis.

Shortcomings of existing sparse-view methods: Most methods rely on multiple external priors (monocular depth, surface normals, pre-trained features), resulting in complex pipelines with high computational cost.

Evaluation reliability issues: Under unposed settings, conventional camera-alignment-based Chamfer distance evaluation is unreliable.

Sparfels aims for the best of both worlds: leveraging MASt3R for strong initialization and 2DGS for efficient refinement.

Method

Overall Architecture

Given a small set of unposed color images: 1. Initialization: MASt3R produces globally aligned point clouds, initial camera poses, and dense correspondences. 2. Optimization: Initialize bundle-adjusting 2DGS → jointly optimize Gaussian parameters and camera extrinsics. 3. Output: Extract a triangular mesh from the converged 2DGS depth maps via TSDF fusion.

Key Designs

1. Scene Initialization

• Global geometric alignment: Construct an image connectivity graph; obtain pairwise point maps and correspondences via MASt3R; optimize globally consistent point maps and camera parameters.
• Surfel initialization: For each point in the global point cloud, PCA is applied to the local covariance matrix; the eigenvector corresponding to the smallest eigenvalue is taken as the normal direction, defining a local coordinate frame \([\mathbf{u}, \mathbf{v}, \mathbf{n}]\) to initialize 2D Gaussian orientations. This normal-aware initialization is critical for surface quality.

2. Correspondence Loss

Dense pixel correspondences from MASt3R are used to enforce cross-view geometric consistency:

\[\mathcal{L}_{corr} = w_{p_n,p_m} \rho(p_m - \pi(P_m^{-1}P_n\pi^{-1}(p_n, d_n)))\]

where \(d_n\) is the 2DGS splat depth and \(\rho\) is the Huber loss. This constraint guides camera optimization through depth reprojection error. Unlike InstantSplat and similar methods, it explicitly exploits MASt3R correspondences rather than relying solely on photometric loss.

3. Color Variance Regularization Loss (Core Innovation)

Analyzing splatting rendering from a statistical moments perspective: the rendered color is the expectation of color along the ray, \(C = \mathbb{E}_{t \sim p(t)}[c(t)]\). To improve robustness, the variance of color along the ray is minimized:

\[\mathcal{L}_{var} = \mathbb{V}ar_{t \sim p(t)}[c(t)] = \mathbb{E}[c(t)^2] - C^2\]

In practice, the 2DGS CUDA kernel is modified to render both color and squared color simultaneously, enabling efficient variance computation. Reducing variance lowers rendering uncertainty, yielding sharper and more multi-view consistent reconstructions.

Loss & Training

Total objective: \(\mathcal{L} = \lambda_{photo}\mathcal{L}_{photo} + \lambda_{corr}\mathcal{L}_{corr} + \lambda_{var}\mathcal{L}_{var}\)

• \(\mathcal{L}_{photo}\): standard L1 + SSIM + 2DGS geometric regularization (depth-normal consistency + depth distortion)
• \(\lambda_{photo}=1.0\), \(\lambda_{corr}=5 \times 10^{-5}\)
• \(\lambda_{var}\): cosine annealing schedule decaying from 1.0 to 0.0
• Optimization iterations: 1k for DTU, 2k–4k for others; single-stage joint optimization without multi-stage strategies
• Novel view testing: Gaussian parameters are fixed; only the test camera is optimized for 1k iterations

Key Experimental Results

Main Results

DTU 3-view reconstruction (Rel↓ / NC↑):

Method	Rel↓ (Mean)	NC↑ (Mean)
MASt3R	7.34	0.830
UFORecon	42.77	0.371
SparseCraft	6.50	—
InstantSplat2DGS	5.73	—
Sparfels	4.82	—

Sparfels achieves a mean Rel of 4.82 across 15 scenes, substantially outperforming InstantSplat2DGS (5.73) and SparseCraft (6.50).

Novel view synthesis (Tanks&Temples + MVImgNet + MipNeRF360, 3/6/12 views): Sparfels achieves or surpasses state-of-the-art performance on NVS metrics (PSNR/SSIM/LPIPS) and camera pose estimation accuracy (ATE) across multiple datasets.

Ablation Study

Loss component ablation (DTU 3-view, Rel↓):

Config	Photo	Corr	Var	Rel↓
Baseline	✓			5.73
+Correspondence loss	✓	✓		5.21
+Variance loss	✓		✓	5.35
Full	✓	✓	✓	4.82

The correspondence loss and variance regularization contribute approximately 0.5 and 0.4 Rel improvement respectively; their combination yields the best results.

Key Findings

• Extremely fast: Reconstruction completes in under 3 minutes on average (consumer GPU), far faster than NeRF-based methods (hours).
• Single model dependency: Only MASt3R is required as an external model; no additional depth or normal networks are needed.
• Correspondence-guided camera optimization is more accurate than purely photometric optimization (confirmed by ATE metrics).
• The color variance loss not only improves quantitative metrics but also qualitatively produces sharper mesh details.
• The cosine annealing schedule for variance weight is critical: strong regularization early in training stabilizes optimization, while relaxation later recovers fine details.
• Normal-aware initialization has a significant impact on surface quality.

Highlights & Insights

1. Statistical perspective on color variance loss: Variance regularization is derived from a distributionally robust optimization viewpoint; the theoretical upper bound corresponds to an L1 loss plus a variance term, providing theoretical grounding beyond empirical design.
2. Efficient grafting strategy: Rather than training a new model, the multiple capabilities of MASt3R (point cloud, cameras, correspondences) are each systematically grafted into the 2DGS optimization pipeline.
3. Improved evaluation methodology: Screen-space depth/normal evaluation is proposed for the unposed setting, avoiding unreliable camera-alignment-based metrics.
4. Efficient CUDA implementation: Variance computation is realized by modifying the splatting kernel with negligible additional overhead.

Limitations & Future Work

1. In extremely sparse settings (e.g., 2 views), MASt3R initialization quality degrades, limiting reconstruction accuracy.
2. Correspondences in texture-less or repetitively textured regions may be unreliable.
3. The current method supports only static scenes and does not handle dynamic objects.
4. The resolution and threshold parameters of the TSDF mesh extraction step require manual tuning.
5. Integration with stronger foundation models (e.g., larger DUSt3R variants) is a promising direction for future work.

Related Work & Insights

• Unlike InstantSplat, which uses 3DGS, Sparfels adopts 2DGS for improved surface consistency.
• The correspondence loss draws on the cross-view constraint idea from SPARF, but uses MASt3R rather than SfM correspondences.
• The color variance loss is conceptually analogous to depth distortion regularization in NeRF, but innovates from the perspective of rendered color.

Rating

• Novelty: ⭐⭐⭐⭐ (The color variance loss is a theoretically grounded novel design; the overall pipeline also contributes meaningful innovations.)
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ (5 datasets, 3 tasks, detailed ablations.)
• Writing Quality: ⭐⭐⭐⭐ (Method descriptions are clear; theoretical derivations are complete.)
• Value: ⭐⭐⭐⭐ (The efficiency advantage of sub-3-minute reconstruction is significant; strong practical utility.)

Related Papers

• [ICCV 2025] SpatialSplat: Efficient Semantic 3D from Sparse Unposed Images
• [ICCV 2025] RegGS: Unposed Sparse Views Gaussian Splatting with 3DGS Registration
• [ICCV 2025] TRAN-D: 2D Gaussian Splatting-based Sparse-view Transparent Object Depth Reconstruction via Physics Simulation for Scene Update
• [ICCV 2025] Baking Gaussian Splatting into Diffusion Denoiser for Fast and Scalable Single-stage Image-to-3D Generation and Reconstruction
• [ICCV 2025] CF³: Compact and Fast 3D Feature Fields