Skip to content

ReconSplat: Generalizable 3D Scene Reconstruction Beyond Observed Views

Conference: ECCV 2026
Paper: ECCV Original
Code: https://github.com/xxx
Area: 3D Vision
Keywords: 3D Gaussian Splatting, Novel View Synthesis, Feed-forward 3D Reconstruction, Multi-view Latent Diffusion Models, Scene Extrapolation

TL;DR

ReconSplat presents a feed-forward 3D scene reconstruction and view extrapolation framework that rasterizes 3D variational Gaussian distributions into 2D latent priors, guiding a multi-view latent diffusion model (MV-LDM) to jointly synthesize photorealistic novel views and geometrically consistent depth maps across unobserved camera viewpoints.

Background & Motivation

Reconstructing high-fidelity digital 3D representations of real-world scenes in a feed-forward manner from a sparse set of unposed or posed images (e.g., 2 to 4 views) is a foundational capability for spatial computing, embodied robotics, and digital twins. Recent feed-forward 3D Gaussian splatting (3DGS) regression architecturesβ€”such as pixelSplat, MVSplat, and DepthSplatβ€”have eliminated the computationally prohibitive per-scene optimization required by classical NeRF and 3DGS pipelines, delivering immediate 3D scene representations directly from multi-view image features. However, these deterministic geometric regressors are fundamentally bounded by the spatial convex hull of the input camera frustums. Once a target viewpoint involves a wide baseline or extrapolates into unobserved scene regions, the absence of projecting 3D Gaussians inevitably leads to catastrophic failures, manifested as severe black voids, fragmented surfaces, or floating artifacts.

To extend synthesis beyond narrow interpolation, emerging generative paradigms (such as latentSplat and MVSplat360) incorporate generative adversarial networks (GANs) or video diffusion models into feed-forward pipelines to inpaint missing regions. Nevertheless, existing generative approaches struggle with a longstanding trade-off between visual plausibility and physical geometric consistency. Purely generative models frequently hallucinate realistic-looking textures that violate 3D multi-view geometry, yielding noisy, discontinuous depth predictions and inconsistent parallax that collapse upon 3D point cloud unprojection. Conversely, geometry-centric regression methods like DepthSplat yield accurate depth within observed coverage but completely lack generative completion priors for unobserved camera frustums.

This paper tackles the challenge by deeply unifying deterministic 3D Gaussian geometric grounding with cross-view generative priors in a unified latent space. Core idea: construct a differentiable 3D variational Gaussian latent field with closed-form rasterize-then-sample moment matching, and feed the resulting 2D appearance and geometry latent distributions as control signals into a multi-view latent diffusion model (MV-LDM) to jointly denoise mutually aligned novel views and metric relative depth.

Method

Overall Architecture

ReconSplat adopts a two-stage feed-forward pipeline operating on \(N\) sparse calibrated RGB context views \(I_c = \{I_c^i\}_{i=1}^N\). In Stage 1, a multi-view encoder regresses a 3D Gaussian splatting representation \(\mathcal{G}\) where each primitive encodes 4-dimensional variational latent distributions for both appearance and geometry. For any target viewpoint \(P^\odot\), a closed-form alpha-blending rasterization projects the 3D distribution parameters onto the 2D latent grid, from which preliminary target latents \(\tilde{Z}^\odot\) are sampled. In Stage 2, a multi-view latent diffusion model (MV-LDM) built upon Stable Diffusion 2.1 receives these preliminary latents via ControlNet-style residual projections into its U-Net decoder. The diffusion backbone performs joint denoising of appearance and depth through cross-view 3D self-attention with relative camera positional encodings (PRoPE) and optional 3D convolutions, producing refined latents decoded into photorealistic novel views and sharp relative depth maps via lightweight LoRA-adapted VAE decoders.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Sparse Context Views<br/>RGB Images + Camera Poses"] --> B["Feed-forward 3D Latent Field<br/>MVSplat backbone predicts 3DGS geometry + variational latents"]
    B --> C["Variational Rasterization<br/>Moment matching Ξ±-blending onto 2D latent grid"]
    C --> D["Dual-Branch LoRA VAE<br/>LoRA-adapted SD-VAE for appearance & geometry decoding"]
    D --> E["Multi-View Latent Diffusion<br/>Joint appearance-depth denoising via cross-view 3D attention"]
    E --> F["Output: 3D-Consistent Novel Views & Depth<br/>Enables wide-baseline extrapolation & dense point cloud unprojection"]

Key Designs

1. Variational 3D Gaussian latent field and closed-form rasterize-then-sample blending: tackling latent manifold distortion and multi-view mixing
Prior variational 3D representations such as latentSplat adopt a sample-then-rasterize scheme, sampling latent codes per Gaussian primitive before alpha-compositing them in pixel space. However, weighted mixtures of non-linear samples distort the underlying Gaussian statistics, degrading VAE reconstruction fidelity and preventing direct latent regularization. ReconSplat inverts this paradigm into a rasterize-then-sample formulation. Each predicted 3D Gaussian primitive carries geometric attributes \((X_k, \Sigma_k, \alpha_k, c_k)\) augmented with 4D variational mean and diagonal covariance parameters for appearance \((\mu_k^a, \sigma_k^a)\) and geometry \((\mu_k^g, \sigma_k^g)\). Since ray compositing yields a weighted mixture distribution that is non-Gaussian, the paper approximates the mixture via analytical first- and second-moment matching: $$ \bar{\boldsymbol{\mu}}p = \mathbf{m}} = \frac{1}{A_p}\sum_{k}\alpha_{p,k}T_{p,k}\boldsymbol{\muk, \quad \bar{\boldsymbol{\sigma}}_p^2 = \mathbf{m}} - \mathbf{m{1,p}^2 = \frac{1}{A_p}\sum_p^2 $$ where }\alpha_{p,k}T_{p,k}\big(\boldsymbol{\sigma}_k^2 + \boldsymbol{\mu}_k^2\big) - \bar{\boldsymbol{\mu}\(A_p = \sum_k \alpha_{p,k}T_{p,k}\) denotes accumulated opacity and \(T_{p,k} = \prod_{t < k}(1 - \alpha_{p,t})\) is transmittance along ray \(p\). Rasterizing these moments to the \(H/8 \times W/8\) latent grid allows explicit KL-divergence regularization against the standard normal prior \(\mathcal{N}(0, I)\) directly in 2D latent space, aligning preliminary latents with the pre-trained SD-VAE manifold.

2. Dual-branch LoRA latent autoencoders with geometry-aware supervision: decoupling multi-view appearance and depth reconstruction
Directly supervising the 3D latent field using a frozen off-the-shelf SD-VAE encoder induces early training saturation and view inconsistencies because standard 2D VAEs lack multi-view awareness and cannot handle geometric depth. ReconSplat freezes the bulk of the SD-VAE decoder and introduces parameter-efficient Low-Rank Adaptation (LoRA) updates (~2% of decoder parameters), training two independent decoders \(D_a\) and \(D_g\) for appearance and depth respectively. Geometry decoder \(D_g\) regresses relative depth \(\hat{D}\) under a multi-term objective combining the generalized Charbonnier regression loss (\(\alpha=1\)), depth structural dissimilarity, and multi-scale gradient matching: $$ \mathcal{L}_\text{gm}(\mathbf{D}, \hat{\mathbf{D}}) = |\nabla_x \mathbf{D} - \nabla_x \hat{\mathbf{D}}| + |\nabla_y \mathbf{D} - \nabla_y \hat{\mathbf{D}}| $$ This dedicated formulation enforces sharp depth boundaries and continuous surface normals while mitigating catastrophic representation drift in the appearance stream.

3. ControlNet-style conditioning and cross-view 3D attention diffusion: enabling seamless extrapolation with spatiotemporal coherence
Because the feed-forward 3DGS field only covers surfaces observed within the context frustums, target views in extrapolation regimes contain extensive unobserved voids. To inpaint these regions while preserving observed geometric structure, ReconSplat designs an MV-LDM built on SD2.1. Rather than altering U-Net input channel dimensions, preliminary rasterized latents \(\tilde{Z}^\odot\) are projected via \(\phi\) and added directly into the U-Net middle block and decoder skip connections in a ControlNet fashion. To guarantee multi-view consistency, zero-initialized cross-view 3D self-attention layers with camera-conditioned PRoPE (Cameras as Relative Positional Encoding) are inserted into each U-Net block. For smooth continuous camera trajectories, optional camera-conditioned 3D convolution pathways (modulated via adaLN) can be activated during fine-tuning (updating only 77M parameters), effectively suppressing inter-frame flicker in video-like camera sweeps.

Loss & Training

The Stage 1 latent field and LoRA autoencoders are trained end-to-end using: $$ \mathcal{L} = \mathcal{L}a(\hat{\mathbf{I}}) + \mathcal{L}_g(\hat{\mathbf{D}}) + \omega_a \mathcal{L}\text{KL}(q_a) + \omega_g \mathcal{L}\text{KL}(q_g) + \omega\text{aux}\mathcal{L}\text{aux}(\hat{\mathbf{I}}\mathcal{G}) $$ where appearance loss \(\mathcal{L}_a\) includes \(L_1\), LPIPS, and an adaptively weighted patch GAN objective, while \(\mathcal{L}_\text{aux}\) provides the sole gradient flow for updating the structural Gaussian primitives \((X_k, \Sigma_k, \alpha_k, c_k)\).

Stage 2 trains the MV-LDM under a \(v\)-prediction formulation: $$ \mathcal{L}{\boldsymbol{v}}(\theta) := \mathbb{E}}, \mathbf{z0} \big[ |\boldsymbol{v} - \boldsymbol{v}\theta(\mathcal{Z}_t; \phi(\tilde{\mathcal{Z}}^\odot), \mathcal{P}^\odot, \mathcal{Z}_c, \mathcal{P}_c)|^2 \big] $$ using annealed multi-resolution noise schedules to simultaneously regularize global composition and high-frequency textures. Classifier-free guidance drops context latents and conditionings with probability \(p_\text{uncond}=0.1\). Pre-training is conducted for 200K steps at \(256 \times 256\) followed by 50K fine-tuning steps using \(2\times\) bilinearly upsampled inputs, effectively shifting the latent downscaling factor from \(f=8\) to \(f=4\) and bridging the resolution gap with SD-VAE.

Key Experimental Results

Main Results

Quantitative benchmarking on RealEstate10K (RE10K) under challenging wide-baseline extrapolation and DL3DV-10K across multi-round camera trajectories:

Dataset & Setting Method FID ↓ LPIPS ↓ DISTS ↓ PSNR ↑ SSIM ↑
RE10K Extrapolation pixelSplat (CVPR'24) 11.18 0.259 0.154 20.65 0.728
MVSplat (ECCV'24) 9.83 0.245 0.145 20.49 0.752
DepthSplat (CVPR'25) 11.81 0.227 0.139 20.71 0.776
latentSplat (ECCV'24) 6.93 0.234 0.127 21.60 0.732
MVSplat360 (NeurIPS'24) 5.49 0.245 0.148 20.46 0.748
ReconSplat (Ours) 4.89 0.222 0.120 21.92 0.735
DL3DV-10K Two-round (n=300) pixelSplat (CVPR'24) 132.79 0.472 0.287 17.15 0.494
MVSplat (ECCV'24) 97.73 0.476 0.235 15.59 0.358
DepthSplat* (CVPR'25) 55.84 0.297 0.155 19.41 0.633
latentSplat (ECCV'24) 73.08 0.411 0.182 16.32 0.472
MVSplat360 (NeurIPS'24) 52.86 0.461 0.191 14.17 0.332
ReconSplat (Ours) 43.87 0.351 0.137 17.58 0.508

(Note: DepthSplat was trained on the entire DL3DV-10K dataset using ~5Γ— more data; among directly comparable setups, ReconSplat demonstrates broad superiority, achieving the best FID across all methods).*

In cross-dataset zero-shot depth evaluation on ScanNet++ (12 iPhone indoor video sequences), ReconSplat achieves an AbsRel of 0.206, RMSE of 0.467, and RMSElog of 0.238, substantially outperforming MVSplat360 (AbsRel 0.410, RMSE 0.753) and DepthSplat (AbsRel 0.255, RMSE 0.492).

Ablation Study

Ablation of architectural components on RE10K under extrapolation prior to \(2\times\) input upsampling fine-tuning (ft):

Variant / Configuration PSNR ↑ SSIM ↑ LPIPS ↓ FID ↓ Note
Full Model 21.92 0.735 0.222 4.89 Complete model with 3DGS conditioning and \(2\times\) fine-tuning
w/o ft (no \(2\times\) upsampling ft) 21.26 0.704 0.241 5.84 Latent resolution mismatch causes a 0.95 FID penalty
w/o ft, G (no 3DGS latent conditioning) 20.75 0.693 0.249 6.07 Lack of 3D spatial grounding causes severe degradation across all metrics
w/o ft, D (no generative depth modeling) 21.61 0.718 0.232 5.84 Single-modality appearance optimization simplifies learning but sacrifices geometry
w/o ft, \(\epsilon_\text{mr}\) (no multi-resolution noise) 21.27 0.705 0.241 6.02 Omitting scale-annealed noise impairs global perceptual realism

Key Findings

  • Crucial role of 3DGS latent guidance: Removing the 3DGS feature conditioning (w/o G) leads to a substantial 1.17 dB drop in PSNR and degrades FID from 4.89 to 6.07, proving that 2D diffusion models must be physically anchored by explicit 3D Gaussian geometry to prevent cross-view hallucination drift.
  • Cross-modality depth co-generation benefits: Jointly modeling appearance and geometry enforces mutual physical constraints; ablating depth supervision yields misaligned multi-view projections that degrade dense point cloud unprojection.
  • Robust scalability with context view count: Evaluated on DL3DV-10K with \(N=6\) and \(N=8\) context inputs (despite being trained on \(N=4\)), ReconSplat's FID improves to 30.13 and 27.40 respectively, whereas latentSplat triggers an Out-Of-Memory (OOM) error at 8 context views due to heavy epipolar attention.

Highlights & Insights

  • Analytical moment matching for variational Gaussian rendering: Derives an exact closed-form expression for pixel-wise mean and variance under alpha-compositing, resolving latent distribution collapse and enabling standard KL regularization in latent space.
  • Non-invasive ControlNet conditioning preserves generative priors: Projects 3DGS preliminary latents into U-Net decoder skip pathways rather than modifying input channels, preserving pre-trained Stable Diffusion weights and generalizability.
  • Feed-forward RGB-to-geometry completion: Produces metric-aligned novel depth alongside photorealistic novel views, enabling feed-forward dense 3D point cloud reconstruction of unobserved areas in complex scenes.

Limitations & Future Work

  • Dependence on initial 3DGS quality: If extreme occlusions, lack of texture, or motion blur cause severe geometric outliers in Stage 1 Gaussian regression, erroneous spatial features may mislead downstream diffusion inpainting.
  • Non-native 3D latent manifold: The backbone relies on 2D SD-VAE representations, which lack strict \(SE(3)\) equivariance under 3D spatial transformations.
  • Future directions: Integrating native 3D multi-view VAE architectures and exploring large feed-forward transformer geometry backends (e.g., DUSt3R/VGGT) for joint end-to-end representation learning.
  • vs DepthSplat: DepthSplat relies on cost-volume depth regression and is restricted to narrow-baseline interpolation, breaking down in extrapolation; ReconSplat integrates MV-LDM to synthesize novel content and plausible geometry for unseen viewpoints.
  • vs latentSplat & MVSplat360: Both leverage generative priors for 3D extrapolation but lack joint appearance-geometry modeling, producing noisy, fragmented depth; ReconSplat introduces variational parameter rasterization and joint appearance-depth diffusion to produce crisp, geometrically consistent depth alongside photorealistic novel views.

Rating

  • Novelty: β­β­β­β­β˜† Closed-form moment matching for variational 3DGS rasterization combined with joint multi-view appearance-depth diffusion is technically sound and innovative.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive benchmarks on RE10K, DL3DV-10K extrapolation, ScanNet++ depth evaluation, and dense point cloud unprojections.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear mathematical derivations, well-structured pipeline diagrams, and thorough ablations.
  • Value: ⭐⭐⭐⭐⭐ Directly addresses the critical dichotomy between visual plausibility and physical geometric consistency in sparse-view 3D synthesis.