Sparse auto-regressive modeling for scene generation from multi-view images¶
Conference: ECCV 2026
Paper: ECCV Official
Area: 3D Vision
Keywords: 3D scene generation, autoregressive model, 3D Gaussian Splatting, sparse voxel latent space, masked prediction
TL;DR¶
Addressing the challenge of full 3D scene completion from sparse, unconstrained views without 3D ground truth, SPAR3S proposes a sparse voxel-aligned 3D latent masked autoregressive framework combining token-wise diffusion with differentiable 3D Gaussian Splatting to achieve high-fidelity novel view synthesis and unobserved geometry extrapolation.
Background & Motivation¶
Recent advances in 3D Gaussian Splatting (3DGS) and feed-forward geometric regression have unlocked real-time photorealistic novel view synthesis from few input images. Deterministic pixel-aligned models (such as PixelSplat and DepthSplat) can efficiently lift observed pixels into 3D Gaussians. However, these methods are strictly bounded by the visual camera frustums of the input views; when confronted with wide baselines, severe occlusions, or large viewpoint changes, they inevitably fail to infer unobserved scene content. Alternative strategies that deploy 2D multi-view or video diffusion models face an intractable dilemma: generating novel views requires knowing plausible camera trajectories beforehand, yet sampling sensible camera poses requires knowing the scene layout. Furthermore, 2D generative diffusion models lack an explicit 3D substrate, suffering from severe geometric inconsistencies, drifting, and visual flickering across views.
Directly conducting generative modeling in an explicit 3D space guarantees multi-view geometric consistency by construction, but this direction has long encountered two fundamental barriers. First, dense volumetric representations (e.g., standard 3D voxel grids) suffer from cubic scaling with spatial resolution, making high-resolution scene modeling computationally prohibitive. Second, large-scale, high-fidelity 3D indoor scene ground-truth assets remain scarce compared to 2D image and video corpora, which severely impedes fully-supervised 3D generative training.
The angle of attack in this work leverages the intrinsic spatial sparsity of real 3D scenes and the differentiable projection capability of 3D Gaussian Splatting. The scene is modeled exclusively over occupied voxels in a compact 3D latent space, trained in a completely self-supervised manner using only photometric reconstruction from multi-view images. Core Idea: Construct a sparse voxel-aligned 3D latent space that only materializes occupied voxels decoded into 3D Gaussians via differentiable rendering, and train a masked autoregressive transformer equipped with token-wise diffusion to jointly infer voxel occupancy and latent features under a BFS region-growing ordering.
Method¶
Overall Architecture¶
The SPAR3S pipeline operates in two core stages. The first stage is a sparse voxel-aligned 3D latent auto-encoder (\(E_\theta, D_\phi\)): it extracts pointmaps to filter out empty voxels, aggregates 2D features into 3D voxel queries via geometry-guided bidirectional cross-attention, compresses them into a compact latent bottleneck with KL regularization, and decodes them into per-voxel 3D Gaussians supervised purely by photometric reprojection error. The second stage is a spatial masked autoregressive generative model (\(H_\psi\)): conditioned on sparse observed latents and trained against dense multi-view latents, it jointly infers voxel occupancy probabilities via a binary classification head and continuous latent token distributions via a lightweight token-wise diffusion denoising head under a topology-driven BFS ordering.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Sparse Multi-View Inputs + Pointmap Estimation"] --> B["Sparse Voxel Latent Construction<br/>Pointmap normalization and empty voxel pruning"]
B --> C["Bidirectional Cross-Attention Encoder<br/>Geometry-guided attention with projection bins"]
C --> D["Hierarchical Downsampling to Bottleneck<br/>KL divergence regularized continuous latent space"]
D --> E["Upsampling and 3D Gaussian Decoder<br/>False-positive noise injection preserves sparsity"]
E -->|Differentiable 3DGS rendering| F["Multi-View Photometric Supervision<br/>Zero 3D ground truth requirement"]
D -.->|Provides conditioning latents| G["Masked Autoregressive Diffusion Model<br/>Jointly predicts occupancy and latent features"]
G --> H["BFS Region-Growing Inference<br/>k-NN topology-driven sequential completion"]
H --> E
Key Designs¶
1. Sparse Voxel-Aligned 3D Latent Space via Photometric Supervision: Bypassing Cubic Complexity and 3D Ground Truth
Dense volumetric grids scale cubically (\(O(N^3)\)), making high-resolution 3D generative modeling intractable. SPAR3S extracts an initial geometric prior from input pointmaps (estimated via MASt3R-SfM or depth back-projection), normalizes the bounding box into \([0, 1]^3\), and instantiates 3D queries \(z_{\text{3D}}^0\) strictly at occupied voxel locations while discarding empty space entirely. To facilitate cross-modal aggregation between 2D patches and 3D voxels, bidirectional cross-attention blocks inject geometry-guided attention logits: $\(A^{\text{guided}}_{v, p} = \alpha \cdot \sigma(A^{\text{learned}}_{v, :})_p + (1 - \alpha) \cdot \sigma(A^{\text{bincount}}_{v, :})_p\)$ where \(A^{\text{bincount}}_{v, p}\) accumulates the projection count of voxel \(v\) onto patch \(p\). Crucially, no 3D mesh or SDF supervision is needed. Decoded 3D Gaussians (with \(M=63\) splats per occupied voxel) are directly projected into camera planes via differentiable rendering \(\mathcal{R}\), optimized end-to-end using photometric \(L_2\) reconstruction loss and a low-weight KL penalty.
2. Masked Autoregression with Joint Occupancy and Token-Wise Diffusion: Modeling Dynamic Topology and Continuous Latents
Scene completion over unobserved 3D regions involves unknown sequence lengths and non-trivial geometry. Conventional discrete codebooks (e.g., VQ-VAE) compromise high-frequency texture fidelity, whereas standard autoregressive regressors struggle with multi-modal continuous distributions. SPAR3S categorizes the spatial volume into occupied \(\mathcal{F}\), observed empty \(\mathcal{E}\), and unobserved \(\mathcal{U}\) sets. The generative transformer employs a dual-head design: an Occupancy Head trained via binary cross-entropy \(\mathcal{L}_{\text{occ}} = \text{BCE}(\hat{o}_{\mathcal{T}}, o_{\mathcal{T}})\) to predict physical existence, and a lightweight token-wise diffusion head \(h_\epsilon\) optimized under a DDPM noise schedule to sample continuous latent features:
$\(\mathcal{L}_{\text{diff}} = \mathbb{E}_{\epsilon \sim \mathcal{N}(0, I), t} \left\| \epsilon - h_\epsilon(z^t_{\text{mask}}, t) \right\|_2^2\)$
A secondary diffusion head with a stopgrad boundary refines the conditioning tokens, resolving observation uncertainty under minimal computational overhead.
3. BFS Region-Growing Ordering via k-NN Graph: Preserving Physical Spatial Continuity
Conventional autoregressive models impose a 1D raster-scan order, which violates 3D geometric proximity and induces severe error propagation. SPAR3S constructs a symmetric sparse \(k\)-NN graph over 3D coordinates using known conditioning voxels as seeds. Breadth-First Search (BFS) computes the shortest path distance from seeds to every candidate voxel, partitioning candidates into topological depth levels \(l(v)\). Autoregressive factorization progresses from proximal to distal shells: $\(P(\mathbf{z}) = \prod_{l=1}^{\max_v l(v)} P\left(\mathbf{z}_{\{l(v)=l\}} \mid \mathbf{z}_{\{l(v)<l\}}\right)\)$ At inference, each shell is evaluated by the occupancy head; voxels exceeding threshold \(\tau\) receive synthesized latents via diffusion sampling, while sub-threshold voxels are pruned. This outward expansion enforces spatial smoothness and structural coherence across complex layouts.
4. Hierarchical Coarse-to-Fine Upsampling with False-Positive Noise Injection: Mitigating Cascading Errors
Reconstructing fine 3D details from a compact bottleneck requires multi-scale upsampling. Relying on hard binary classification during upsampling risks permanent information loss whenever false negatives occur. SPAR3S injects false-positive noise (\(\epsilon_{\text{FP}} \approx 25\%\)) into the occupancy mask during training, training the decoder to remain resilient against classification inaccuracies. At test time, a hierarchical strategy applies a high-recall threshold at the coarse grid to maintain structural coverage, followed by a high-precision threshold at the fine grid to discard spurious artifacts.
Loss & Training¶
The auto-encoder is optimized via image photometric reconstruction and latent regularizing KL divergence: $\(\mathcal{L}_{\text{AE}} = \frac{1}{|\mathcal{I}|} \sum_{i=1}^{|\mathcal{I}|} \left\| I_i - \mathcal{R}\left(D_\phi\left(E_\theta(I_i, P_i)\right), C_i\right) \right\|_2^2 + \beta \mathcal{L}_{\text{KL}}\left(\mathcal{N}(\mu_z, \sigma_z) \parallel \mathcal{N}(0, I)\right)\)$ where \(\beta = 0.1\) and the latent channel dimension is set to 32. The generative transformer is trained with combined occupancy classification and diffusion objectives: $\(\mathcal{L}_{\text{GEN}} = \mathcal{L}_{\text{occ}} + \mathcal{L}_{\text{diff}} = \text{BCE}(\hat{o}_{\mathcal{T}}, o_{\mathcal{T}}) + \mathbb{E}_{\epsilon, t} \left\| \epsilon - h_\epsilon(z^t_{\text{target}}, t) \right\|_2^2\)$ Training randomly samples 4 conditioning views and 12 target views per scene with masking ratios uniformly chosen from \([0.5, 1.0]\). Both stages are trained from scratch on a single A100 GPU in approximately 4 days each.
Key Experimental Results¶
Main Results¶
Quantitative novel view synthesis benchmark under the challenging 2-view wide-baseline setting at resolution \(224 \times 224\) on 3DFront and RealEstate10k:
| Dataset | Category | Method | FID ↓ | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
|---|---|---|---|---|---|---|
| 3DFront | Reconstruction | 3DGS [Kerbl et al.] | 260 | 8.65 | 0.14 | 0.79 |
| 3DFront | Reconstruction | PixelSplat [Charatan et al.] | 165 | 9.07 | 0.18 | 0.67 |
| 3DFront | Reconstruction | DepthSplat [Xu et al.] | 110 | 13.76 | 0.49 | 0.55 |
| 3DFront | Generation | DiffusioNeRF [Wynn et al.] | 229 | 12.80 | 0.19 | 0.73 |
| 3DFront | Generation | MVSplat360 [Chen et al.] | 111 | 9.72 | 0.35 | 0.70 |
| 3DFront | Generation | LatentSplat [Wimbauer et al.] | 180 | 13.92 | 0.33 | 0.61 |
| 3DFront | Ours | SPAR3S (Ours) | 59 | 15.18 | 0.62 | 0.50 |
| RealEstate10k | Reconstruction | 3DGS | 271 | 7.92 | 0.12 | 0.79 |
| RealEstate10k | Reconstruction | PixelSplat | 155 | 13.73 | 0.41 | 0.50 |
| RealEstate10k | Reconstruction | DepthSplat | 82 | 13.25 | 0.46 | 0.41 |
| RealEstate10k | Generation | MVSplat360 | 66 | 15.40 | 0.49 | 0.40 |
| RealEstate10k | Generation | LatentSplat | 53 | 16.09 | 0.48 | 0.39 |
| RealEstate10k | Ours | SPAR3S (Ours) | 41 | 16.72 | 0.57 | 0.36 |
Ablation Study¶
Ablation of key generative components evaluated on 3DFront with 4 conditioning views (Tab. 2):
| Config | PSNR ↑ | SSIM ↑ | LPIPS ↓ | Note |
|---|---|---|---|---|
| SPAR3S (full model) | 15.18 | 0.62 | 0.50 | Full architecture with token diffusion, 3D RPE, and BFS ordering |
| w/o diff. | 13.90 | 0.53 | 0.54 | Replaced with linear regression head; severe drop in multi-modal texture fidelity |
| w/o 3D RPE | 13.74 | 0.49 | 0.58 | Removed 3D relative positional encoding; impairs structural spatial reasoning |
| w/o BFS ordering | 14.81 | 0.59 | 0.53 | Random autoregressive ordering; disrupts topological continuity |
| w/o occ. refinement | 14.48 | 0.56 | 0.56 | Omitted inference occupancy refinement; introduces boundary floating artifacts |
| SPAR3S w/ gt. occ. (Oracle) | 17.90 | 0.68 | 0.45 | Provided ground-truth occupancy mask; highlights potential in geometry estimation |
Key Findings¶
- Crucial Role of Token Diffusion: Replacing the token-wise diffusion head with a deterministic linear head (w/o diff.) causes a 1.28 dB drop in PSNR and degrades SSIM from 0.62 to 0.53, demonstrating that continuous stochastic diffusion is essential for capturing plausible multi-modal appearances in occluded spaces.
- Primacy of 3D Positional Encodings: Omitting 3D relative positional encoding (w/o 3D RPE) incurs the largest performance collapse (-1.44 dB PSNR, +0.08 LPIPS), establishing explicit relative spatial coordinates as vital for transformers operating on irregular 3D point distributions.
- Oracle Occupancy Upper Bound: When supplied with ground-truth occupancy masks, PSNR surges to 17.90 dB with SSIM reaching 0.68. This proves that current reconstruction limits stem primarily from binary geometric occupancy classification rather than latent appearance regression.
- Hyperparameter Dynamics: Injecting approximately 25% false-positive noise during upsampling delivers the most robust balance against test-time classification noise; allocating 63 Gaussians per voxel reaches the optimal performance plateau.
Highlights & Insights¶
- Self-Supervised 3D Latent Learning: Seamlessly bridges 2D multi-view images and explicit 3D Gaussian representations via differentiable rendering, eliminating the need for expensive 3D synthetic or scanned supervision.
- Sparsity-Driven Volumetric Scalability: Confines compute strictly to occupied physical voxels, circumventing the cubic memory explosion that typically paralyzes 3D generative networks.
- Topologically Grounded BFS Sequence Formulation: Formulates autoregressive sequence ordering via graph geodesics rather than arbitrary 1D rasterization, aligning generative dynamics with physical spatial continuity.
Limitations & Future Work¶
- Reliance on Upstream Geometry Initialization: Scene framing and initial voxel pruning rely heavily on pointmaps from MASt3R-SfM, which can falter in textureless, specular, or dimly lit scenes.
- Error Accumulation in Extended Spaces: In massive multi-room environments, iterative BFS region growing can still accumulate minor drift in the absence of explicit loop closure.
- Static Scene Assumption: The current framework assumes rigid, static scenes and cannot directly handle dynamic non-rigid motions.
Related Work & Insights¶
- vs PixelSplat / DepthSplat: Feed-forward reconstruction models excel in overlapping camera frustums but cannot extrapolate into occluded zones. SPAR3S integrates generative extrapolation, slashing 2-view FID on 3DFront from 110-165 down to 59.
- vs MVSplat360 / LatentSplat: MVSplat360 relies on 2D video diffusion iterations that suffer from view drift; LatentSplat is bounded by epipolar geometry constraints that break under non-overlapping views. SPAR3S maintains an explicit 3D sparse substrate that preserves global geometric consistency regardless of camera placement.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Elegant integration of sparse voxel latent space, masked autoregression, token diffusion, and photometric 3DGS supervision without 3D ground truth.
- Experimental Thoroughness: ⭐⭐⭐⭐☆ Comprehensive baselines and ablations across both synthetic (3DFront) and real-world (RealEstate10k) benchmarks.
- Writing Quality: ⭐⭐⭐⭐⭐ Methodological progression is structured logically, with clear mathematical formulations and informative visual figures.
- Value: ⭐⭐⭐⭐⭐ Establishes a scalable, geometrically consistent paradigm for unconstrained sparse-view 3D scene generation.