VolSplat: Rethinking Feed-Forward 3D Gaussian Splatting with Voxel-Aligned Prediction¶

Conference: ECCV 2026
arXiv: 2509.19297
Code: To be open-sourced
Area: 3D Vision
Keywords: 3D Gaussian Splatting, feed-forward reconstruction, voxel alignment, sparse-view novel view synthesis, voxel feature grid

TL;DR¶

VolSplat shifts the primitive prediction paradigm of feed-forward 3D Gaussian Splatting from pixel-aligned to voxel-aligned: it first projects multi-view 2D features back to 3D space to form a voxel feature grid through depth maps, refines them using residual learning via a sparse 3D U-Net, and then predicts Gaussian parameters per voxel, thus fundamentally decoupling the 3D representation from the rigid binding of the input image resolution to achieve adaptive Gaussian density allocation.

Background & Motivation¶

Feed-Forward 3D Gaussian Splatting (Feed-Forward 3DGS) has emerged as one of the dominant paradigms in novel view synthesis. Unlike traditional NeRF or 3DGS that optimize scene-by-scene, feed-forward methods directly predict 3D Gaussian parameters from sparse multi-view images through a single forward network pass, enabling rapid reconstruction of new scenes during inference. Among these, pixel-aligned methods represented by pixelSplat, MVSplat, and DepthSplat dominate: their core idea is to extract pixel-wise features from the 2D feature maps of each view and project them back into 3D space, where each pixel corresponds to a 3D Gaussian. This paradigm is simple and intuitive, leveraging mature 2D feature extractors and cross-view matching mechanisms to achieve outstanding performance.

However, pixel-aligned methods suffer from two deep-rooted limitations. First, the positions of Gaussians are determined by depth map back-projection, and the discrete sampling of the 2D image grid makes the correspondence between different views in 3D space heavily dependent on 2D feature matching—which essentially attempts to solve "2D-to-3D correspondence matching", making it highly prone to inconsistent multi-view alignment caused by camera calibration errors or inaccurate depth estimation, leading to a large number of floaters. Second, every input pixel is fixed to generate one Gaussian, rigidly locking the total number of Gaussian primitives to \(H \times W \times N\) (resolution \(\times\) number of views), which cannot adaptively adjust based on scene complexity: flat walls are allocated just as many Gaussians (redundancy), while fine geometric structures suffer from an insufficient allocation. Both issues fundamentally stem from the structural contradiction where "3D representation is hijacked by the 2D pixel grid".

The core breakthrough of this paper lies in: instead of struggling with cross-view matching in 2D space before mapping to 3D, it is better to directly address the problem in 3D space. Core Idea: Shift the primitive alignment paradigm of feed-forward 3DGS from pixel-level to voxel-level—first aggregate multi-view 2D features into a unified 3D voxel grid through depth map back-projection, perform 3D spatial context refinement using a sparse 3D U-Net, and then directly regress Gaussian parameters per voxel. This ensures that the Gaussian distribution is determined by the 3D structure of the scene itself rather than the input image resolution.

Method¶

Overall Architecture¶

The pipeline of VolSplat can be summarized into four stages: 2D feature extraction and depth prediction \(\rightarrow\) 3D voxel feature construction \(\rightarrow\) sparse 3D feature refinement \(\rightarrow\) voxel-wise Gaussian prediction and rendering. Different from pixel-wise methods that directly predict per-pixel Gaussians on 2D feature maps before back-projection, VolSplat first aggregates features into a 3D voxel grid, completes 3D context reasoning within the voxel space, and then predicts Gaussians per voxel, completely decoupling the Gaussian distribution from the 2D pixel grid.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-view Images + Poses"] --> B["2D Feature Extraction<br/>ResNet + Cross-view Attention"]
    B --> C["Cost Volume Construction<br/>Plane Sweeping + Depth Prediction"]
    C --> D["Feature Back-projection + Voxelization<br/>Depth Back-projection → Average Pooling Aggregation"]
    D --> E["Sparse 3D U-Net<br/>Voxel Feature Residual Refinement"]
    E --> F["Per-voxel Gaussian Prediction<br/>Center Offset + Opacity + Covariance + SH"]
    F --> G["Differentiable Rendering<br/>MSE + LPIPS Loss"]

Key Designs¶

1. Voxel-aligned 3D Feature Construction: Escaping Pixel Grids, Unified 3D Space Fusion

Pixel-wise methods independently predict Gaussians on each view's 2D feature map, and cross-view 3D consistency relies entirely on 2D feature matching—where corresponding points across different views are affected by projection distortion and discretization errors in 2D space, limiting matching accuracy. VolSplat's approach is to: first back-project 2D features into 3D point clouds in the world coordinate system using the predicted depth maps from each view, and then voxelize these scattered points (by dividing by voxel size \(v_s\) followed by rounding to obtain indices). Feature average pooling is performed for all points falling into the same voxel to obtain a sparse 3D voxel feature grid \(V\). This operation achieves three things simultaneously—fusing multi-view information into a unified 3D container, naturally resolving cross-view alignment ambiguities (since features from all views collide in the same 3D space rather than matching across multiple 2D planes), and converting data from a dense 2D grid to a sparse 3D representation (only voxels occupied by point clouds store features). The voxel size \(v_s\) is a key hyper-parameter: too small, spatial context is lost and computational overhead is high; too large, geometric quantization becomes coarse. The optimal compromise is \(v_s = 0.1\) (corresponding to a PSNR of 29.40 with 604K Gaussians per scene).

2. Sparse 3D U-Net Residual Refinement: Imposing 3D Geometric Priors in Voxel Space

Directly predicting Gaussians from the initial voxel features obtained via back-projection is insufficient, as the initial features are only aggregated from single points and lack 3D neighborhood context. VolSplat introduces a sparse convolutional 3D U-Net (based on MinkowskiEngine) to refine voxel features in a residual manner: the network predicts a residual field \(R\), and the refined feature \(V' = V + R\). The advantage of the residual design is that the network only needs to learn the correction terms (fine-tuning geometric details and consistency cues) instead of reconstructing the entire feature content from scratch, which stabilizes training and preserves the coarse-grained information of the initial voxel features. The encoder-decoder structure of the U-Net achieves multi-scale fusion via skip-connections, enabling voxel features to simultaneously perceive local details (such as edges, textures) and global geometric context (such as plane extension directions). Ablation studies show that removing the refinement module drops the PSNR from 29.40 to 27.47 (-1.93dB), and replacing it with a plain 3D CNN or removing the residual design yields significant drops of 1.39dB and 1.48dB, respectively, validating the necessity of each design choice.

3. Adaptive Per-Voxel Gaussian Prediction: Letting Scene Complexity Determine Gaussian Density

A core issue of pixel-wise methods is that the Gaussian density is locked by the input resolution. VolSplat's innovation is: each occupied voxel independently predicts one Gaussian (rather than one per pixel). The number of predicted Gaussians depends solely on the number of voxels that actually generate 3D points in the scene. Specifically, for each occupied voxel, the network outputs a 38-dimensional parameter vector containing the center offset \(\bar{\mu}\), opacity \(\bar{\alpha}\), covariance \(\Sigma\), and spherical harmonics color coefficients \(c\). The center offset is constrained by sigmoid and mapped to the local range of the voxel: \(\mu = r \cdot (\sigma(\bar{\mu}) - 0.5) + \text{Center}\), where \(r\) is \(3\) times the voxel size, allowing the Gaussian center to move symmetrically around the voxel. The direct consequence of this design is: in simple areas like flat walls, there are fewer voxels and therefore fewer Gaussians; in complex geometric structures (such as the edges of a faucet), voxels are denser, naturally yielding more Gaussians. The visual comparison in Figure 7 clearly demonstrates this adaptive allocation—DepthSplat's Gaussian distribution is a uniform rectangular array, whereas VolSplat's Gaussians naturally cluster on object boundaries. Since the number of voxels is typically much smaller than \(H \times W \times N\), VolSplat actually saves Gaussian primitives in practice (604K vs. millions), while maintaining an inference VRAM of only 4.65GB, on par with the lightweight baseline MVSplat.

Loss & Training¶

The training loss is a combination of MSE and LPIPS:

\[ \mathcal{L} = \sum_{m=1}^{M} \left( \mathcal{L}_{\mathrm{MSE}}(I_{\mathrm{render}}^{(m)}, I_{\mathrm{gt}}^{(m)}) + 0.05 \cdot \mathcal{L}_{\mathrm{LPIPS}}(I_{\mathrm{render}}^{(m)}, I_{\mathrm{gt}}^{(m)}) \right) \]

Following the training protocol of DepthSplat: 6 input views + 8 target views per sample, AdamW optimizer, cosine learning rate decay, using a lower learning rate of \(2\times 10^{-6}\) for the Depth Anything V2 backbone, and \(2\times 10^{-4}\) for the other layers. After training for 150K iterations on RealEstate10K, it is fine-tuned on ScanNet and evaluated on ACID zero-shot.

Key Experimental Results¶

Main Results¶

RealEstate10K (Indoor & Outdoor Real Estate) — Different Number of Input Views

Input Views	Metric	VolSplat	DepthSplat (SOTA Pixel-Aligned)	Gain
6 Views	PSNR ↑	31.30	30.52	+0.78
6 Views	SSIM ↑	0.941	0.931	+0.010
6 Views	LPIPS ↓	0.075	0.079	-0.004
12 Views	PSNR ↑	29.40	28.54	+0.86
24 Views	PSNR ↑	27.21	26.26	+0.95

ScanNet (Indoor Scenes) — 6 Views

Method	PSNR ↑	SSIM ↑	LPIPS ↓
FreeSplat	27.45	0.829	0.222
WorldMirror	25.83	0.819	0.136
VolSplat	28.41	0.906	0.127

ACID (Outdoor Natural Scenes) — Zero-Shot Cross-Dataset Generalization

Method	PSNR ↑	SSIM ↑	LPIPS ↓
MVSplat	28.15	0.841	0.147
DepthSplat	28.37	0.847	0.141
VolSplat	32.65	0.932	0.092

Ablation Study¶

Configuration	PSNR	SSIM	LPIPS	Description
Full model	29.40	0.928	0.085	Complete VolSplat
w/o decoder	27.47	0.901	0.102	Removing 3D U-Net refinement, performance drops significantly
w/ 3D CNN (non-U-Net)	28.01	0.919	0.098	U-Net outperforms general 3D CNN by 1.39dB
w/o residual	27.92	0.908	0.101	Residual design contributes 1.48dB

Key Findings¶

The biggest advantage of voxel alignment is zero-shot generalization: On ACID, VolSplat outperforms all baselines significantly with 32.65 PSNR (the second place DepthSplat only achieves 28.37), indicating that voxel-space feature fusion naturally possesses superior domain transfer capability, unlike pixel-aligned methods which are highly sensitive to the training data's depth distribution.
Adaptive Gaussian allocation shows significant effectiveness: Both visualization and density statistics confirm that VolSplat's Gaussian distribution varies with the scene's geometric complexity—sparse in simple regions and dense at complex boundaries, while pixel-aligned methods maintain a uniform rectangular array.
Surprisingly efficient: Despite introducing 3D feature processing, VolSplat's inference time (0.575s) and GPU memory usage (4.65GB) are comparable to MVSplat (0.369s / 4.70GB) and outperform DepthSplat (0.513s / 8.00GB), demonstrating that sparse data structures successfully control computational overhead.
The more input views, the more pronounced the voxel-alignment advantage: From 6 views to 24 views, the PSNR gain of VolSplat over the best baseline increases from +0.78 to +0.95dB, showing that the 3D feature grid can more effectively exploit multi-view redundancy under high-density inputs.

Highlights & Insights¶

The paradigm shift "from 2D to 3D" is the core insight: The consensus in feed-forward 3DGS has always been "cross-view matching on 2D feature maps followed by back-projection". This work experimentally proves that performing feature aggregation and inference directly in 3D voxel space avoids numerous alignment errors. This concept is simple yet disrupts the default assumptions of this field.
Clever design of residual refinement: Instead of directly predicting refined features, predicting residuals combined with U-Net's multi-scale fusion substantially improves reconstruction quality while maintaining training stability—an advantage unique to voxel space, making it difficult to design a similar 3D context-reasoning mechanism within a 2D pixel grid.
Efficient implementation of sparse voxels: Leveraging MinkowskiEngine to implement sparse 3D convolutions keeps the model's actual inference GPU memory and speed comparable to pure 2D methods, proving that "introducing 3D processing does not helical increase overhead". This opens the path for incorporating more complex 3D geometric networks into feed-forward reconstruction.
A massive leap in zero-shot generalization: The PSNR gap of 32.65 vs. 28.37 on ACID is a strong signal—demonstrating that voxel features possess far stronger domain invariance than pixel features, which is a decisive advantage for cross-domain deployment in real-world applications.

Limitations & Future Work¶

Static Scene Assumption: The current model assumes that the scene remains completely static during acquisition. Cost Volume and the 3D U-Net rely on multi-view geometric consistency; dynamic objects (e.g., pedestrians, swaying leaves) will produce ghosting or blurring. Extending VolSplat to dynamic or deformable scenes is a natural next step.
Reliance on Depth Estimation Quality: Although voxel alignment itself mitigates floaters caused by inaccurate depth, the construction of initial voxel features still depends on predicted depth maps. If depth estimation fails completely under extreme conditions (e.g., textureless regions, specular reflections), the voxel grid itself will exhibit holes.
Scaling to Large-Scale Scenes: Current experiments are mainly conducted on indoor or object-centric indoor/outdoor scenes (\(256 \times 256\) resolution). When extending to large-scale outdoor scenes (such as street views, aerial imaging), the size of the voxel grid will expand rapidly, requiring further optimization in sparsification strategies.

vs. pixelSplat / MVSplat / DepthSplat: These methods are all pixel-aligned—predicting Gaussians per pixel on 2D feature maps followed by back-projection. VolSplat does the opposite, first aggregating features into a 3D voxel grid and then predicting Gaussians. The difference lies in VolSplat performing 3D reasoning "within voxel space" rather than matching "across 2D pixels".
vs. AnySplat / WorldMirror: These methods add post-processing (graph network modeling among Gaussians, cross-view Gaussian fusion) on top of pixel alignment, but they still operate under the premise of pixel-wise Gaussians. VolSplat fundamentally changes the primitive generation paradigm, thereby eliminating the need for these post-processing steps and resulting in a cleaner, unified framework.
vs. EVolSplat: EVolSplat uses voxel features in autonomous driving scenarios, but is limited to city street views and requires an explicit point cloud as an intermediate representation. VolSplat removes the reliance on point cloud priors and extends voxel alignment to general reconstruction scenarios.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Shifting the core paradigm of feed-forward 3DGS from pixel alignment to voxel alignment is simple yet highly disruptive, exploring a long-ignored design space choice in this field.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Covers three major benchmarks (RealEstate10K / ScanNet / ACID), multiple input view configurations (6/12/24), cross-domain zero-shot evaluation, and comprehensive ablations covering components, voxel sizes, and efficiency comparisons.
Writing Quality: ⭐⭐⭐⭐⭐ Clear motivation, in-depth analysis (quantifying the two key limitations of pixel alignment), well-structured method descriptions, and well-aligned figures/tables.
Value: ⭐⭐⭐⭐⭐ Feed-forward 3DGS is currently a highly active research area. VolSplat provides a simple yet effective alternative paradigm with distinct zero-shot generalization benefits, offering high utility and research transfer value.