Skip to content

SpatialSplat: Efficient Semantic 3D from Sparse Unposed Images

Conference: ICCV 2025
arXiv: 2505.23044
Code: GitHub
Area: 3D Vision
Keywords: Semantic 3DGS, Feed-forward Reconstruction, Unposed Images, Dual-field Architecture, Gaussian Selection

TL;DR

SpatialSplat is proposed to generate compact semantic 3D Gaussians from sparse unposed images via feed-forward inference, leveraging a dual-field semantic representation and a selective Gaussian mechanism that reduces representation parameters by 60% while surpassing state-of-the-art methods.

Background & Motivation

Semantics-aware 3D reconstruction, which recovers semantic 3D structure from 2D images, is a foundational technology for robotics, autonomous driving, and VR/AR. Existing feed-forward 3DGS methods face two core challenges when incorporating semantics:

Redundancy in per-pixel Gaussian prediction โ€” overlapping regions produce a large number of redundant primitives, incurring unnecessary memory overhead.

Compression loss of high-dimensional semantic features โ€” 512-dimensional language features must be compressed to 64โ€“128 dimensions to be attached to each primitive, leading to irreversible information loss.

Existing methods (e.g., LSM) naively append compressed features to every pixel-level Gaussian, which is neither efficient nor accurate.

Key Findings

  1. Redundant primitives share similar geometry and appearance and can be identified directly from image features (without geometric priors).

Per-primitive semantics are unnecessary โ€” Gaussians within the same instance exhibit high semantic consistency; coarse-grained semantics combined with fine-grained instance information are sufficient.

Method

Dual-field Semantic Representation

The dense semantic feature field is decomposed into two components:

Fine-grained instance-aware radiance field \(\mathcal{F}_I\): - Each Gaussian carries a low-dimensional instance feature \(\boldsymbol{f}_I \in \mathbb{R}^N\) and an importance score \(\boldsymbol{\beta}\) - Guided by 2D foundation models (e.g., SAM)

Coarse-grained semantic feature field \(\mathcal{F}_S\): - Predicted at a resolution downsampled by a factor of \(S\), substantially reducing the number of primitives - Retains uncompressed semantic features \(\boldsymbol{f}_S \in \mathbb{R}^M\) - A small number of primitives suffices to encode complete semantics (due to intra-instance semantic consistency)

Selective Gaussian Mechanism (SGM)

An importance score \(\beta_i\) is predicted for each primitive and multiplied into the opacity to modify alpha blending:

\[\boldsymbol{c} = \sum_{i=1}^n \boldsymbol{c}_i \boldsymbol{\alpha}_i \boldsymbol{\beta}_i \prod_{j=1}^{i-1}(1 - \boldsymbol{\alpha}_j \boldsymbol{\beta}_j)\]

A Leaky ReLU-style thresholding is applied: $\(\beta_i = \begin{cases} \beta_i & \text{if } \beta_i > \tau \\ \beta_i \times 10^{-3} & \text{if } \beta_i < \tau \end{cases}\)$

A BCE loss combined with L1 regularization drives \(\beta_i\) toward binary values of 0 or 1: $\(\mathcal{L}_I = \mathcal{L}_{BCE}(\boldsymbol{S}, \hat{\boldsymbol{S}}) + \frac{1}{\|\boldsymbol{S}\|}\sum_{\beta_i \in \boldsymbol{S}} \beta_i\)$

3D Geometry Prediction

A pure ViT encoder-decoder is adopted without geometric priors. Scale ambiguity is resolved by injecting camera intrinsics (without depth supervision).

Key Experimental Results

ScanNet Semantic 3D Reconstruction

Method Feed-forward Source mIoUโ†‘ Target mIoUโ†‘ PSNRโ†‘ SSIMโ†‘ LPIPSโ†“
L-Seg โœ— 0.5541 0.5558 N/A N/A N/A
NeRF-DFF โœ— 0.5381 0.5137 22.49 0.765 0.283
Feature-3DGS โœ— 0.4992 0.3223 17.96 0.581 0.489
NoPoSplat โœ” N/A N/A 25.70 0.816 0.188
LSM โœ” 0.5141 0.5104 24.12 0.796 0.253
SpatialSplat-Lite โœ” 0.5272 0.5265 25.45 0.803 0.204
SpatialSplat โœ” 0.5593 0.5587 25.46 0.805 0.205

Parameter Efficiency

SpatialSplat uses only 40% of the representation parameters of the baseline while outperforming it across all metrics.

Key Findings

  1. The dual-field architecture achieves superior semantic segmentation and rendering quality with only 40% of the parameters.
  2. The selective Gaussian mechanism effectively identifies and removes redundant primitives without requiring geometric priors.
  3. Coarse-grained uncompressed semantics outperform fine-grained compressed semantics, validating the "uncompressed but sparse" strategy over "compressed but dense."
  4. This constitutes the first feed-forward 3DGS framework that jointly learns semantic and instance priors.

Highlights & Insights

  1. Decoupled semantic representation โ€” The decomposition into coarse semantics and fine-grained instance features is both novel and efficient.
  2. Counterintuitive choice of no compression โ€” Retaining full semantic features on a small number of primitives proves more effective than compressing and distributing them across all primitives.
  3. Redundancy identification from images โ€” This bypasses the need for precise camera extrinsics to detect overlapping regions.
  4. No 3D supervision โ€” Learning is entirely guided by 2D foundation models.

Limitations & Future Work

  • The downsampling rate \(S\) for the coarse semantic field must be set in advance.
  • Segmentation accuracy at instance boundaries is constrained by the quality of the 2D foundation model.
  • The importance score threshold \(\tau\) requires manual tuning.
  • Feed-forward 3DGS: pixelSplat, MVSplat, NoPoSplat
  • Feature field distillation: LERF, LangSplat, Feature-3DGS, LSM
  • Compact 3DGS: Scaffold-GS, HAC, LightGaussian

Rating

  • Novelty: โญโญโญโญ (dual-field architecture + selective Gaussians)
  • Technical Depth: โญโญโญโญ (complete SGM design + loss formulation)
  • Experimental Thoroughness: โญโญโญโญ (comprehensive comparison with diverse baselines)
  • Practical Value: โญโญโญโญโญ (60% parameter reduction with high deployment utility)