Skip to content

PointSplat: Compact Gaussian Splatting via Human-Centric Prediction

Conference: ECCV 2026
Paper: ECCV Official
Area: 3D Vision
Keywords: 3D Gaussian Splatting, Feed-Forward Reconstruction, Compact Representation, Human Novel View Synthesis, Ray Casting

TL;DR

PointSplat introduces a human-centric feed-forward 3D Gaussian Splatting framework that directly infers compact Gaussian primitives in 3D space from a space-carved, ray-cast geometric proxy, achieving superior novel view synthesis quality over view-centric baselines while utilizing only about 33% of the Gaussian primitives.

Background & Motivation

Reconstructing high-fidelity 3D human representations from sparse multi-view images on the fly is foundational for immersive live streaming, holographic telepresence, and interactive virtual reality. In bandwidth-constrained and computationally limited practical deployments, representation compactness is just as vital as visual fidelity. Traditional light-field methods rely on dense camera rigs and dedicated capture systems, resulting in prohibitive hardware costs and poor real-world scalability. Recent feed-forward neural reconstruction models such as GS-LRM, DepthSplat, and GPS-Gaussian have considerably relaxed capture requirements by inferring 3D Gaussians from sparse cameras in real time.

However, prevailing feed-forward frameworks predominantly adopt a view-centric design. In this paradigm, neural networks predict pixel-aligned depth or Gaussian feature maps independently for each input view and subsequently unproject them into 3D world space. Because different camera views observe the same foreground human body, this view-by-view unprojection inevitably leads to severe inter-view redundancy, repeatedly representing identical anatomical regions across views. As input views multiply or image resolutions scale up, Gaussian primitive counts inflate rapidly, precipitating steep transmission overhead, heavy GPU memory consumption, and disruptive floaters or noisy artifacts in novel-view renderings. Post-processing pruning strategies partially mitigate primitive bloat, yet they introduce empirical opacity thresholds that struggle to balance speed against fine surface detail.

This paper addresses the bottleneck by breaking away from view-centric unprojection and conducting feed-forward inference directly in native 3D space, focusing representation capacity exclusively on the foreground object of interest. Core idea: construct a compact surface proxy via space carving and voxelized ray casting to eliminate internal redundancy and establish explicit 2D–3D correspondences, followed by a Point-Image Transformer that directly predicts 3D Gaussian parameters in a single forward pass.

Method

Overall Architecture

Given calibrated multi-view images and corresponding foreground human silhouettes, PointSplat reconstructs a compact 3D Gaussian Splatting (3DGS) field in a single feed-forward step. The architecture operates across four coordinated stages: first, it extracts appearance tokens by combining RGB pixel values with Plücker ray embeddings and applies mask-guided top-\(n\) sampling to retain informative foreground tokens; second, it estimates a coarse geometric visual hull via multi-view space carving, voxelizes the space, and executes ray casting to prune invisible interior points while establishing explicit 2D-to-3D geometric alignments; third, a Point-Image Transformer employs alternating attention mechanisms to deeply fuse 2D appearance features with 3D spatial geometry tokens; finally, a multi-head regression decoder directly predicts 3D coordinate offsets alongside Gaussian scale, rotation, spherical harmonics, and opacity attributes, trained end-to-end under photometric and perceptual RGB supervision.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-View Images & Foreground Masks"] --> B["Appearance Token Extraction & Masked Sampling"]
    A --> C["Space Carving & Voxelized Ray Casting"]
    B --> D["Point-Image Transformer Feature Fusion"]
    C --> D
    D --> E["3DGS Attribute Regression & Position Offset Addition"]
    E --> F["Differentiable Splatting & Novel View Rendering"]

Key Designs

1. Appearance Token Extraction and Masked Sampling: Retaining Foreground Semantics while Bounding Sequence Length To equip 2D image features with explicit spatial awareness, each pixel is parameterized by its Plücker ray coordinate \(\mathbf{f}_{\mathrm{ray}} = [\mathbf{d}, \mathbf{o} \times \mathbf{d}]\), which is concatenated with pixel color \(\mathbf{c}\) to form \(\mathbf{f}_{\mathrm{pixel}} = [\mathbf{c}, \mathbf{f}_{\mathrm{ray}}]\). Partitioning high-resolution multi-view images into uniform patches would introduce quadratic attention complexity dominated by irrelevant background pixels. To address this, PointSplat computes patch saliency scores \(s_i\) by aggregating binary foreground mask values within each patch and selects the top-\(n\) informative tokens via masked sampling. These sampled tokens are mapped through a linear projection layer into embedding dimension \(C\), yielding compact appearance tokens \(\mathbf{T}_{\mathrm{appearance}} \in \mathbb{R}^{n \times C}\) that preserve fine clothing and facial textures while discarding empty background computation.

2. Voxelized Ray Casting and Geometry Encoding: Pruning Internal Redundancy and Building Explicit 2D–3D Correspondences Directly leveraging visual hulls from classical space carving yields solid geometry plagued by dense internal points that never contribute to surface rendering, which unnecessarily inflates primitive counts and dilutes surface gradients. PointSplat discretizes the spatial region into a structured voxel grid with a voxel resolution of 0.005. Rays parameterized as \(\mathbf{r}(t) = \mathbf{c} + t\mathbf{d}\) are cast from anchor camera origins through pixels, using a Digital Differential Analyzer (DDA) to rapidly traverse intersecting voxels. Within each traversed voxel, the candidate point minimizing the orthogonal point-to-ray distance is retained as a valid surface point: $\(d(\mathbf{p}_v, \mathbf{r}) = \frac{\|(\mathbf{p}_v - \mathbf{c}) \times \mathbf{d}\|}{\|\mathbf{d}\|},\quad \mathbf{p}_v^* = \arg\min_{\mathbf{p}_v \in \mathbf{v}} d(\mathbf{p}_v, \mathbf{r})\)$ Voxels not intersected by any camera ray are pruned immediately. The retained surface points \(\mathbf{p}_s^*\) are mapped through \(L\)-frequency sinusoidal positional encodings \(\gamma(\mathbf{p}_s^*)\) and concatenated with their corresponding anchor ray embeddings \(\mathbf{f}_{\mathrm{ray}}\), followed by linear projection and layer normalization to produce geometry tokens \(\mathbf{T}_{\mathrm{geometry}} \in \mathbb{R}^{M \times C}\). This mechanism strips out occluded interior geometry while forging explicit geometric correspondences between 2D pixels and 3D spatial points.

3. Point-Image Transformer and Direct 3DGS Parameter Prediction: Cross-Modal Reasoning and Primitive Synthesis Cross-modal interaction between appearance tokens \(\mathbf{T}_{\mathrm{appearance}}\) and geometry tokens \(\mathbf{T}_{\mathrm{geometry}}\) is orchestrated by a 4-block transformer utilizing alternating attention across three complementary hierarchies: global self-attention unifies broad context across all modalities; point-wise self-attention enforces local 3D structural smoothness among spatial neighbors; and image-wise self-attention coordinates frame-wise intra-view details and cross-view multi-angle consistency. The resulting Gaussian tokens are mapped through separate linear projection heads with appropriate activation constraints to regress \(K=16\) Gaussians per token. For each Gaussian primitive, the network predicts color, scale, rotation quaternion, opacity, and spatial displacement \(\mathbf{o}\): $\(\mathbf{p}_{\mathrm{final}} = \mathbf{p}_s^* + \mathbf{o}\)$ By predicting residual offsets over the ray-cast geometric proxy, the model bypasses pre-trained parametric human templates (such as SMPL) and dense intermediate depth regression, directly synthesizing well-anchored, surface-conforming 3D Gaussians.

Loss & Training

The entire pipeline is trained end-to-end using RGB supervision alone, requiring no 3D ground-truth meshes, point clouds, or depth maps. The training loss combines photometric pixel-wise fidelity with perceptual quality: $\(\mathcal{L} = \lambda_{\mathrm{L1}} \mathcal{L}_{\mathrm{L1}} + \lambda_{\mathrm{LPIPS}} \mathcal{L}_{\mathrm{LPIPS}}\)$ Balancing weights are both set to \(\lambda_{\mathrm{L1}} = 1.0\) and \(\lambda_{\mathrm{LPIPS}} = 1.0\). The model applies FlashAttention-2, gradient checkpointing, and mixed precision to optimize computational throughput. Optimization uses an initial learning rate of \(2 \times 10^{-4}\) with a 10% linear warmup, trained for 300k iterations with a total batch size of 32 on NVIDIA H20 GPUs. Inference runtime is benchmarked on an NVIDIA A6000 GPU (48GB).

Key Experimental Results

Main Results

On real-world dynamic human capture benchmarks DNA-Rendering (16 evaluation sequences) and ActorsHQ (12 evaluation sequences), evaluated under eight \(512 \times 512\) input views using pure PyTorch implementations:

Dataset Method PSNR ↑ SSIM ↑ LPIPS ↓ GS Count (k) Inference Time (s)
DNA-Rendering [5] GS-LRM [51] 18.25 0.582 0.364 204k 0.32
DNA-Rendering [5] AnySplat [14] 20.97 0.790 0.143 86k 0.53
DNA-Rendering [5] GPS-Gaussian [55] 22.35 0.797 0.172 51k × N 0.07 × N
DNA-Rendering [5] DepthSplat [43] 23.98 0.821 0.131 212k 0.45
DNA-Rendering [5] LVSM [15] 23.24 0.802 0.135 - 1.0 × N
DNA-Rendering [5] Ours (PointSplat) 27.18 0.891 0.071 71k 0.40
ActorsHQ [13] GS-LRM [51] 16.96 0.675 0.299 204k 0.32
ActorsHQ [13] AnySplat [14] 19.11 0.699 0.238 86k 0.53
ActorsHQ [13] GPS-Gaussian [55] 23.81 0.887 0.084 51k × N 0.07 × N
ActorsHQ [13] DepthSplat [43] 22.39 0.797 0.157 212k 0.45
ActorsHQ [13] LVSM [15] 24.00 0.804 0.128 - 1.0 × N
ActorsHQ [13] Ours (PointSplat) 27.62 0.887 0.084 71k 0.40

On synthetic datasets THuman2.0 and RenderPeople (Table 2 protocol) under 8 input views, PointSplat attains 34.30 dB PSNR / 0.015 LPIPS on THuman2.0 (outperforming DepthSplat's 26.32 dB / 0.048 and GPS-Gaussian's 31.08 dB / 0.032) while generating only 75k Gaussians compared to DepthSplat's 270k.

Under zero-shot evaluation at \(1024 \times 1024\) resolution on DNA-Rendering (Table 3), PointSplat achieves 26.54 dB PSNR / 0.123 LPIPS, markedly outperforming feed-forward baselines GPS-GS (20.63 dB) and DepthSplat (19.82 dB) as well as per-scene optimization methods GauHuman (17.60 dB) and LongVolCap (24.21 dB).

Ablation Study

Ablations on key model components, geometry proxy alternatives, and voxel resolutions on DNA-Rendering:

Configuration PSNR ↑ SSIM ↑ LPIPS ↓ GS Count (k) Time (s) Description (Tables 5 & 6)
Full Model (AA + Ray Casting + Voxel Encode) 27.18 0.891 0.071 71k 0.4 Complete framework
w/o Voxel Encode 26.60 0.876 0.091 160k 0.7 Point positional encoding removed; coherence degrades
w/o Ray Casting 26.32 0.856 0.093 224k 0.7 Internal points kept; Gaussian count inflates 3.1×
Full Attention (replacing Alternating Attention) 27.09 0.887 0.076 71k 1.1 Significant computational overhead with slightly lower PSNR

Geometry proxy type comparison (Table 7): Visual Hull proxy achieves 27.18 dB PSNR, clearly superior to VGGT Depth (25.95 dB), Bounding Box (23.89 dB), and Frustum Sample (22.92 dB).

Key Findings

  • Ray casting is essential for compactness: Removing the ray casting module causes the Gaussian primitive count to surge from 71k to 224k (+215%), raises inference latency from 0.4s to 0.7s, and degrades LPIPS from 0.071 to 0.093. This confirms that hidden interior points cause severe redundancy and introduce blending noise during rendering.
  • Superior robustness across input view counts: When evaluated under an ultra-sparse 4-camera setup (Table 4), view-centric baselines suffer severe degradation (GPS-GS drops to 13.71 dB PSNR; DepthSplat to 19.55 dB), whereas PointSplat reliably preserves visual quality at 23.16 dB PSNR / 0.112 LPIPS. At 16 cameras, PointSplat scales smoothly to 27.31 dB, confirming the resilience of native 3D predictions over multi-view 2D unprojections.

Highlights & Insights

  • Native 3D feed-forward prediction over view-centric unprojection: By formulating Gaussian regression directly over a 3D geometric surface rather than merging independent 2D pixel-aligned Gaussian maps, PointSplat eliminates inter-view redundancy at the architectural level.
  • Zero-parameter DDA ray-casting for clean 2D–3D anchoring: Using DDA grid traversal to locate the closest surface points to view rays prunes internal visual hull points with zero learned parameters while naturally grounding 3D primitives onto 2D camera rays.
  • Template-free human reconstruction: PointSplat eschews rigid SMPL/SMPL-X parametric meshes and pre-trained human priors, preserving high rendering fidelity on loose clothing, handheld items, and complex multi-person scenes (e.g., PKU-DyMVHumans).

Limitations & Future Work

  • Reliance on accurate silhouettes and camera calibration: The geometric proxy construction depends on multi-view space carving; corrupted foreground masks or inaccurate camera extrinsics will lead to truncated visual hulls and downstream reconstruction defects.
  • Constrained to bounded foreground targets: The space carving and voxel grid formulations are designed for object- or human-centric foregrounds and cannot be readily applied to unbounded expansive outdoor scenes.
  • Lack of temporal dynamic modeling: The model processes multi-view video frames independently; introducing temporal consistency constraints, dynamic deformation fields, or compact 4D Gaussian representations represents a natural next step.
  • vs DepthSplat / GPS-Gaussian: DepthSplat and GPS-Gaussian predict per-view pixel-aligned Gaussians, creating 200k to 500k+ primitives across multiple viewpoints and introducing heavy inter-view floaters; PointSplat performs predictions directly in 3D space, reaching sharper novel views with only 71k Gaussians.
  • vs LHM / GauHuman / Diffuman4D: These methods either strictly require SMPL parameterization or rely on minutes-long per-scene test-time optimization; PointSplat operates strictly feed-forward without parametric mesh constraints, handling diverse garments effortlessly.
  • vs AnySplat: AnySplat explores feed-forward voxel aggregation for unposed arbitrary scenes but relies on specialized geometric supervision; PointSplat demonstrates that under calibrated views, space carving coupled with ray casting and pure RGB supervision delivers state-of-the-art human view synthesis.

Rating

  • Novelty: ⭐⭐⭐⭐☆ Moves past standard view-centric 2D unprojection by integrating space carving, DDA ray casting, and an alternating-attention Point-Image Transformer for native 3D Gaussian prediction.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive benchmarks across DNA-Rendering, ActorsHQ, THuman2.0, RenderPeople, and DyMVHumans, complete with 1024 high-resolution zero-shot tests and rigorous component ablations.
  • Writing Quality: ⭐⭐⭐⭐⭐ Well-structured narrative with crisp architectural formulations and precise empirical reporting.
  • Value: ⭐⭐⭐⭐⭐ Provides an efficient, ultra-compact 3D representation pipeline ideally suited for bandwidth-limited real-time holographic communication and immersive human broadcasting.