Skip to content

InstGS: Shared-Template Gaussian Instancing for Object-Redundancy-Free Rendering

Conference: ECCV 2026
Paper: CVF Open Access
Code: https://github.com/Strange-tech/InstGS
Area: 3D Vision
Keywords: 3D Gaussian Splatting, instanced rendering, shared template, memory compression, real-time rendering

TL;DR

InstGS introduces GPU-accelerated instanced rendering to 3D Gaussian Splatting by extracting a shared geometric template via gradient-driven instance segmentation and applying learnable lightweight offsets in position, base color, and opacity, achieving massive memory savings and higher frame rates with negligible fidelity loss in repetitive scenes.

Background & Motivation

Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) represent two dominant paradigms for novel view synthesis, achieving remarkable photorealism across diverse multi-view reconstruction benchmarks. Among them, 3DGS leverages explicit 3D Gaussian primitives and a tile-based CUDA rasterizer, bypassing the expensive point-sampling neural evaluations required by NeRF and enabling true real-time rendering. However, when 3DGS is deployed to large-scale or structured environments featuring massive numbers of repetitive objects—such as urban streetscapes with identical buildings, industrial campuses with vehicles, or indoor conference rooms populated by hundreds of identical chairs—every single Gaussian primitive is treated and parameterized as a completely independent entity. Consequently, the explosion in total primitive count imposes severe GPU memory footprints and saturates PCIe memory bandwidth.

Existing acceleration and compression techniques primarily concentrate on pruning individual primitives, scalar/vector quantization, hash-grid coding, or constructing level-of-detail (LOD) octrees. While these approaches compress the storage footprint of individual objects, they fundamentally treat all spatial regions independently and ignore high-level structural redundancy. Moreover, because most compression techniques require explicit decompression into standard Gaussian primitives prior to rendering, the runtime memory consumption and rendering pipeline bottlenecks remain largely unaddressed. Meanwhile, recent modular or instance-level 3DGS pipelines (e.g., Splat-and-Replace or Proc-GS) attempt scene assembly via naive duplication of identical Gaussian clusters. However, they rely heavily on CPU-side duplication, cannot accommodate fine-grained inter-instance discrepancies in texture, geometry, or lighting, and lack low-level GPU rasterization support.

To resolve this fundamental tension, InstGS draws inspiration from classic computer graphics instancing: repetitive objects in the physical world share a common underlying geometry, differing only in slight local deformations, surface coatings, or opacities. Core idea: decouple the representation of repetitive objects into a single shared Gaussian template and lightweight per-instance learnable offsets, and implement a fully differentiable GPU instancing rasterizer that achieves high-FPS, memory-compact rendering while faithfully capturing subtle instance-specific variations.

Method

Overall Architecture

The input to InstGS is a pre-trained vanilla 3DGS scene representation. The overall framework operates in two primary phases: First, in Template Initialization, open-vocabulary 2D detectors and cross-view mask trackers extract target instances across training images, followed by gradient-driven 3D segmentation that isolates relevant Gaussians; these segmented clusters are subsequently aligned and fused via coarse-to-fine point cloud registration into a canonical shared template. Second, in Differentiable Gaussian Instancing and Joint Optimization, a custom CUDA rasterizer receives the shared template alongside lightweight per-instance offsets in position, base color, and opacity, projecting instances on-the-fly and jointly optimizing all parameters via chain-rule gradient backpropagation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Pre-trained 3DGS Scene"] --> B["Gradient-Driven Instance Segmentation<br/>Backprop 2D masks to 3D Gaussians"]
    B --> C["Two-Stage Registration & Merging<br/>FPFH-ICP alignment & canonical sampling"]
    C --> D["Template & Lightweight Offsets Decoupling<br/>Shared geometry vs per-instance variation"]
    D --> E["GPU Instancing Rasterizer & Joint Optimization<br/>On-the-fly CUDA mapping & gradient accumulation"]
    E --> F["Output: Memory-Efficient High-FPS Rendering"]

Key Designs

1. Gradient-Driven Cross-Frame Instance Segmentation: Precise Primitive Extraction without Extra Training Existing 3D semantic Gaussian segmentation methods rely on high-dimensional contrastive feature distillation, demanding substantial training overhead and remaining fragile to cross-view occlusions and illumination shifts. InstGS sidesteps auxiliary neural fields by directly utilizing 2D foundation models. Grounding DINO detects bounding boxes from text prompts, SAM segments pixel masks, and a view-consensus clustering algorithm tracks consistent masks \(M^k \in \{0, 1\}^{H \times W}\) across frames. By examining the 3DGS pixel rendering formulation where positive pixel derivatives indicate contributing Gaussians, a masked MSE loss activates the gradients of associated 3D Gaussians. To prevent large background Gaussians from bleeding into object boundaries, a weighted background removal gradient is introduced: $\(\mathbf{g} = \sum_{\mathbf{v}^k \in \mathcal{V}} \left( \nabla \mathcal{L}_{\text{GSeg}}(\mathbf{v}^k, M^k) - \alpha \nabla \mathcal{L}_{\text{GSeg}}(\mathbf{v}^k, 1 - M^k) \right)\)$ Setting \(\alpha = 0.8\) ensures that out-of-view object Gaussians still retain positive gradients. Gaussians with positive total gradients are classified as instance components, while those with negative gradients are assigned to the static background.

2. Two-Stage Registration and Merging: Constructing a Canonical Shared Geometric Template After segmenting individual Gaussian clusters belonging to the same semantic class, they must be aligned into a unified canonical coordinate frame. Designating the first instance as the fixed reference, InstGS employs a two-tier registration pipeline: Fast Point Feature Histograms (FPFH) determine an initial global coarse alignment, followed by point-to-point Iterative Closest Point (ICP) optimization to converge on an accurate \(4 \times 4\) rigid spatial transformation \(T_k = [R_k \mid t_k]\) for each instance \(k\). Once all instances are mapped into canonical space, a merging and uniform resampling step produces the final template, where the target primitive count is set to the average Gaussian count per instance. Only canonical 3D coordinates and zero-order spherical harmonics (base color) are preserved, while scales, rotations, and opacities are freshly initialized to prevent instance-specific bias.

3. Template and Lightweight Offsets Decoupling: Balancing Compactness and Expressiveness Assigning offsets to every Gaussian parameter (full covariance, full spherical harmonics) would negate memory savings and mirror unconstrained separate representations; conversely, prohibiting offsets prevents capturing real-world variations such as varied racing car liveries or minor manufacturing tolerances. Through ablation analysis, InstGS restricts offsets to three lightweight attributes: position offset \(\Delta p_i^k\), base color offset \(\Delta c_i^k\) (corresponding to \(c^{\{0,0\}}\)), and opacity offset \(\Delta o_i^k\). Here, \(\Delta p\) accommodates subtle structural discrepancies, \(\Delta c\) absorbs diffuse surface color and livery changes, and \(\Delta o\) allows redundant template primitives to be rendered fully transparent for instances missing specific sub-components. Furthermore, initializing \(\Delta p\) using the spatial difference between template and instance points eliminates floaters and blurry fog artifacts from the outset.

4. Differentiable GPU Instancing Rasterizer and Backpropagation: Native End-to-End Execution To achieve real-time rendering speedups without CPU data duplication, InstGS implements custom CUDA kernels for hardware-level instancing. The host memory only stores one shared template along with a compact offset table. During forward rendering, each CUDA thread handles one primitive and applies the transformation and additive offsets on-the-fly: $\(g_i^k = T_k(g_i) + \Delta_i^k\)$ Spatial transformation \(T_k\) rotates and translates the primitive position and orientation, while attribute offsets are added element-wise. During backpropagation under standard \(L_1\) and D-SSIM photometric supervision, gradients of the instance-specific offsets are accumulated directly by active threads: $\(\frac{\partial \mathcal{L}}{\partial \Delta_i^k} = \frac{\partial \mathcal{L}}{\partial g_i^k}\)$ Meanwhile, gradients targeting the shared template are projected back through the orthogonal rotation Jacobian \(J_k\) and aggregated across all \(K\) instances: $\(\frac{\partial \mathcal{L}}{\partial \mathbf{g}_i} = \sum_{k} J_k^\top \frac{\partial \mathcal{L}}{\partial \mathbf{g}_i^k}\)$ This formulation guarantees smooth, stable, end-to-end convergence of both shared structures and individual instance details.

Key Experimental Results

Main Results

The authors evaluated InstGS on the self-collected ReScene dataset comprising over 20 synthetic and drone-captured real-world scenes with repetitive structures, benchmarking against InstantNGP, 3DGS, Hierarchical 3DGS (3DGS*), Splat-and-Replace (Splat&Replace), Self-Organizing Gaussians (SOG), OctreeGS, and CompactGS.

Dataset Metric 3DGS (Baseline) CompactGS Ours (InstGS) Gain / Relative Performance
Synthetic PSNR (dB) ↑ 34.9 34.8 35.2 Outperforms vanilla 3DGS (+0.3 dB)
Synthetic SSIM ↑ 0.985 0.982 0.983 Preserves fine structural details
Synthetic LPIPS ↓ 0.026 0.028 0.026 Matches state-of-the-art perceptual fidelity
Synthetic CPU Memory ↓ 198.4 MB 23.9 MB 20.4 MB 89.7% memory reduction (~10x compression)
Synthetic Rendering FPS ↑ 117 196 204 +74.4% frame rate speedup
Real PSNR (dB) ↑ 27.4 26.9 27.8 Best reconstruction quality (+0.4 dB vs 3DGS)
Real SSIM ↑ 0.912 0.906 0.913 Higher structural fidelity
Real LPIPS ↓ 0.089 0.101 0.089 Matches top perceptual scores
Real CPU Memory ↓ 312.7 MB 42.3 MB 37.9 MB 87.9% memory reduction
Real Rendering FPS ↑ 98 113 128 +30.6% frame rate speedup

In the 8-Formula racing car teaser scene (original paper Table 2), InstGS achieves 35.12 dB PSNR (virtually matching 3DGS's 35.18 dB), while slashing memory from 200.51 MB to 41.24 MB and boosting rendering speed from 115 FPS to 201 FPS, requiring only 3 min 11 sec of post-initialization fine-tuning.

Ablation Study

The paper systematically investigates template initialization strategies and offset configurations (original paper Figure 5).

Initialization Strategy Offset Configuration PSNR (dB) SSIM Observation / Note
Max Instance Selection \(\Delta c_{\{l,m\}}\) (Full SH) 17.9 0.767 Severe artifacts without geometric and opacity adaptation
Max Instance Selection \(\Delta c_{\{l,m\}} + \Delta o + \Delta p\) 20.4 0.885 Offset offsets sub-optimal template selection
Merge & Registration (Ours) \(\Delta c_{\{l,m\}}\) (Full SH) 18.5 0.803 Robust baseline template
Merge & Registration (Ours) \(\Delta c_{\{l,m\}} + \Delta o\) 21.5 0.909 Opacity offset drastically suppresses ghosting
Merge & Registration (Ours) \(\Delta c_{\{l,m\}} + \Delta o + \Delta p\) 21.7 0.916 Position offset aligns fine geometric boundaries
Merge & Registration (Ours) \(\Delta c_{\{l,m\}} + \Delta o + \Delta p + \Delta \text{cov}\) 22.0 0.922 Marginal gain (+0.3 dB) at heavy memory cost
Merge & Registration (Ours) \(\Delta c_{\{0,0\}} + \Delta o + \Delta p\) (Final) 21.6 0.913 Optimal Pareto frontier: zero-order color offset retains fidelity

Key Findings

  • Base color combined with opacity is the sweet spot: High-degree spherical harmonic offsets cannot resolve surface misalignment. Introducing \(\Delta p\) and \(\Delta o\) produces a >3.1 dB leap in PSNR. Reducing color offsets to the 0-th order base term \(\Delta c_{\{0,0\}}\) costs only 0.1 dB while preserving extreme parameter compactness.
  • Covariance offsets introduce parameter bloat: Full covariance offsets cause over-deformation and severe storage expansion, defeating the purpose of template-based instancing.
  • Position offset initialization is essential: Initializing \(\Delta p\) with true Euclidean coordinate differences between instances and the template resolves blurry fog artifacts within 1,000 iterations, whereas zero-initialization struggles to escape sub-optimal local minima.

Highlights & Insights

  • Bridging classic graphics instancing and neural 3DGS: Demonstrates how classic GPU draw-call instancing paradigms can be married with differentiable Gaussian splatting, replacing redundant independent representations with GPU-side on-the-fly attribute transformations.
  • Zero-training gradient-driven segmentation: Cleverly turns 3DGS rasterization gradients into an instance extraction tool using 2D tracking masks, eliminating the need to train complex 3D segmentation networks or contrastive feature fields.
  • Multi-template scene scalability: Capably handles multiple disparate object categories simultaneously (e.g., chairs, tables, and vases within a single room) through multi-template concatenation and CUDA kernel pointer offsets.

Limitations & Future Work

  • Reliance on repetitive object priors: The computational and storage gains scale proportionally with the number and spatial coverage of repetitive objects. The method provides no compression benefit in purely heterogenous, unstructured scenes.
  • Breakdown under extreme topological divergence: When applied to radically dissimilar shapes (e.g., attempting to instance a truck and a train from a single template, as analyzed in Figure 10), opacity offsets cannot completely erase structural differences, resulting in visible ghosting and missing surfaces.
  • Future directions: The authors highlight substantial promise for interactive scene editing and digital twin generation, allowing users to modify or animate a single master template and propagate edits across thousands of instanced objects in real time.
  • vs Splat-and-Replace: Splat-and-Replace duplicates identical Gaussian clusters on the CPU, causing severe visual degradation (PSNR drops to 23.28 dB on the teaser scene) when instances have texture or lighting variations; InstGS handles complex liveries via GPU offsets, achieving 35.12 dB.
  • vs OctreeGS / CompactGS: Compression methods compress individual primitives using vector quantization or tree structures, but do not exploit cross-instance geometric redundancy; InstGS achieves higher rendering FPS (204 vs 196) and lower memory footprint (20.4 MB vs 23.9 MB).
  • vs SAGA / LangSplat: Traditional 3D Gaussian segmentation methods require substantial GPU memory and long training times to distill 2D CLIP/SAM features; InstGS achieves cleaner 3D instance extraction using direct gradient propagation with zero additional training.

Rating

  • Novelty: ⭐⭐⭐⭐☆ (Pioneering GPU-level differentiable Gaussian instancing for repetitive scene rendering)
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Comprehensive benchmarking on the diverse ReScene dataset, meticulous component ablations, and transparent failure case analysis)
  • Writing Quality: ⭐⭐⭐⭐⭐ (Rigorous mathematical formulation, clear pipeline figures, and coherent structure)
  • Value: ⭐⭐⭐⭐⭐ (Delivers an effective, production-oriented rendering paradigm for robotics simulation, city reconstruction, and interactive virtual environments)