Skip to content

Incremental Online Scene Reconstruction by 3D Gaussian Triangulation

Conference: ECCV 2026
Paper: ECCV Official
Area: 3D Vision
Keywords: Gaussian Triangulation, Incremental Reconstruction, 3D Gaussian Splatting, Dense Geometric Gaussians, Online Mapping

TL;DR

This paper introduces an online incremental scene reconstruction framework that unifies high-fidelity explicit mesh generation and photorealistic rendering by directly triangulating planar 3D Gaussian surfels guided by plane-based pulling constraints and dynamic historical freezing.

Background & Motivation

Incremental 3D scene reconstruction is a foundational capability for real-world computer vision and robotics, underpinning mission-critical systems in augmented reality and autonomous navigation. Traditional online approaches, such as KinectFusion and its volumetric TSDF variants, incrementally fuse depth observations into regular voxel grids and extract triangle meshes via Marching Cubes. However, their reliance on rigid voxel discretizations incurs prohibitive memory footprints and restricts spatial scalability, while the mesh extraction frequency lags substantially behind incoming sensor streams, preventing low-latency geometric feedback for time-critical decision-making.

Recently, 3D Gaussian Splatting (3DGS) has emerged as a groundbreaking explicit radiance field representation, enabling real-time, photorealistic novel view synthesis. Nonetheless, its discrete, unstructured, and semi-transparent primitives pose severe challenges for surface reconstruction. Prevailing surface-reconstruction extensions, such as SuGaR, 2D GS, and GOF, typically decouple radiance optimization from mesh generation. They require optimizing the entire Gaussian field across all training frames, constructing an intermediate global implicit scalar field, and executing Poisson Surface Reconstruction or Marching Cubes as an offline post-processing step. Whenever a new frame arrives, these methods must re-optimize the full scene and recompute the global extraction pipeline from scratch, making them ill-suited for online incremental setups.

Bridging the divide between high-fidelity Gaussian rendering and low-latency online mesh extraction requires treating optimized Gaussian primitives themselves as structural mesh vertices, eliminating intermediate global volumetric representations. Core idea: constrain 3D Gaussians into high-opacity planar elliptical surfels, rectify input depth sensor noise via a plane-based pulling constraint against oriented points, perform fast direct triangulation through tangent-plane polar angle sorting, and freeze fully converged historical regions dynamically to achieve scalable, constant-memory online incremental mesh reconstruction and photorealistic rendering.

Method

Overall Architecture

The framework ingests an incoming stream of \(M\) RGB-D frames \(\{I_i, D_i\}_{i=1}^M\) with known camera poses. It adopts the Dense Geometric Gaussian Representation as a shared substrate for both volumetric color rendering and explicit surface extraction. During online mapping, geometric Gaussians are incrementally initialized and supplemented based on incoming oriented point clouds and photometric/geometric rendering residuals. The active Gaussian primitives are optimized within a sliding window under photometric rendering losses, depth losses, and structural geometric constraints (local plane pulling and normal consistency). Subsequently, a high-confidence Geometric Gaussian Set is extracted and directly converted into local triangle meshes via compressed octree neighbor search, position smoothing, and tangent-plane polar angle sorting. Finally, an isotropic local remeshing step stitches new meshes with existing surfaces seamlessly, while fully optimized historical regions are dynamically frozen to ensure bounded memory and computation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Stream<br/>Continuous RGB-D Frames"] --> B["Dense Geometric Gaussian Representation<br/>Planar Elliptical Surfel Initialization"]
    B --> C["Online Mapping & Gaussian Optimization<br/>Loss-Driven Addition & Pruning"]
    C --> D["Plane Pulling & Normal Constraints<br/>Surface Alignment & Noise Rectification"]
    D --> E["Geometric Gaussian Selection & Direct Triangulation<br/>Tangent-Plane Pruning & Polar Angle Meshing"]
    E --> F["Local Remeshing & Dynamic Freezing<br/>Isotropic Seam Stitching & History Freezing"]
    F --> G["Output Representations<br/>Incremental Watertight Meshes + Photorealistic NVS"]

Key Designs

1. Dense Geometric Gaussian Representation: Planar Elliptical Surfels for Explicit Geometry Standard 3DGS targets volumetric radiance rendering, permitting semi-transparent, volumetric ellipsoids that lack definite surface boundaries. To adapt Gaussians as explicit mesh vertices, this method flattens each primitive into a planar elliptical surfel by constraining its normal-axis scale component to zero (\(s_{i3} \to 10^{-6}\)) and enforcing near-solid opacity (\(\alpha_i \to 0.99\)). The rendering engine adopts a dual-branch design: color is integrated via standard \(\alpha\)-blending, whereas the geometric depth map \(\hat{D}(u, v)\) is analytically computed as the ray intersection with the frontmost opaque Gaussian surfel. This prevents the geometric blurring and pseudo-surfaces inherent in transmissive depth blending, ensuring sharp geometric localization.

2. Plane-Based Pulling and Normal Alignment Constraints: Rectifying Sensor Noise Real-world depth sensors suffer from high-frequency noise and missing observations, which destabilize unconstrained gradient backpropagation. The method models the oriented point cloud \(P\) as a discrete approximation of the zero-level surface \(Z(f)\) and imposes a point-to-plane pulling loss: $\(\mathcal{L}_{plane} = \sum_{(\boldsymbol{\mu}_i, \mathbf{p}_i) \in \mathcal{C}} |\mathbf{n}_i^\top (\boldsymbol{\mu}_i - \mathbf{p}_i)|\)$ where \(\boldsymbol{\mu}_i\) denotes the Gaussian centroid, and \(\mathbf{p}_i\) and \(\mathbf{n}_i\) represent the position and normal of the corresponding point in \(P\). This term pulls Gaussian centers onto the local tangent plane along the surface normal while granting tangential mobility to faithfully recover detailed appearances. A normal consistency loss \(\mathcal{L}_{n} = \sum |\mathbf{n}^{g \top}_i \cdot \mathbf{n}_i|\) further aligns the shortest axis of each surfel with local geometric normals, and a sparsity regularizer \(\mathcal{L}_{sparse}\) polarizes opacities toward 0 or 1, forming coherent, watertight surface sheets.

3. Fast Direct Gaussian Triangulation: Local Topology Without Implicit Fields Extracting explicit meshes from implicit fields via Marching Cubes is notoriously slow and memory-intensive. Here, the system directly triangulates the Geometric Gaussian Set \(G_{geo} = \{G_g \mid \alpha_g > 0.95, \, |(\mathbf{T}_{cw} \boldsymbol{\mu}_g)_z - \hat{D}| < \tau\}\), filtering out non-geometric floating artifacts. For each central Gaussian, candidate neighbors within an adaptive radius are retrieved using a compressed octree, filtered by mutual visibility and normal agreement (\(\langle\mathbf{n}_i, \mathbf{n}_j\rangle > 0.9\)), and fused to refine the central position. The valid neighbors are then projected onto the central Gaussian's tangent plane, sorted by polar angle around the center pivot, and connected into triangles between angularly adjacent neighbors subtending an angle \(> 10^\circ\). This direct 2D projection-based meshing operates in milliseconds and avoids degenerate sliver faces.

4. Local Remeshing and Dynamic Freezing: Long-Sequence Scalability Directly joining newly observed meshes with prior reconstructed geometry can introduce boundary seams, self-intersections, and vertex misalignments due to viewpoint discrepancies. An isotropic remeshing routine applies edge splitting (\(> 1.5\bar{L}\)), edge collapsing (\(< 0.5\bar{L}\)), edge flipping toward a target valence of 6, and 3 iterations of Laplacian smoothing, yielding clean topological transitions. Furthermore, to prevent resource exhaustion during long-sequence scanning, the system tracks the multi-view observation count \(N_{obs}\) and optimization loss for each region. When a region accumulates \(N_{obs} \ge 10\) observations and its loss converges below \(\epsilon_{gs} = 0.1\), its Gaussians and mesh elements are removed from the active computation graph and permanently frozen, bounding memory consumption to the local active window.

Loss & Training

The overall objective function combines photometric rendering, depth fidelity, geometric surface pulling, and opacity regularization: $\(\mathcal{L} = \lambda_c \mathcal{L}_{color} + \lambda_d \mathcal{L}_{depth} + \lambda_p \mathcal{L}_{plane} + \lambda_n \mathcal{L}_{n} + \lambda_s \mathcal{L}_{sparse}\)$ where \(\mathcal{L}_{color} = \sum |\mathbf{C}_i - \hat{\mathbf{C}}_i|\) and \(\mathcal{L}_{depth} = \sum |\mathbf{D}_i - \hat{\mathbf{D}}_i|\). The hyperparameters are set to \(\lambda_c = 0.8\), \(\lambda_d = 1.0\), \(\lambda_p = 0.05\), \(\lambda_n = 0.2\), and \(\lambda_s = 0.001\). Geometric Gaussian selection employs a depth residual threshold \(\tau = 0.001\). During online processing, the active sliding window and optimization iteration counts are configured as (6 frames, 50 iterations) for Replica and (3 frames, 75 iterations) for ScanNet++.

Key Experimental Results

Main Results

The reconstruction quality was benchmarked on the Replica dataset (8 sequences) against volumetric TSDF (KinectFusion), neural implicit SLAM (NICE-SLAM), and Gaussian SLAM baselines (MonoGS, RTG-SLAM) using Accuracy error (Acc. in cm), Accuracy Ratio (< 5cm), and Completion Ratio (< 5cm). Novel view rendering performance was evaluated on ScanNet++.

Dataset / Sequence Metric Ours RTG-SLAM [19] MonoGS [16] NICE-SLAM [29] KinectFusion [18]
Replica Avg. Acc. [cm] \(\downarrow\) 1.34 1.41 2.77 8.16 12.76
Replica Avg. Acc. Ratio [%] \(\uparrow\) 99.70 99.54 86.73 80.42 34.34
Replica Avg. Comp. Ratio [%] \(\uparrow\) 85.95 81.71 81.83 85.01 62.84
ScanNet++ Avg. (Novel View) PSNR [dB] \(\uparrow\) 24.40 21.65 - - -
ScanNet++ Avg. (Novel View) SSIM \(\uparrow\) 0.85 0.83 - - -
ScanNet++ Avg. (Novel View) LPIPS \(\downarrow\) 0.32 0.33 - - -
Replica Off0 Mapping Speed [FPS] \(\uparrow\) 10.34 3.65 1.48 - -
Replica Off0 Peak Memory [MB] \(\downarrow\) 2325 2751 5434 - -

In terms of mesh extraction efficiency evaluated on the Office0 scene from Replica, compared against Gaussian Opacity Fields (GOF): GOF using Marching Cubes took 444.26 s and generated a 964.95 MB mesh; GOF using Poisson Surface Reconstruction required 415.27 s with a 14.05 MB mesh. In contrast, the proposed direct Gaussian triangulation completed mesh extraction in merely 5.34 s with a compact mesh size of 3.32 MB—achieving an ~80x speedup while maintaining high geometric compactness.

Ablation Study

Ablations on Replica (Office1) validate the necessity of the architectural modules and individual loss formulations:

Config Acc. [cm] \(\downarrow\) Acc. Ratio [%] \(\uparrow\) Comp. Ratio [%] \(\uparrow\) Note
Full Method 0.88 99.99 88.32 Complete framework with all modules and losses
w/o Planar Constraint 1.19 99.04 88.58 Unconstrained 3D ellipsoids degrade surface fitting
w/o Neighborhood Selection 1.26 98.59 85.14 Missing visibility/normal filters causes topological errors
w/o Direct Triangulation (Poisson) 0.89 99.98 84.49 Offline Poisson achieves similar accuracy but sacrifices speed
w/o Local Remeshing 1.32 97.99 91.10 Discrepancies at inter-frame seams damage geometric fidelity
w/o Freezing 0.92 99.59 89.27 Full-scene gradient backprop causes resource overhead
Baseline Losses 1.13 98.58 86.34 Rendering losses only without geometric constraints
+ \(\mathcal{L}_{sparse}\) 1.09 98.92 86.50 Reduces semi-transparent floating artifacts
+ \(\mathcal{L}_n\) + \(\mathcal{L}_{sparse}\) 0.95 98.72 86.13 Enforces surface normal alignment
+ \(\mathcal{L}_{plane}\) + \(\mathcal{L}_{sparse}\) 0.93 99.78 86.50 Pulls Gaussians directly onto local planes
+ \(\mathcal{L}_{plane}\) + \(\mathcal{L}_n\) 0.90 99.77 86.04 Combines position pulling and orientation alignment

Key Findings

  • Crucial Impact of Geometric Pulling: Ablation reveals that incorporating \(\mathcal{L}_{plane}\) and \(\mathcal{L}_n\) drastically reduces reconstruction error from 1.13 cm to 0.88 cm. Relying solely on depth and color rendering gradients results in rough, wavy geometry, whereas planar pulling forces surfels to adhere tightly to the physical underlying surface.
  • Speed and Compactness of Direct Triangulation: By bypassing global implicit grid queries in favor of local 2D polar-angle sorting on tangent planes, mesh extraction is accelerated from hundreds of seconds down to 5.34 s, simultaneously eliminating redundant volumetric faces and yielding an ultra-compact 3.32 MB mesh.
  • Selection Threshold Sensitivity: The depth residual parameter \(\tau\) governs which Gaussians are admitted into the geometric set. Setting \(\tau = 0.0001\) overly filters primitives, leading to holes and missing geometry, whereas \(\tau = 0.1\) admits floating visual artifacts that compromise surface accuracy. A value of \(\tau = 0.001\) provides the ideal balance between completeness and precision.

Highlights & Insights

  • Repurposing 3DGS as Direct Mesh Topology Primitives: Demonstrates that 3D Gaussian Splatting can serve as more than an image synthesizer or an implicit field generator; flattened surfel Gaussians can function directly as explicit mesh vertices for real-time triangulation.
  • Tangentially Relaxed Normal Pulling: Restricting the geometric pulling force strictly to the normal direction rectifies sensor depth errors while preserving the tangential degrees of freedom necessary for Gaussian primitives to fit intricate visual textures.
  • Dynamic Bounded-Memory History Freezing: Leveraging observation counters and loss thresholds to identify fully converged regions and detach them from the active computational graph resolves the fundamental memory bloat of long-sequence Gaussian mapping.

Limitations & Future Work

  • Reliance on Explicit Depth Input: The approach depends on initial oriented point clouds and depth streams to anchor surface pulling; in scenarios with severe depth sensor failure (e.g., highly reflective surfaces, dark regions, or distant outdoor environments), reconstruction fidelity degrades. Exploring monocular RGB-only surface pulling is an important future direction.
  • Lack of Global Completion for Unobserved Areas: Because topology is formed strictly via local direct triangulation over observed primitives, unobserved occluded crevices cannot be hallucinated as complete surfaces, unlike neural implicit representations that leverage global inductive shape priors.
  • vs TSDF / KinectFusion [18]: KinectFusion is constrained by fixed voxel resolutions and cubic memory growth; this framework adapts surfel density dynamically and uses only 2.3 GB of GPU memory while delivering photorealistic rendering alongside explicit meshes.
  • vs SuGaR [7] & GOF [27]: SuGaR and GOF require complete sequence capture followed by offline implicit field extraction taking hundreds of seconds; this paper achieves incremental online reconstruction with a 5.34-second extraction time and seamless local remeshing.
  • vs RTG-SLAM [19]: RTG-SLAM constructs real-time Gaussian maps but relies on external ball-pivoting point-cloud heuristics for offline meshing; the proposed method natively enforces planar surface constraints during mapping, directly yielding high-accuracy watertight meshes.

Rating

  • Novelty: ⭐⭐⭐⭐☆ Elegantly unifies planar Gaussian surfels with direct tangent-space triangulation and incremental freezing, eliminating the need for intermediate implicit grids.
  • Experimental Thoroughness: ⭐⭐⭐⭐☆ Comprehensive evaluation on Replica and ScanNet++ covering geometric accuracy, rendering quality, runtime FPS, peak memory, and detailed module/loss ablations.
  • Writing Quality: ⭐⭐⭐⭐⭐ Well-structured, mathematically rigorous, with clear pipeline descriptions and informative qualitative/quantitative comparisons.
  • Value: ⭐⭐⭐⭐☆ Provides a practical blueprint for real-time robotics and AR applications requiring low-latency geometric meshes alongside photorealistic view synthesis.