Structure Gaussian Splatting SLAM

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/yanyan-li/StructureGS-SLAM
Area: 3D Vision
Keywords: 3D Gaussian Splatting, Dense SLAM, Planar Gaussian Instances, Structure Closure, Pose Graph Optimization

TL;DR

Addressing pose drift and floater artifacts caused by vanishing photometric gradients in low-texture and repetitive indoor scenes, this paper explicitly models ubiquitous planes as persistent Planar Gaussian Instances (PGIs) and introduces a "structure closure" mechanism estimating relative camera poses from non-parallel planes without trajectory revisits, achieving sub-centimeter tracking accuracy and photorealistic dense reconstruction.

Background & Motivation

Dense simultaneous localization and mapping based on 3D Gaussian Splatting (3DGS) has recently shown remarkable rendering efficiency and geometric fidelity. By projecting differentiable Gaussian ellipsoids onto camera viewpoints and minimizing photometric and geometric depth residuals against RGB-D observations, these systems jointly optimize camera trajectories and 3D scene representations. However, this dense pixel-alignment paradigm suffers from fundamental vulnerabilities in typical indoor environments: across large textureless expanses such as blank walls and floors, photometric gradients vanish almost entirely; meanwhile, in repetitive corridors or symmetric rooms, visual cues become degenerate. Under such conditions, Gaussian primitives readily drift away from genuine physical surfaces and proliferate into floating artifacts (floaters), causing severe global pose drift.

Existing GS-SLAM systems mitigate long-term drift primarily through classical loop closure or restrictive global layout assumptions, such as Manhattan or Atlanta world priors. However, these mechanisms rely on demanding prerequisites: loop closure is fundamentally opportunistic because it strictly requires the camera to revisit previously observed areas with significant visual overlap, failing completely during exploratory or unidirectional traversals; global orthogonality priors, on the other hand, are brittle whenever real-world architecture deviates from ideal perpendicular layouts. Crucially, most existing pipelines treat planar structures merely as transient, frame-level geometric cues during tracking, lacking persistent structural entities maintained and optimized across time.

Indoor environments naturally abound with planar structures like walls, floors, ceilings, and tables. These planes remain invariant across varied viewpoints, and their surface normals and spatial offsets provide strong analytic constraints on camera motion. The core idea is to elevate planar structures into persistent Planar Gaussian Instances (PGIs) within the 3D Gaussian map, establish "structure closures" from geometric co-observations of non-parallel planes without requiring physical trajectory revisits, and anchor the global map via Sparse Global Viewpoints and texture-adaptive joint optimization to eliminate floaters and long-horizon drift.

Method

Overall Architecture

The system accepts a continuous stream of RGB-D frames. Its complete pipeline comprises four core stages: front-end structure-aware tracking and keyframe selection, single-view plane extraction and local-to-global PGI association, structure-closure detection across non-parallel planes, and joint optimization anchored by Sparse Global Viewpoints. The front-end tracks camera poses using an adaptive RGB-Depth consistency loss and inserts keyframes based on both appearance co-visibility and planar structural drift. At each keyframe, 2D planar segments are extracted and back-projected into 3D camera-frame planes, which are matched to global PGIs via geometric gating and projected IoU to assign continuous plane embeddings to Gaussian primitives. Whenever two non-adjacent frames co-observe two or more non-parallel PGIs, an analytic relative pose is calculated and refined via multi-plane ICP, generating structure-closure edges in the pose graph. Finally, active camera poses and Gaussian parameters are jointly optimized within a sliding window while fixed Sparse Global Viewpoints (SGVs) preserve long-range global consistency.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["RGB-D Video Stream"] --> B["Front-End Structure-Aware Tracking & Keyframe Selection"]
    B --> C["Planar Gaussian Instance (PGI) Map Construction & Association"]
    C --> D["Structure Closure & Pose Graph Optimization"]
    D --> E["Sparse Global Viewpoint (SGV) Anchoring"]
    E --> F["Texture-Adaptive RGB-D Consistency & Planar Regularization"]
    F --> G["High-Fidelity 3D Gaussian Map & Low-Drift Trajectory"]

Key Designs

1. Planar Gaussian Instance (PGI) Map Construction & Association: Promoting transient planar cues to persistent map entities Conventional methods leverage planar detections solely as temporary tracking regularizers without cross-view entity continuity, while discrete scalar plane IDs introduce arbitrary numerical orderings that destabilize optimization. To resolve this, each scene Gaussian primitive is augmented into \(G_i = [\mu_i, \Sigma_i, o_i, c_i, \theta_i]\), where \(\theta_i \in \mathbb{R}^3\) denotes a continuous plane instance embedding vector. Sigmoid activations produce instance color predictions \(c_i = \sigma(\theta_i)\), which are supervised against ground-truth plane palettes via an edge-aware Charbonnier loss: $\(E_{\text{pgi}} = \frac{1}{|\Omega|} \sum_{u \in \Omega} w_{\text{edge}}(u) \sqrt{\|\hat{c}(u) - \mathbf{y}(u)\|_2^2 + \varepsilon^2}\)$ For newly added keyframes, the system extracts the \(K\) largest planar segments and merges over-segmented patches. Local planes are matched with global PGIs through a two-stage gating process: first filtering by normal angular difference \(\theta_{ig} = \arccos(n_i^\top n_g)\) and normalized distance difference \(\Delta d_{ig}\), and second validating candidate pairs by computing the intersection-over-union \(\text{IoU}(i, g)\) of their projected 2D masks. A Hungarian algorithm solves the optimal global assignment, and unmatched valid segments instantiate new PGIs, ensuring persistent lifelong structural modeling.

2. Structure Closure & Pose Graph Optimization: Revisit-free geometric constraints from non-parallel planes Visual loop closure fails in long corridors or repetitive rooms due to lack of visual overlap and distinctive features. Structure closure exploits the rigid spatial constraints established by intersecting non-parallel planes observed from disparate viewpoints. When two non-consecutive keyframes co-observe at least two matched PGIs with non-parallel normals, the plane normals analytically determine the relative 3D rotation, while plane-to-camera offsets form linear equations constraining relative translation. To suppress sensor noise, this closed-form initial pose is further refined through a lightweight multi-plane Iterative Closest Point (ICP) registration over the corresponding plane support points and retained non-planar geometric features. The resulting structure-closure edge is incorporated alongside sequential odometry edges into a pose graph \(G = (V, E)\), enabling frequent, early correction of trajectory drift long before any physical loop revisit occurs.

3. Sparse Global Viewpoint (SGV) Anchoring: Harmonizing computational tractability with long-horizon consistency Optimizing the entire historical trajectory at every iteration is computationally prohibitive, whereas pure sliding-window optimization over recent frames inevitably incurs unconstrained cumulative drift. To bridge this gap, the framework designates a compact set of anchor keyframes termed Sparse Global Viewpoints (SGVs) situated outside the active window. SGVs are selected according to strict structural topology: prioritizing keyframes participating in structure-closure edges, followed by keyframes sharing at least two PGIs with the active window, with highly connected pose graph nodes serving as fallbacks. During bundle adjustment, SGV poses remain strictly fixed, serving as global reference anchors that stabilize the sliding-window optimization and prevent drift propagation along the temporal horizon.

4. Texture-Adaptive RGB-D Consistency & Planar Regularization: Balancing visual gradients and structural compactness Standard photometric supervision collapses in textureless areas due to vanishing image gradients. The system introduces a texture-aware hybrid formulation based on image gradients \(G(u, v)\) computed via a Sobel operator, yielding a normalized texture gate: $\(\mathcal{T}(u, v) = 1 - \frac{G(u, v) - G_{\min}}{G_{\max} - G_{\min} + \epsilon}\)$ For low-texture pixels (\(\mathcal{T} \to 1\)), depth supervision via a robust Charbonnier penalty is heavily emphasized, whereas high-texture regions prioritize combined \(L_1\), gradient, and SSIM photometric losses. In addition, to prevent Gaussian primitives on planar surfaces from puffing into volumetric floaters, two geometric regularizers are applied: a flattening loss \(L_{\text{flat}} = \mathbb{E}_i \max(0, s_3 - k \sqrt{s_1 s_2})\) forcing the third principal scale \(s_3\) to contract into a thin surfel-like disc, and a normal alignment loss \(L_n = \mathbb{E}_i |n_g \cdot n_p - 1|\) aligning the Gaussian minor axis with the corresponding PGI normal. This joint constraint maintains clean planar boundaries and eliminates airborne floaters.

Loss & Training

The joint optimization objective over active-window camera poses and Gaussian primitives (with SGVs fixed) is formulated as: $\(\mathcal{L} = \lambda_{\text{pgi}} E_{\text{pgi}} + \lambda_{\text{pho}} E_{\text{pho}} + \lambda_{\text{geo}} E_{\text{geo}} + \lambda_{\text{str}} E_{\text{str}}\)$ where \(E_{\text{pho}}\) integrates color \(L_1\), gradient, and SSIM terms; \(E_{\text{geo}}\) enforces robust depth Charbonnier residuals; and \(E_{\text{str}} = \omega_n L_n + \omega_{\text{flat}} L_{\text{flat}}\) penalizes planar distortion. Time-critical Gaussian rasterization and backward gradient passes are executed via custom CUDA kernels, while tracking and pose-graph optimization run in PyTorch, delivering real-time processing on a single NVIDIA RTX 4090 GPU.

Key Experimental Results

Main Results

The system is evaluated on synthetic indoor scenes from the Replica dataset (8 sequences) and diverse physical sequences from the TUM RGB-D benchmark, benchmarked against leading neural implicit SLAM methods (NICE-SLAM, Point-SLAM, Vox-Fusion) and state-of-the-art Gaussian SLAM frameworks (SplaTAM, MonoGS, GS-SLAM, LoopSplat).

Table 1: Novel view rendering comparison on Replica (PSNR ↑ / SSIM ↑ / LPIPS ↓)

Method	Representation	Avg. PSNR (dB) ↑	Avg. SSIM ↑	Avg. LPIPS ↓
NICE-SLAM	Neural Implicit Grid	24.42	0.809	0.233
Vox-Fusion	Neural Voxel Implicit	24.41	0.801	0.236
Point-SLAM	Neural Point Implicit	35.62	0.977	0.114
SplaTAM	3D Gaussian Splatting	33.89	0.970	0.099
GS-SLAM	3D Gaussian Splatting	34.27	0.975	0.082
LoopSplat	3DGS + Loop Closure	36.63	0.985	0.112
MonoGS	3D Gaussian Splatting	39.31	0.974	0.045
Ours	3DGS + Structure Closure	39.70	0.977	0.042

Table 2: Camera trajectory estimation comparison on Replica (ATE RMSE in cm ↓)

Method	R0	R1	R2	Off0	Off1	Off2	Off3	Off4	Avg. ATE (cm) ↓
NICE-SLAM	0.97	1.31	1.07	0.88	1.00	1.06	1.10	1.13	1.07
Vox-Fusion	1.37	4.70	1.47	8.48	2.04	2.58	1.11	2.94	3.09
Point-SLAM	0.61	0.41	0.37	0.38	0.48	0.54	0.69	0.72	0.53
SplaTAM	0.31	0.40	0.29	0.47	0.27	0.29	0.32	0.72	0.38
MonoGS	0.46	0.31	0.33	0.47	0.48	0.24	0.18	2.06	0.57
GS-SLAM	0.48	0.53	0.33	0.52	0.41	0.59	0.46	0.70	0.50
Loopy-SLAM	0.24	0.24	0.28	0.26	0.40	0.29	0.22	0.35	0.29
LoopSplat	0.28	0.22	0.17	0.22	0.16	0.49	0.20	0.30	0.26
Ours	0.28	0.23	0.26	0.22	0.16	0.13	0.17	0.28	0.22

Table 3: Camera trajectory accuracy on TUM RGB-D dataset (ATE RMSE in cm ↓)

Method	fr1/desk	fr1/desk2	fr2/xyz	fr2/rpy	fr3/tex_far	fr3/tex_near	Avg. ATE (cm) ↓
Loopy-SLAM	3.79	3.38	1.62	0.40	1.22	2.01	2.07
SplaTAM	3.35	6.54	1.24	0.42	1.43	2.41	2.57
MonoGS	1.48	7.03	1.44	0.57	1.72	1.81	2.34
LoopSplat	2.08	3.54	1.58	1.04	3.71	4.66	2.77
Ours	1.60	4.02	1.47	0.42	0.08	1.76	1.56

Ablation Study

Table 4: Ablation on Structure Closure activation interval (Replica R0 & Off0)

Activation Interval (\(k\) keyframes)	R0 ATE (cm) ↓	R0 PSNR (dB) ↑	R0 LPIPS ↓	Off0 ATE (cm) ↓	Off0 PSNR (dB) ↑	Off0 LPIPS ↓
Every 4 keyframes	0.32	36.59	0.051	0.25	43.61	0.031
Every 10 keyframes	0.37	36.51	0.047	0.32	43.44	0.032
Every 25 keyframes	0.53	36.04	0.054	0.61	42.91	0.039

Table 5: Pose estimation robustness in textureless sequences (TUM RGB-D ATE RMSE in cm ↓)

Sequence	MonoGS	LoopSplat	Ours	Observation
fr3/notex_far	16.63	8.18	7.47	Accurate tracking across distant textureless walls
fr3/notex_near	55.59	Failed / Diverged (\(\times\))	16.36	Preserves stable map where competitors fail

Key Findings

• High-frequency structure closures inhibit drift accumulation: Table 4 demonstrates that even when structure closure is evaluated only once every 10 keyframes (~50 frames), tracking accuracy outperforms standard baselines. Widening the interval to 25 frames degrades ATE to 0.53 cm and 0.61 cm, proving that persistent planar constraints supply timely corrections.
• Superior survival in texture-deprived environments: In the challenging fr3/notex_near sequence where walls and floors lack high-frequency textures, LoopSplat tracking diverges completely (\(\times\)) and MonoGS suffers 55.59 cm error. Our system maintains a viable trajectory with 16.36 cm ATE, demonstrating the vital stabilization provided by PGI geometric priors.
• Continuous embeddings eliminate ordering artifacts: Comparative analysis reveals that discrete scalar plane IDs impose artificial ordinal penalties between arbitrarily indexed planes during continuous gradient descent. In contrast, 3D continuous embeddings combined with color palette projection provide smooth, discriminative supervision that preserves crisp instance boundaries.

Highlights & Insights

• Revisit-independent loop closure paradigm: Traditional SLAM systems depend strictly on returning to previously explored physical vantage points. Structure closure recognizes that intersecting non-parallel planes establish a global geometric anchor regardless of camera orientation, enabling drift correction in exploratory paths and long hallways.
• Differentiable plane instance field rendering: By encoding plane instance identities into continuous Gaussian attributes and mapping them via a consistent color palette, the system unifies structural segmentation and radiance field rendering into a single rasterization pipeline.
• Geometry-enforced Gaussian thinning: Applying flattening and normal alignment losses compels Gaussians on planes to function as thin surfels. This physically grounded constraint curtails unconstrained 3D expansion and eradicates floating artifacts without requiring heuristics.

Limitations & Future Work

• Dependency on non-parallel planar structures: Structure closure relies on observing at least two non-parallel dominant planes simultaneously. The system degenerates to standard tracking in non-structural environments, open fields, or scenes dominated by curved geometry or a single infinite plane.
• Sensitivity to severe occlusion and segmentation errors: Front-end single-view plane segmentation can suffer from clutter, dynamic foreground objects, or dramatic lighting variations, potentially admitting false planar patches into PGI initialization.
• Future directions: Integrating foundation segmentation models (such as SAM) could improve initial plane boundary discovery under complex clutter, while expanding structural instances to quadrics or higher-order parametric surfaces would broaden applicability to general outdoor environments.

Related Work & Insights

• vs. MonoGS [14]: MonoGS pioneered incremental Gaussian SLAM via pure photometric alignment but drifts heavily on textureless surfaces and generates floating artifacts. Our framework integrates PGIs and texture-aware gates, reducing average Replica ATE from 0.57 cm to 0.22 cm while improving PSNR from 39.31 dB to 39.70 dB.
• vs. LoopSplat [34]: LoopSplat executes visual loop closures by registering Gaussian submaps upon trajectory revisits. Our method forms structure closures without needing physical revisits, outperforming LoopSplat on Replica (0.22 cm vs. 0.26 cm) and preventing tracking failure in texture-free sequences.
• vs. ManhattanSLAM [32]: Traditional structural SLAM systems enforce global Manhattan or Atlanta frame orthogonality. Our method requires only local pairs of non-parallel planes, allowing robust operation in arbitrary non-orthogonal architectural layouts.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Conceptualizing persistent Planar Gaussian Instances and revisit-free structure closures is highly original and practical]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive comparisons across rendering metrics, trajectory accuracy, structure closure intervals, and textureless stress tests]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical formulations, well-organized methodology, and insightful qualitative and ablation discussions]
• Value: ⭐⭐⭐⭐⭐ [Presents an impactful architectural blueprint for resolving geometry degradation and drift in dense neural/Gaussian SLAM]