VIGS-SLAM: Visual Inertial Gaussian Splatting SLAM

Conference: ECCV2026
Paper: ECCV Paper
Area: 3D Vision
Keywords: visual-inertial SLAM, Gaussian Splatting, IMU initialization, joint bundle adjustment, loop consistency

TL;DR

VIGS-SLAM jointly optimizes dense visual correspondences and IMU constraints, stabilizes startup through staged initialization, and propagates loop corrections to its Gaussian map, achieving 2.79 cm average ATE RMSE on EuRoC while improving novel-view rendering in difficult scenes.

Background & Motivation

Gaussian Splatting allows SLAM to produce dense, renderable scenes rather than only sparse landmarks. However, an attractive map does not automatically imply reliable localization: visual systems such as Splat-SLAM and HI-SLAM2 still depend on image correspondences. Motion blur, low texture, exposure changes, and dropped frames weaken these correspondences, while incorrect poses place subsequent Gaussians at incorrect locations. DROID-SLAM provides dense correspondences and per-pixel disparities that suit Gaussian initialization, but it still lacks motion constraints independent of images. High-frequency angular-velocity and acceleration measurements from an IMU complement this gap and help recover the metric scale missing from monocular vision.

The difficulty is not merely connecting an IMU, but preventing incorrect inertial states from damaging visual estimation. Gravity direction, scale, velocity, and bias are coupled during startup, so immediate joint optimization can begin from an unsuitable initialization. Bias should not remain permanently fixed either, because accumulated integration errors may otherwise be interpreted as genuine motion. The paper further argues that DBA-Fusion first solves visual BA and then passes its linearized result to an inertial optimizer, introducing a linearization gap between the stages. Simply appending Gaussian mapping after localization also does not ensure that the existing map remains consistent after loop closure corrects the trajectory.

The proposed system therefore follows a continuous estimation process: make inertial parameters usable, fuse visual and inertial information in one linear system, and propagate global trajectory corrections into the map. Its target is online localization and reconstruction from RGB plus IMU, not geometry supplied directly by LiDAR or a depth sensor. Core Idea: first establish a stable scale relationship between vision and inertia, then jointly optimize motion and depth, and propagate every loop-induced pose and scale correction to the corresponding Gaussians so that reliable tracking produces a consistent dense map.

Method

Overall Architecture

Inputs are a monocular image sequence and raw IMU measurements; outputs are the camera trajectory and a Gaussian map encoding color, opacity, and spatial shape. EuRoC supplies grayscale images and the other evaluated datasets provide RGB; LiDAR used to construct evaluation reference trajectories is not an input to this system. Staged IMU initialization precedes joint visual-inertial tracking, which estimates keyframe poses, disparities, velocities, and biases. Incremental Gaussian mapping runs alongside tracking and consumes its geometry; loop-consistent updates correct both the global pose graph and existing Gaussians. Here, joint optimization primarily refers to visual and inertial states, not to Gaussian rendering losses directly participating in frontend tracking.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Images + IMU"] --> Init["Staged IMU Initialization"]
    Init --> Track["Joint Visual-Inertial Tracking"]
    Track -->|Poses and depths| Map["Incremental Gaussian Mapping"]
    Track -->|Loop detected| Loop["Loop-Consistent Updates"]
    Map -->|Keyframe-anchored Gaussians| Loop
    Loop -->|Correct pose graph| Track
    Loop -->|Correct Gaussians| Map
    Map --> Output["Camera trajectory + Gaussian map"]
    Track --> Output

Key Designs

1. Staged IMU Initialization: establish scale and gravity before coupling observations

The first stage uses only visual residuals from the initial keyframes to estimate consistent poses and disparities up to scale. It establishes inter-image geometry without requiring an immediately correct gravity direction or sensor bias. The second stage accumulates additional visual keyframes and initially optimizes only the gravity-alignment rotation to align gravity between the inertial and current world frames. With visual poses held fixed, per-keyframe velocities, IMU biases, and a global log-scale parameter enter inertial optimization to recover metric scale. The fixed quantities are the current visual geometry estimates; the additional scale parameter expresses that geometry in physical length units. Only the third stage jointly minimizes visual and inertial residuals to refine the resulting states.

These are not three unrelated initializers, but a progressive release of coupling between variables. Vision supplies shape and relative motion, inertia then explains gravity, velocity, and scale, and joint refinement finally lets both sources adjust together. This reduces contamination of image geometry by unreliable initial bias or gravity estimates. In Table 6, removing three-stage initialization specifically means removing the intermediate inertial-only optimization stage, not removing every startup operation. The available main-text cache does not give numerical values for the two initialization keyframe counts and refers to supplementary material, so no configuration is invented here.

2. Joint Visual-Inertial Tracking: update motion, biases, and disparities in one linear system

The frontend selects keyframes according to optical-flow magnitude relative to the previous keyframe and enforces a maximum keyframe time interval to prevent excessively long IMU preintegration spans. Each keyframe maintains its pose, per-pixel disparities, velocity, and gyroscope and accelerometer biases. IMU preintegration predicts the new keyframe pose, which initializes visual correspondences to existing keyframes through geometric warping. If the trace of the preintegration covariance exceeds a threshold, initialization falls back to the previous pose rather than forcing a highly uncertain inertial prediction into the visual module. A frozen ConvGRU correspondence-update module then repeatedly refines dense correspondences and their confidence. The visual residual compares projections from current poses and disparities with the refined correspondences, weighted by confidence.

On the inertial side, high-frequency measurements between keyframes are preintegrated into relative rotation, position, and velocity, together with covariance and Jacobians for bias correction. Residuals measure disagreement between current motion states and these preintegrated measurements, with an additional temporal smoothness constraint on neighboring keyframe biases. Thus, per-keyframe bias estimation allows temporal variation without allowing every bias to vary arbitrarily. The implementation accounts for camera-IMU extrinsics, although the main equations omit this transformation for simplicity. Within the local sliding window, visual and inertial contributions enter the same Levenberg-Marquardt normal equations, solved with custom CUDA kernels. The disparity block is diagonal because each visual residual depends on the corresponding pixel depth, allowing efficient Schur elimination of these variables. Unlike finishing visual optimization before supplying a linearized factor to an independent inertial backend, this formulation updates poses, velocities, and biases from a shared current linearization.

3. Incremental Gaussian Mapping: turn tracking depths into an explicit renderable scene

Each new keyframe's disparities are converted to depths and unprojected using its current pose to initialize Gaussians. Gaussian means come from point positions, colors from corresponding pixels, and initial opacity is 0.5; anisotropic covariance describes spatial shape. Colors use direct RGB rather than spherical-harmonic coefficients, reducing online optimization complexity while limiting view-dependent appearance modeling. The system merges these Gaussians into the global map and performs 10 mapping iterations for each new keyframe. Each iteration samples keyframes from the current frontend graph and adds 2 global keyframes, rendering their colors and depths for supervision. Local views help update recently observed regions, while global views retain constraints from earlier map content.

Rendered color is obtained by alpha-blending visible Gaussians, and rendered depth is compared with frontend-estimated depth. This is ongoing map-parameter optimization for the current sequence, not offline training with ground-truth depth labels. Isotropic regularization discourages excessive elongation in sparsely observed regions; it does not restrict the representation itself to spherical Gaussians. Because mapping depends on frontend geometry, tracking errors affect the map, and improved visual-inertial estimates can improve rendering through this route. The authors explicitly state that the current Gaussian map does not directly improve tracking, so the data flow should not be interpreted as rendering errors optimizing camera poses in the frontend.

4. Loop-Consistent Updates: move old Gaussians together with the corrected trajectory

A loop-detection thread uses optical-flow differences between new and old keyframes to establish loop edges, while a global pose graph continually receives relative-pose constraints from the frontend. After detecting a loop, the system uses pose graph bundle adjustment, PGBA, rather than full global BA over all historical visual relationships. Expensive dense visual updates are restricted to loop pairs, while other graph edges use lightweight relative-pose constraints. Optimization takes place in Sim(3), correcting rotation and translation as well as long-term scale drift. This trades some accuracy for faster global correction and should not be confused with complete end-of-run refinement.

After trajectory correction, each Gaussian receives the transformation of the keyframe that initialized and anchors it. Its old world-frame center is first transformed into the old camera frame, scaled by the keyframe's scale change, and transformed back into the world using the updated pose. Covariance must also rotate and scale by the square of the scale change; otherwise the center is corrected but the Gaussian shape and support remain inconsistent. Color and opacity remain unchanged because this operation corrects geometry rather than relearning appearance. The updates are batched, avoiding reinitialization and optimization of every Gaussian. Table 7 isolates removal of this propagation step, showing that loop closure must correct the map's geometry as well as the trajectory.

A Worked Example

Consider a camera moving down a corridor, producing blurred images during a rapid turn, and later revisiting an earlier area; this is an explanatory scenario, not an additional experiment. After initialization, high-frequency IMU measurements provide the next keyframe's motion estimate, from which the correspondence module refines dense visual matches. If that inertial prediction is uncertain, initialization falls back to the previous pose instead of trusting the IMU unconditionally. Local joint optimization updates pose, velocity, disparity, and bias so that both observation sources explain the motion. The new keyframe's depths initialize Gaussians with opacity 0.5, followed by 10 mapping iterations. Upon revisiting the old area, PGBA adjusts poses and scales through loop edges, and Gaussians anchored to old keyframes move, rotate, and scale accordingly. The result is a more consistent reconstruction of one scene rather than retaining conflicting pre-loop and post-loop geometry.

Loss & Training

Tracking combines visual reprojection and preintegrated inertial residuals; bias smoothness is part of the inertial constraints. The system diagram marks ConvGRU as frozen: online optimization updates sequence states and map parameters rather than retraining the correspondence network. The two mapping data terms are legible in the main text and compare rendered color with input color, and rendered depth with tracking depth:

\[ L_c=\|\hat I-I\|_1,\qquad L_d=\|\hat D-D\|_1. \]

The total mapping loss weights these terms and isotropic regularization; verifiable numerical weights are absent from the available main text, so no hyperparameters are supplied here. Depth supervision comes from system estimates rather than an independent sensor, primarily enforcing consistency with tracking geometry rather than directly establishing surface accuracy. The descriptions of acceleration and angular-velocity symbols on page 5 appear interchanged; this note uses physical measurement names instead of inheriting that ambiguity. Several matrix equations, the PGBA objective, and Gaussian-update equations are damaged in text extraction, so the explanation follows readable prose rather than presenting reconstructed matrices as exact source equations.

Key Experimental Results

Main Results

ATE RMSE is computed after aligning the estimated and reference trajectories with evo, in cm, with lower values preferred; the main text does not specify all methods' alignment degrees of freedom here. The following table selects average tracking errors from Tables 1, 2, 4, and 5 and FAST-LIVO2 rendering metrics from Table 3, on pages 9, 9, 11, 12, and 11, respectively. Sensor configurations differ: HI-SLAM2 uses images only, whereas VINGS-Mono and VIGS-SLAM use images plus IMU.

Dataset / metric	HI-SLAM2	VINGS-Mono	VIGS-SLAM	Source
EuRoC / ATE RMSE, cm, lower is better	2.94	24.47	2.79	Table 1, p. 9
RPNG / ATE RMSE, cm, lower is better	N/A	6.65	1.68	Table 2, p. 9
UTMM / ATE RMSE, cm, lower is better	8.10	12.54	3.24	Table 4, p. 11
FAST-LIVO2 / ATE RMSE, cm, lower is better	10.22	104.80	6.08	Table 5, p. 12
FAST-LIVO2 / PSNR, higher is better	21.49	10.36	23.15	Table 3, p. 11
FAST-LIVO2 / SSIM, higher is better	0.692	0.343	0.729	Table 3, p. 11
FAST-LIVO2 / LPIPS, lower is better	0.560	0.724	0.487	Table 3, p. 11

HI-SLAM2 fails on one RPNG sequence, so its tracking average is N/A; the RPNG rendering comparison in Table 3 excludes that failed sequence and has a different evaluation scope. Rendering is evaluated only on images that no method selected as keyframes, excluding views involved in mapping. For DROID-SLAM, Splat-SLAM, HI-SLAM2, and VIGS-SLAM, the main evaluation excludes final global BA and color refinement, measuring online output rather than the final offline-refined upper bound. Some traditional baselines and DROID-SLAM use EuRoC numbers from prior papers, so not every method was rerun in the same implementation environment. FAST-LIVO2 reference poses come from its LiDAR-fusion system, while self-captured reference poses come from MindCloud LiDAR-visual-inertial fusion; these should not all be called independent motion-capture ground truth.

Ablation Study

Table 6, page 14: EuRoC images are subsampled with stride = 10 while retaining all original IMU readings; Recall@10cm is the proportion of reference poses with translation error below 10 cm. In strided evaluation, failure to produce a valid trajectory receives zero recall rather than evaluation being limited to successful segments.

Config	Average ATE RMSE, cm, lower is better	Average Recall@10cm, %, higher is better
No IMU bias estimation; bias fixed to zero	338.99	0.05
No IMU fusion	88.54	32.50
No inertial-only initialization stage	50.65	68.32
No IMU-based keyframe pose initialization	40.94	64.95
No loop closure	23.78	40.63
One global bias instead of per-keyframe biases	8.83	90.01
Full system	3.39	98.99

Table 7, page 14: only Gaussian updates after loop closure are disabled on FAST-LIVO2, isolating their effect on rendering.

Config	Average PSNR, higher is better	Average SSIM, higher is better	Average LPIPS, lower is better
No loop-closure Gaussian update	22.06	0.677	0.541
Full system	23.15	0.729	0.487

Key Findings

• Bias modeling is substantial: fixing biases to zero yields 338.99 cm versus 3.39 cm for the full system; even global bias estimation gives 8.83 cm, worse than per-keyframe modeling.
• Trajectory and map benefits of loop closure are separately testable: Table 6 removes loop closure entirely, whereas Table 7 removes only Gaussian updates, so these are different ablations.
• The best EuRoC average does not imply superiority on every sequence: on V2_03 in Table 1, VIGS-SLAM obtains 3.27 cm and HI-SLAM2 obtains 1.92 cm.
• The self-captured dataset contains 18 difficult sequences, and Figure 5 on page 13 reports a mean ATE of 0.588 m for VIGS-SLAM; its unit is m, not the cm used above.

Highlights & Insights

• Fusion happens at the solver level. Sharing one state update between vision and inertia is closer to the contribution than merely connecting two modules in a system diagram.
• Map consistency is the second task of loop closure. Keeping keyframe anchors for Gaussians turns trajectory corrections into inexpensive batched geometric updates.
• Initialization and runtime bias modeling are both essential. Table 6 shows that better observations improve robustness only when states are interpretable and initial estimates are usable.

Limitations & Future Work

• The authors acknowledge that mapping does not yet improve tracking: rendering-loss-based pose refinement provides only marginal gains, plausibly because online maps are insufficiently sharp for reliable gradients.
• Robustness is demonstrated in difficult dynamic scenes, but there is no dedicated dynamic-object representation or motion-decomposition module, so correct reconstruction of independently moving objects is not established.
• Some reference trajectories come from other fusion systems, while baseline sources, failure handling, and sensor configurations are not fully identical; comparisons require these conditions.
• Real-time operation is the authors' system claim; the available readable main text contains no hardware configuration and end-to-end timing table, so it does not establish a frame rate on a specific device.
• Future investigations could select rendering regions for tracking by map confidence and test stability under sensor timing offsets or extrinsic errors; these are reader suggestions, not verified results.

Related Work & Insights

• DROID-SLAM / HI-SLAM2: VIGS-SLAM builds on dense correspondences and efficient geometric optimization while adding joint inertial-state estimation; the Gaussian representation is not its sole innovation.
• DBA-Fusion / VINGS-Mono: these also combine learned correspondences and inertia, but VIGS-SLAM emphasizes direct fusion within each normal-equation solve to reduce the linearization gap of cascaded optimization.
• MM3DGS-SLAM: its released implementation primarily targets inputs including LiDAR; VIGS-SLAM only cites its existing non-LiDAR tracking results on UTMM, and rendering scores are not directly comparable.
• Transferable insight: other keyframe-anchored dense representations may benefit from explicitly propagating global state corrections into the map rather than correcting only the displayed trajectory.

Rating

• Novelty: 4/5. The contribution integrates joint optimization, staged initialization, and loop-consistent mapping rather than introducing a new Gaussian representation.
• Experimental Thoroughness: 4/5. Evaluation covers 5 datasets with tracking and rendering ablations, but supplementary implementation details were not available for this reading.
• Writing Quality: 4/5. The narrative and ablation mapping are clear; damaged extracted equations and ambiguous symbol descriptions limit equation-level verification.
• Value: 4/5. The system offers useful guidance for online dense SLAM under low frame rates and degraded vision, while deployment efficiency requires further verification.