Skip to content

UniSim-SLAM: Feed-Forward SLAM with Unified Sim(3) Optimization

Conference: ECCV 2026
Paper: ECCV Official Page
Project: https://vision3d-lab.github.io/unisim-slam/
Area: 3D Vision
Keywords: Visual SLAM, Feed-Forward SLAM, Sim(3) Factor Graph, Scale Drift, Multi-View Submap

TL;DR

Addressing the fundamental trade-off between two-view low-latency tracking and multi-view constraint richness in feed-forward SLAM, UniSim-SLAM formulates a unified Sim(3) multi-level factor graph that jointly optimizes global keyframe and submap poses, reducing trajectory error by 38.5% on TUM RGB-D and 45.9% on 7-Scenes under uncalibrated settings.

Background & Motivation

Feed-forward visual SLAM based on 3D geometric foundation models (such as DUSt3R and VGGT) is reshaping the visual SLAM paradigm by bypassing handcrafted feature matching and expensive photometric bundle adjustment. These models regress dense geometry and relative camera poses directly from uncalibrated image sets in a single pass, providing remarkable robustness across challenging visual conditions. However, feed-forward geometric estimates are strongly conditioned on the input view configuration: even for identical images, predicted reconstruction scales and relative poses fluctuate depending on the specific co-visible view set. Chaining these configuration-dependent local reconstructions sequentially across long trajectories inevitably accumulates geometric inconsistencies and severe scale drift.

This dilemma manifests as a fundamental trade-off between tracking latency and geometric constraint richness in existing feed-forward architectures. Two-view feed-forward SLAM (such as ViSTA-SLAM) is computationally lightweight and operates immediately upon receiving a new frame, maintaining continuous temporal connectivity with minimal latency; however, relying solely on pairwise geometric cues leaves it vulnerable to drift accumulation over time. Conversely, multi-view submap SLAM (such as VGGT-SLAM) captures rich multi-view geometric constraints and higher local reconstruction fidelity, but requires accumulating a fixed window of keyframes before inference, precluding per-frame online tracking. Furthermore, when consecutive submaps lack sufficient overlap, global geometric corrections cannot propagate across the trajectory. Prior systems adopt either two-view or multi-view inference in isolation, failing to reconcile low-latency tracking with long-term global consistency.

Naively combining two-view and multi-view predictions introduces significant challenges because both regimes operate in heterogeneous local coordinate systems with independent scale references and arbitrary coordinate origins. Core idea: interpret two-view temporal constraints and multi-view submap geometries as complementary constraints defined in heterogeneous local coordinate frames, and formulate a unified multi-level factor graph on the Sim(3) manifold to jointly optimize global keyframe and submap poses through temporal, bridge, and inter-submap edges.

Method

Overall Architecture

UniSim-SLAM draws on the classical SLAM principle of pairing a lightweight frontend for low-latency odometry with a periodic backend for globally consistent refinement. The frontend continuously samples keyframes and feeds consecutive pairs into a feed-forward model to obtain relative Sim(3) transformations and dense depth maps, incrementally initializing a connected global trajectory backbone with zero tracking lag. In parallel, the backend accumulates sliding keyframe windows and asynchronously executes multi-view feed-forward inference to produce dense local geometries and submap-local poses. Crucially, all heterogeneous predictions are embedded into a unified multi-level factor graph on the Sim(3) manifold, comprising global keyframe pose nodes and submap pose nodes. The entire graph is jointly refined using the Levenberg-Marquardt algorithm with a robust Huber loss over Lie algebra \(\mathfrak{sim}(3)\) residuals.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Incoming RGB Stream<br/>Keyframe Stride Sampling"] --> B["Two-View Tracking & Trajectory Initialization<br/>Instant pairwise relative pose & depth estimation"]
    B --> C["View-to-Submap Bridge & Scale Anchoring<br/>Align local submaps to global poses via depth statistics"]
    C --> D["Inter-Submap Tie & Direct Scale Alignment<br/>Tie constraints on shared views & cross-submap scale edges"]
    D --> E["Unified Sim(3) Multi-Level Factor Graph Optimization<br/>Joint Levenberg-Marquardt solver on Lie algebra sim(3)"]
    E --> F["Scale-Consistent Global Trajectory & Dense 3D Point Map"]

Key Designs

1. Two-View Tracking & Trajectory Initialization: Ensuring Low-Latency Tracking and Global Graph Connectivity Pure multi-view feed-forward systems suffer from latency bottlenecks due to frame accumulation windows and risk complete graph disconnection when submaps do not overlap. To eliminate this issue, the frontend takes temporally consecutive keyframe pairs \((I_i, I_j)\) with \(j=i+1\) and queries a lightweight feed-forward model \(f_{\text{2v}}\) to predict dense depth maps \(\hat{D}_i^{\text{2v}}, \hat{D}_j^{\text{2v}}\) and relative transformations \(\hat{T}_{ij}^{\text{2v}} \in \mathrm{Sim}(3)\) (with scale initialized to 1). Setting the initial keyframe \(I_0\) as the global origin, global poses are initialized online via recursive composition \(T_j = T_i \hat{T}_{ij}^{\text{2v}}\). These sequential transformations form the temporal edge set \(\mathcal{E}^{\text{temp}}\) in the factor graph, yielding residuals in the Lie algebra \(\mathfrak{sim}(3)\): $\(\mathbf{e}_{ij}^{\text{2v}} = \log \left( (\hat{T}_{ij}^{\text{2v}})^{-1} (T_i^{-1} T_j) \right)\)$ These temporal edges establish a densely connected baseline backbone. Even when multi-view submaps are sparse or separated without geometric overlap, higher-level geometric and scale corrections can propagate smoothly across the entire trajectory along this temporal backbone.

2. View-to-Submap Bridge & Scale Anchoring: Aligning Submap Frames and Stabilizing Relative Scale Once a window \(\mathcal{W}_m\) of consecutive keyframes accumulates in the backend, the multi-view foundation model \(f_{\text{mv}}\) predicts submap-local poses and depth maps \(\{\hat{D}_{mi}^{\text{mv}}, \hat{T}_{mi}^{\text{mv}}\}_{i \in \mathcal{W}_m}\) relative to the submap's central keyframe origin. To integrate these local coordinates into the global trajectory, UniSim-SLAM assigns a global submap pose variable \(S_m \in \mathrm{Sim}(3)\) to each submap and constructs view-to-submap edges \(\mathcal{E}^{\text{v2s}}\). Each edge incorporates two complementary residuals: a geometric bridge residual \(\mathbf{e}_{mi}^{\text{br}} = \log \left( (\hat{T}_{mi}^{\text{mv}})^{-1} S_m^{-1} T_i \right)\) enforcing alignment between the global pose \(T_i\) and the submap-transformed pose \(S_m \hat{T}_{mi}^{\text{mv}}\), and a depth-statistic scale anchoring residual: $\(\mathbf{e}_{mi}^{\text{anch}} = \log s_i - \log s_m - \log \left( \mathrm{median}\left(\frac{\hat{D}_{mi}^{\text{mv}}}{\hat{D}_i^{\text{2v}}}\right) \right)\)$ where \(s_i\) and \(s_m\) denote the scale components of \(T_i\) and \(S_m\), respectively. Because pose alignment alone cannot eliminate relative scale drift between the global trajectory and submap reference frames, introducing per-view median depth ratios explicitly anchors and stabilizes the relative scale across heterogeneous inference regimes.

3. Inter-Submap Tie & Direct Scale Alignment: Preventing Drift and Eliminating Compensating Degeneracies While view-to-submap bridge edges indirectly couple neighboring submaps through shared keyframes, nonlinear graph optimization can settle into degenerate solutions where submap misalignments are satisfied by bending or distorting intermediate keyframe poses. To enforce rigid consistency directly between submap coordinate systems, UniSim-SLAM introduces direct submap-to-submap edges \(\mathcal{E}^{\text{s2s}}\) for submap pairs with overlapping keyframes \(\mathcal{V}_{mn} = \mathcal{W}_m \cap \mathcal{W}_n \neq \emptyset\). For every shared view \(i \in \mathcal{V}_{mn}\), a pose tie residual enforces identical global coordinates: \(\mathbf{e}_{mn,i}^{\text{tie}} = \log \left( (S_m \hat{T}_{mi}^{\text{mv}})^{-1} (S_n \hat{T}_{ni}^{\text{mv}}) \right)\). In parallel, multi-view depth predictions across shared views are aggregated into a robust relative scale residual: $\(\mathbf{e}_{mn}^{\text{sc}} = \log s_n - \log s_m - \log \hat{s}_{mn}, \quad \text{where} \ \hat{s}_{mn} = \mathrm{median}_{i \in \mathcal{V}_{mn}} \left( \mathrm{median}\left(\frac{\hat{D}_{mi}^{\text{mv}}}{\hat{D}_{ni}^{\text{mv}}}\right) \right)\)$ This direct constraint couples submap origins and relative scales directly, preventing scale drift between independently inferred submaps and eliminating degenerate parameter trade-offs.

4. Unified Sim(3) Multi-Level Factor Graph Optimization: Joint Solution on the Lie Manifold Prior to optimization, submap poses \(S_m\) are initialized as \(S_m^{(0)}\) using the two-view global pose of the origin frame and the relative scale ratio \(s_{mi}^{\text{rel}} = \mathrm{median}(\hat{D}_{mi}^{\text{mv}} / \hat{D}_i^{\text{2v}})\), scaling translation accordingly. The system then minimizes the combined nonlinear least-squares objective with Huber loss \(\rho(\cdot)\) over all edge types: $\(\min_{\{T_i\}, \{S_m\}} \sum_{(i,j) \in \mathcal{E}^{\text{temp}}} \rho(\|\mathbf{e}_{ij}^{\text{2v}}\|) + \sum_{(m,i) \in \mathcal{E}^{\text{v2s}}} \left( \rho(\|\mathbf{e}_{mi}^{\text{br}}\|) + \rho(\|\mathbf{e}_{mi}^{\text{anch}}\|) \right) + \sum_{(m,n) \in \mathcal{E}^{\text{s2s}}} \left( \sum_{i \in \mathcal{V}_{mn}} \rho(\|\mathbf{e}_{mn,i}^{\text{tie}}\|) + \rho(\|\mathbf{e}_{mn}^{\text{sc}}\|) \right)\)$ All variables reside on the \(\mathrm{Sim}(3)\) manifold and are iteratively refined via Levenberg-Marquardt using \(\mathfrak{sim}(3)\) Lie algebra perturbations. This unified formulation eliminates isolated stage boundaries, facilitating bidirectional constraint propagation between fast temporal odometry and rigid multi-view geometry. A visual loop closure module additionally constructs joint loop submaps upon image retrieval and geometric verification, eliminating long-horizon accumulated drift.

A Worked Example

Consider a sequence tracking 7 keyframes \((I_1 \sim I_7)\): 1. Frontend Tracking: Upon frame arrivals, the two-view model predicts relative transformations \(\hat{T}_{12}^{\text{2v}}, \dots, \hat{T}_{67}^{\text{2v}}\), instantly emitting initialized poses \(T_1 \sim T_7\) and linking them with 6 temporal edges \(\mathbf{e}^{\text{2v}}\). 2. Backend Submap Construction: With window \(w=3, \phi=1\) (default \(w=16, \phi=2\)), the backend spawns Submap 1 (frames 1-3), Submap 2 (frames 3-5), and Submap 3 (frames 5-7). Submap 1 yields local poses \(\hat{T}_{11}^{\text{mv}}, \hat{T}_{12}^{\text{mv}}, \hat{T}_{13}^{\text{mv}}\) and introduces global submap variable \(S_1\). 3. Cross-Level Edge Formulation: Bridge edges \(\mathbf{e}_{mi}^{\text{br}}\) and scale anchoring edges \(\mathbf{e}_{mi}^{\text{anch}}\) link each keyframe to its host submaps (frame 3 connects to both \(S_1\) and \(S_2\)). At shared frame 3, a tie edge \(\mathbf{e}_{12,3}^{\text{tie}}\) and depth ratio scale edge \(\mathbf{e}_{12}^{\text{sc}}\) bind \(S_1\) and \(S_2\) directly. 4. Joint Optimization: The solver optimizes all \(\{T_1 \dots T_7\}\) and \(\{S_1, S_2, S_3\}\) jointly on \(\mathrm{Sim}(3)\). Multi-view curvature and scale corrections lock overlapping frames 3 and 5 while propagating along temporal edges to intermediate views, producing a globally scale-consistent trajectory.

Key Experimental Results

Main Results

Evaluation is conducted on TUM RGB-D and 7-Scenes benchmarks using Absolute Trajectory Error (ATE RMSE [m]) after Sim(3) alignment. All primary comparisons are under the uncalibrated (Uncalib.) setting alongside calibrated (Calib.) methods.

Table 1: Quantitative trajectory results on TUM RGB-D (ATE RMSE [m] ↓) | Method | Calibration | 360 | desk | desk2 | floor | plant | room | rpy | teddy | xyz | Avg | |---|---|---|---|---|---|---|---|---|---|---|---| | DROID-SLAM | Calib. | 0.111 | 0.018 | 0.042 | 0.021 | 0.016 | 0.049 | 0.026 | 0.048 | 0.012 | 0.038 | | MASt3R-SLAM | Calib. | 0.049 | 0.016 | 0.024 | 0.025 | 0.020 | 0.061 | 0.027 | 0.041 | 0.009 | 0.030 | | CUT3R | Uncalib. | 0.174 | 0.592 | 0.546 | 0.662 | 0.467 | 0.911 | 0.051 | 0.845 | 0.129 | 0.486 | | SLAM3R | Uncalib. | 0.211 | 0.861 | 0.967 | 0.790 | 0.755 | 1.013 | 0.063 | 0.986 | 0.185 | 0.648 | | MASt3R-SLAM | Uncalib. | 0.070 | 0.032 | 0.055 | 0.056 | 0.035 | 0.118 | 0.041 | 0.116 | 0.020 | 0.060 | | VGGT-SLAM | Uncalib. | 0.063 | 0.031 | 0.048 | 0.152 | 0.023 | 0.133 | 0.038 | 0.039 | 0.020 | 0.061 | | ViSTA-SLAM | Uncalib. | 0.104 | 0.030 | 0.030 | 0.070 | 0.052 | 0.067 | 0.023 | 0.080 | 0.015 | 0.052 | | UniSim-SLAM (Ours) | Uncalib. | 0.067 | 0.018 | 0.022 | 0.034 | 0.029 | 0.056 | 0.021 | 0.031 | 0.013 | 0.032* |

Table 2: Quantitative trajectory results on 7-Scenes (ATE RMSE [m] ↓) | Method | Calibration | chess | fire | heads | office | pumpkin | kitchen | stairs | Avg | |---|---|---|---|---|---|---|---|---|---| | DROID-SLAM | Calib. | 0.018 | 0.027 | 0.021 | 0.041 | 0.025 | 0.016 | 0.017 | 0.024 | | MASt3R-SLAM | Calib. | 0.082 | 0.030 | 0.024 | 0.052 | 0.050 | 0.044 | 0.027 | 0.044 | | CUT3R | Uncalib. | 0.514 | 0.110 | 0.197 | 0.430 | 0.346 | 0.202 | 0.385 | 0.312 | | SLAM3R | Uncalib. | 0.131 | 0.044 | 0.040 | 0.058 | 0.100 | 0.064 | 0.116 | 0.079 | | MASt3R-SLAM | Uncalib. | 0.090 | 0.058 | 0.039 | 0.072 | 0.084 | 0.062 | 0.071 | 0.068 | | VGGT-SLAM | Uncalib. | 0.039 | 0.024 | 0.041 | 0.032 | 0.050 | 0.034 | 0.042 | 0.037 | | ViSTA-SLAM | Uncalib. | 0.075 | 0.035 | 0.030 | 0.064 | 0.065 | 0.041 | 0.036 | 0.049 | | UniSim-SLAM (Ours) | Uncalib. | 0.017 | 0.018 | 0.026 | 0.024 | 0.022 | 0.016 | 0.019 | 0.020 |

Table 3: Dense 3D reconstruction quality on 7-Scenes | Method | Calibration | Accuracy [m] ↓ | Completion [m] ↓ | Chamfer Distance [m] ↓ | |---|---|---|---|---| | DROID-SLAM | Calib. | 0.111 | 0.049 | 0.080 | | MASt3R-SLAM | Calib. | 0.064 | 0.068 | 0.066 | | Spann3R @5 | Uncalib. | 0.095 | 0.041 | 0.068 | | CUT3R | Uncalib. | 0.107 | 0.059 | 0.083 | | SLAM3R | Uncalib. | 0.069 | 0.153 | 0.111 | | MASt3R-SLAM | Uncalib. | 0.054 | 0.048 | 0.051 | | VGGT-SLAM | Uncalib. | 0.039 | 0.051 | 0.045 | | ViSTA-SLAM | Uncalib. | 0.041 | 0.056 | 0.049 | | UniSim-SLAM (Ours) | Uncalib. | 0.035 | 0.046 | 0.041 |

Ablation Study

Table 4: Graph constraint ablations on 7-Scenes (ATE RMSE [m] ↓) | Configuration | \(\phi = 0\) (Non-overlapping) | \(\phi = 2\) (Default overlap) | Note | |---|---|---|---| | Ours (full) | 0.032 | 0.020 | Full factor graph with loop closure | | w/o Backend | 0.124 | 0.124 | Pure two-view frontend odometry without backend | | w/o LC | 0.061 | 0.037 | Disables loop closure; maintains strong local consistency | | w/o \(\mathbf{e}^{\text{2v}}\) | 0.101 | 0.021 | Removes two-view temporal edges; breaks graph at \(\phi=0\) | | w/o \(\mathbf{e}^{\text{anch}}\) | 0.083 | 0.020 | Removes depth scale anchoring; suffers scale drift without overlap | | w/o \(\mathbf{e}^{\text{br}}\) | 0.116 | 0.063 | Cuts submap-to-frame bridge; severe global trajectory drift | | w/o \(\mathbf{e}^{\text{tie}}\) | 0.040 | 0.027 | Removes pose tie constraints; allows degenerate view shifts | | w/o \(\mathbf{e}^{\text{sc}}\) | 0.048 | 0.031 | Removes direct inter-submap scale edges; scale drift across submaps |

Table 5: Tracking latency and accuracy trade-off on 7-Scenes | Method | Frontend Model | Latency [ms] ↓ | ATE RMSE [m] ↓ | |---|---|---|---| | MASt3R-SLAM | MASt3R | 90 | 0.068 | | VGGT-SLAM | VGGT | 3410 | 0.037 | | ViSTA-SLAM | STA | 35 | 0.049 | | Ours + STA | STA | 35 | 0.027 | | Ours (full) | VGGT | 197 | 0.020 |

Key Findings

  • Two-view temporal edges provide indispensable connectivity when submaps do not overlap: Under \(\phi=0\), removing \(\mathbf{e}^{\text{2v}}\) inflates ATE error from 0.032m to 0.101m (over 3x degradation), proving that temporal edges serve as the backbone for propagating backend corrections.
  • View-to-submap bridge edges are the primary driver of trajectory correction: Removing \(\mathbf{e}^{\text{br}}\) under \(\phi=2\) causes ATE to deteriorate from 0.020m to 0.063m, the largest single drop among graph edges, confirming that direct coupling between submap predictions and keyframe poses is vital.
  • Scale anchoring and tie constraints eliminate scale drift in planar scenes: On TUM's challenging floor sequence, where lack of visual depth variation causes standard feed-forward models to drift severely (VGGT-SLAM at 0.152m), UniSim-SLAM's joint Sim(3) constraints compress error to 0.034m.
  • Heterogeneous frontend/backend integration generalizes robustly: Replacing the frontend backbone with the ultra-fast STA model (Ours + STA) preserves 35ms tracking latency while achieving an ATE of 0.027m, outperforming both ViSTA-SLAM (0.049m) and VGGT-SLAM (0.037m) despite cross-model Sim(3) domain differences.

Highlights & Insights

  • Unified Sim(3) formulation for configuration-dependent predictions: Rather than treating view-configuration inconsistencies as noise to be smoothed out, UniSim-SLAM recognizes that feed-forward predictions across different view sets represent distinct local coordinate systems with independent scales, elegantly unifying them via a multi-level Sim(3) Lie group factor graph.
  • Median depth statistics as robust scale anchors: Using pixel-aligned median depth ratios across shared keyframes provides an outlier-resilient scale bridge between heterogeneous predictions, injecting metric scale continuity into uncalibrated monocular streams without explicit feature matching.
  • Flexible decoupled inference architecture: Demonstrating that frontend and backend can utilize entirely different foundation models opens up practical deployment avenues for edge devices, pairing an ultra-lightweight frontend odometry network with an intermittent cloud/server-based multi-view backend.

Limitations & Future Work

  • Author-admitted limitations: The default system using VGGT in the frontend exhibits 197ms tracking latency, which is slower than specialized real-time frontends like STA (35ms). Furthermore, the backend requires GPU resources to process multi-view transformer inference windows.
  • Potential unaddressed constraints: Median depth scale anchoring relies on indoor environments having well-behaved depth distributions. In open outdoor environments with distant skies or severe dynamic foreground occlusions, median depth statistics can degrade and cause scale instability.
  • Future directions: Integrating semantic or dynamic object segmentation masks to filter static background pixels before calculating scale statistics, and formulating incremental Sim(3) solvers (e.g., via iSAM2) to reduce backend optimization overhead.
  • vs. VGGT-SLAM [22]: VGGT-SLAM optimizes 15-DoF projective transformations on the SL(4) manifold using pure multi-view VGGT submaps, incurring 3410ms latency and failing to propagate corrections across non-overlapping submaps. UniSim-SLAM incorporates a two-view frontend and restricts optimization to the geometrically grounded 7-DoF Sim(3) manifold, lowering 7-Scenes ATE by 45.9% with over 17x faster tracking.
  • vs. ViSTA-SLAM [46]: ViSTA-SLAM relies strictly on symmetric two-view associations and pose-graph optimization, lacking multi-view dense geometric rigidity and drifting in planar scenes (0.052m TUM ATE). UniSim-SLAM integrates multi-view submaps to achieve 0.032m ATE while retaining low frontend latency.
  • vs. MASt3R-SLAM [26]: While MASt3R-SLAM achieves high accuracy with known calibration (0.030m TUM ATE), its performance degrades to 0.060m in uncalibrated settings. UniSim-SLAM explicitly handles scale and coordinate heterogeneity under uncalibrated cameras, reaching 0.032m without calibration.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Formulates the first unified multi-level Sim(3) factor graph resolving the fundamental trade-off between two-view odometry and multi-view submaps in feed-forward SLAM.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive benchmarks across TUM RGB-D and 7-Scenes covering trajectory accuracy, 3D point cloud metrics, comprehensive edge ablations, and heterogeneous frontend latency trade-offs.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear mathematical formulation on the Sim(3) manifold with cohesive motivation and intuitive diagrammatic explanations.
  • Value: ⭐⭐⭐⭐⭐ Provides a blueprint for deploying foundation-model-based spatial perception systems in robotics and embodied AI under uncalibrated settings.