SLAM-Former: Putting SLAM into One Transformer

Conference: ECCV2026
Paper: ECCV Paper
Area: 3D Vision
Keywords: dense monocular SLAM, global consistency, frontend-backend cooperation, KV cache, token pruning

TL;DR

SLAM-Former alternates causal incremental mapping and global map refinement with a shared-weight Transformer, feeding refined KV caches back to the frontend to achieve a mean TUM RGB-D ATE of 0.039 m and a 7-Scenes reconstruction Chamfer distance of 0.027 m without camera calibration.

Background & Motivation

Dense monocular simultaneous localization and mapping (SLAM) must estimate the camera trajectory while placing surfaces observed at different times, such as walls and desks, at consistent 3D locations. Geometry foundation models such as DUSt3R, MASt3R, and VGGT have improved direct geometry prediction from images, but assembling local predictions into a persistent map remains difficult. MASt3R-SLAM and VGGT-SLAM reconstruct geometry from image pairs or submaps and connect these local results through external optimization; local deformation, scale differences, and cross-frame inconsistencies can still produce duplicated surfaces.

Another approach processes images sequentially and stores history in recurrent states or KV caches. Methods such as StreamVGGT avoid recomputing all inputs whenever a frame arrives, but typically do not re-estimate previously generated maps. When early geometry is biased, subsequent frames continue to depend on that biased state, so continuous output does not necessarily imply an ability to correct the past using new evidence. A traditional SLAM backend serves precisely this purpose, but the authors seek to incorporate it into the same learnable model instead of adding separate loop detection and graph optimization modules.

SLAM-Former therefore retains the functional distinction between frontend and backend while sharing Transformer weights and exchanging map tokens and attention caches. The frontend supplies geometric initial estimates with sequential context; the backend refines historical representations and changes the memory on which subsequent frontend predictions depend. Core Idea: write global refinements back into the frontend KV cache, rather than updating only the map output, so that subsequent incremental predictions continue from a corrected history.

Method

Overall Architecture

The input is a sequence of monocular RGB images; the outputs are local pointmaps, confidence estimates, and camera poses for keyframes, from which a dense map is assembled. Images are encoded into patch tokens with additional register tokens; the shared backbone aggregates intra-frame and inter-frame information, and task-specific heads decode geometry and poses. One Transformer means that the frontend and backend share a model, not that the network is called only once per frame or that keyframe selection and periodic scheduling disappear.

The system performs Causal Frontend Mapping followed by periodic Global Backend Cache Feedback; Diversity-Based KV Pruning reduces backend attention costs and subsequent frontend cache access. Three attention modes match these operating states during training, and supervision is not an inference input.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Sequential RGB images<br/>Image encoding"] --> B["Causal Frontend Mapping"]
    B -->|Map tokens every 10 keyframes| C["Global Backend Cache Feedback"]
    C --> D["Diversity-Based KV Pruning"]
    D -->|Refresh historical KV| B
    B --> E["Task-head decoding<br/>Local pointmaps and poses"]
    D --> E
    S["Training supervision<br/>Depth, pointmaps, cameras"] -.-> B
    S -.-> C

Pruning occurs within backend attention and supplies the corresponding KV to the subsequent frontend; the arrows represent cooperation and cache flow, not completion of all backend computation before pruning begins. Full KV is the default evaluation setting, and enabling pruning does not remove query positions that require geometric outputs.

Key Designs

1. Causal Frontend Mapping

For each incoming frame, the system pairs it with the most recent keyframe and uses full attention in the same Transformer to predict poses, then decides whether to retain it based on relative motion. The implementation uses a translation threshold of 0.1; this step does not access the full historical KV cache, avoiding full mapping costs for numerous redundant neighboring images. The first two keyframes are initialized jointly; each subsequent keyframe reads the inter-frame attention KV of previous keyframes, generates its own map tokens and new KV, and appends them to the state. This operation depends only on observed frames and is therefore causal, but it cannot by itself revise past errors.

Map tokens and KV caches serve different roles: map tokens are implicit geometric representations produced by the backbone that can be decoded by task heads or refined by the backend; KV caches store attention keys and values at each layer so that new frames can reuse historical information. The frontend does not need to re-encode and process every old image at every step, making it better suited to streaming inputs than repeatedly running a full-sequence model. The model predicts pointmaps in each frame's local coordinate system instead of requiring every point to use a fixed world coordinate system; poses, backend interactions, and training-time geometric constraints establish cross-frame consistency together.

2. Global Backend Cache Feedback

Every 10 keyframes, the backend takes the accumulated map tokens as input, applies full attention for bidirectional interaction among historical frames, and re-estimates the map representation in one forward pass. Unlike causal attention restricted to new frames, this lets old frames benefit from later views; the authors use this mechanism to perform functions traditionally handled by loop detection and global optimization without explicitly solving a pose graph. It is a learned global refinement mechanism, not a guarantee that attention always identifies correct loops or satisfies every geometric constraint.

Crucially, the KV produced by the backend replaces the corresponding historical KV stored by the frontend. Updating only the output point cloud while retaining stale caches would leave new frames conditioned on incorrect history; cache feedback puts correction directly into subsequent inference. Conversely, the backend receives map tokens built sequentially by the frontend rather than unrelated raw images, so its inputs already contain incremental localization and mapping context. To let the same weights process both image tokens and map tokens, training explicitly covers frontend-only, mixed frontend-backend, and backend-only states instead of combining attention regimes only at test time.

3. Diversity-Based KV Pruning

An increasing number of keyframes expands both cache capacity requirements and inter-frame attention costs, but simply deleting patch tokens can also eliminate their dense geometric outputs. Inspired by DivPrune, the authors use cosine distance among non-register patch tokens within each keyframe and greedily select a subset with a large minimum pairwise distance. The objective is to retain representatives of different content rather than spend the budget on similar regions; precomputing pairwise distances is quadratic in the number of patches per frame.

Once the retained subset is selected, only keys and values are pruned, while queries remain intact; the source clearly specifies:

\[ K'=K[S^*],\qquad V'=V[S^*],\qquad Q'=Q. \]

Every original query position can therefore still aggregate retained context and predict geometry: pruning reduces retrievable memory rather than the map positions that require predictions. For \(n\) keyframes and a fixed retention ratio \(\gamma\), the paper describes the dominant attention costs as \(O(\gamma n)\) for the frontend and \(O(\gamma n^2)\) for the backend; this reduces constants but does not make fixed-ratio backend processing linear. When the cache pool grows logarithmically or with the square root of sequence length, the corresponding backend costs are \(O(n\log n)\) or \(O(n^{1.5})\); these attention-scaling analyses are not measured speedup factors for the entire system.

A Worked Example

Consider a camera moving through a room and later seeing a previously observed wall; this illustrates the mechanism rather than introducing an additional experimental sequence. The first two keyframes initialize map tokens and KV; when keyframe 3 arrives, it first passes the paired-frame pose check and then uses existing KV to generate a new local pointmap and pose. When keyframe 10 triggers the backend, all accumulated map tokens are jointly refined, allowing the earlier wall estimate to be influenced by new views.

With a 25% KV retention ratio, every frame retains all queries but uses only the selected quarter of patch KV for the corresponding attention operations. After the backend refreshes the historical frontend cache, keyframe 11 depends on corrected history rather than the original drifting state. This distinguishes the system from optimization performed only after a video ends: backend improvements enter the next round of online prediction.

Loss & Training

Each training iteration sequentially executes three modes with shared weights: Mode 1 uses causal attention to simulate frontend KV accumulation; Mode 2 applies full attention to historical map tokens and causal attention to new images, simulating mixed backend and new frontend KV; Mode 3 applies full attention to map tokens to train backend-only refinement. Map tokens in Mode 3 can originate from different runs or cache states, exposing the backend to frontend representations with different errors. Mode 2 is particularly relevant because corrected history combined with not-yet-refined new frames is the operating state between periodic backend updates.

Each mode supervises depth, pointmaps, and cameras; the clearly readable overall objectives are:

\[ L=L_{\mathrm{depth}}+L_{\mathrm{pmap}}+\lambda L_{\mathrm{cam}},\qquad L_{\mathrm{all}}=L_1+L_2+\beta L_3. \]

Depth is taken from the depth component of each local pointmap; depth and pointmap supervision include confidence weighting and spatial gradient constraints. For pointmap supervision, predicted poses first align local pointmaps to the first frame; camera supervision uses scale-adjusted relative poses and a Huber loss, with scale estimation following Pi3. Operators are missing from Equations (4) through (6) in the cached text, so their exact component formulas are not reconstructed here, nor are the signs and coefficients of confidence terms guessed.

The model is initialized from Pi3 weights, contains 36 layers of frame and global attention in total, and freezes the image encoder and camera head. Training lasts 10 epochs with a batch size of 32 and only 12 frames per batch; testing processes longer sequences. The optimizer is AdamW with a learning rate of \(10^{-5}\) and cosine scheduling, using \(\lambda=100\), \(\beta=10\), and a backend interval of \(T=10\). Training data include ARKitScenes, ScanNet, ScanNet++, HyperSim, BlendedMVS, MegaDepth, and MVS-Synth; training takes 11 hours on 32 A100 GPUs, and evaluation uses a single RTX 4090.

Key Experimental Results

Main Results

ATE denotes absolute trajectory error, reported here as RMSE; Acc. measures predicted-to-ground-truth surface distance, Complet. measures ground-truth-to-predicted surface coverage distance, and Chamfer aggregates bidirectional distances, all lower-is-better. The table compares only methods without calibration, with all values in m; main experiments use full KV by default.

Dataset and metric	SLAM-Former	Comparison method	Comparison value	Source
TUM RGB-D mean ATE	0.039	ViSTA-SLAM	0.052	Table 1
TUM RGB-D mean ATE	0.039	VGGT-SLAM	0.053	Table 1
7-Scenes mean ATE	0.042	MASt3R-SLAM	0.066	Table 2
Replica mean ATE	0.030	EC3R-SLAM	0.041	Table 3
7-Scenes Acc.	0.017	EC3R-SLAM	0.025	Table 4
7-Scenes Complet.	0.037	EC3R-SLAM	0.054	Table 4
7-Scenes Chamfer	0.027	EC3R-SLAM	0.040	Table 4

The reduction in 7-Scenes Acc. from 0.025 to 0.017 is 32.0% relative to the comparison value; the paper's statement that competing error is approximately 47% higher must not be restated as a 47% reduction by this method. Tracking is not better than every calibrated method: calibrated MASt3R-SLAM reaches 0.030 in TUM Table 1, below this method's 0.039; the Replica discussion also explicitly acknowledges worse results than calibrated DROID-SLAM. ViSTA-SLAM's ATE is 0.056 in Table 4 but 0.055 in Table 2; this discrepancy is not silently reconciled.

Ablation Study

Table 6 directly compares performance with and without the backend, reporting TUM RGB-D ATE RMSE in m.

Config	Mean	room	floor	desk	Note
Frontend only	0.134	0.547	0.264	0.041	Causal incremental prediction only
Frontend + Backend	0.039	0.082	0.079	0.018	Global refinement with KV feedback

Mean ATE decreases by approximately 70.9%, with a particularly pronounced improvement on room, consistent with the motivation for correcting accumulated drift in long sequences. However, this ablation introduces backend refinement and KV feedback together, so it does not quantify their independent contributions or separately establish the benefit of Mode 2.

Table 7 further analyzes cache compression on 7-Scenes, again reporting all errors in m.

KV retention setting	ATE	Acc.	Complet.	Chamfer
100%	0.042	0.017	0.036	0.026
50%	0.041	0.017	0.036	0.027
25%	0.042	0.018	0.037	0.027
12.5%	0.042	0.020	0.038	0.029
6.25%	0.055	0.026	0.042	0.034
log KV pool	0.042	0.017	0.037	0.027
sqrt KV pool	0.041	0.017	0.037	0.027

The 100% row in Table 7 reports Complet. 0.036 and Chamfer 0.026, slightly different from 0.037 and 0.027 in Table 4; each table's original values are retained rather than treated as an identical baseline report.

Key Findings

• The backend improves not only final surfaces but also trajectory accuracy; Table 6 provides the most direct quantitative evidence for frontend-backend cooperation.
• At 12.5% KV retention, ATE remains 0.042, but Acc. rises from 0.017 to 0.020, so nearly unchanged localization does not imply unchanged geometry on every metric.
• At 6.25%, several metrics deteriorate substantially; logarithmic and square-root cache pools remain closer to full-KV results, showing that fixed-ratio retention is not the only option.

Highlights & Insights

• Backend refinements are written into attention memory rather than applied only to final predictions. This lets map optimization influence future perception, providing a form of cooperation closer to SLAM than one-shot offline reconstruction.
• Three attention configurations train the same model on caches from different sources during periodic execution. The transferable idea is to explicitly train mixed states at module handoffs instead of assuming that separately trained functions will be compatible.
• Separating complete queries from sparse KV preserves dense output positions. This compression strategy suits visual tasks in which outputs must remain dense but retrievable history is highly redundant.

Limitations & Future Work

• The authors explicitly identify latency and memory scaling issues on long sequences, and the endpoints of Figure 7's curves correspond to memory limits. Fixed-ratio pruning retains quadratic backend attention and does not establish indefinitely long real-time SLAM.
• The three main quantitative benchmarks are indoor datasets; the evidence does not establish reliability in large outdoor environments, scenes with strongly dynamic objects, or severe occlusion. This is a limitation of evaluation coverage.
• Table 6 does not disentangle cache feedback, backend re-estimation, and three-mode training, and independent sensitivity analyses of the backend interval and keyframe threshold are also missing. Follow-up experiments should control these factors separately under matched budgets.
• Full attention performing loop closure is the authors' mechanistic interpretation; the paper does not separately report loop detection accuracy or failure recovery. Deployment-oriented evaluation should examine false loops, prolonged tracking loss, and recovery in dynamic scenes.

Related Work & Insights

• vs MASt3R-SLAM / VGGT-SLAM: These methods use geometry foundation models for image-pair or submap predictions and connect them through external geometric optimization; this method directly updates global map tokens with a shared Transformer and feeds the result into the subsequent frontend. Traditional methods can still be more accurate in some calibrated tracking settings.
• vs StreamVGGT / CUT3R: Streaming models use history to produce new results but lack this method's periodic global re-estimation and cache refresh mechanism. The key addition is allowing new evidence to revise history.
• vs Pi3: Pi3 supplies pretrained geometric capabilities, while this method adds a sequential frontend and backend cooperation. Figure 6 provides a qualitative comparison, but the prose names ofkt0 while the caption names ofkt1; this inconsistent sequence identifier prevents a precise sequence-specific quantitative conclusion.
• Potential extension: Adapting backend scheduling and cache budgets to map inconsistency might be more effective than updating every 10 keyframes. This is a research direction proposed by this note, not a result validated in the paper.

Rating

• Novelty: 4/5. The shared-Transformer frontend-backend loop and KV feedback provide a concrete system contribution, built on established geometry foundation models and attention mechanisms.
• Experimental Thoroughness: 4/5. Three tracking benchmarks, reconstruction evaluations, and pruning analyses are covered, but finer-grained cooperation ablations and long-term outdoor validation are missing.
• Writing Quality: 4/5. Architecture and training modes are clearly connected, while a few cross-table values and figure-prose sequence identifiers require careful reading.
• Value: 4/5. The method provides a concrete, reusable system design for moving from streaming geometry prediction to neural SLAM with retrospective correction.