Geometric Context Transformer for Streaming 3D Reconstruction

Conference: ECCV 2026
Paper: ECCV Official
Area: 3D Vision
Keywords: 3D Foundation Models / Streaming Reconstruction / Geometric Context Attention / Camera Pose Estimation / SLAM

TL;DR

GCT draws on classical SLAM principles to decompose streaming spatial context into anchor context, a local pose-reference window, and lightweight trajectory memory via Geometric Context Attention (GCA), slashing per-frame token growth by ~80× and achieving state-of-the-art streaming reconstruction at 20 FPS across long sequences exceeding 10,000 frames.

Background & Motivation

Human spatial cognition perceives the physical world through continuous sensory streams without exhaustively memorizing every fleeting visual observation. Instead, it selectively retains salient reference anchors, rich local surroundings, and a compact topological trajectory. In contrast, recent feed-forward 3D foundation models, such as VGGT and Depth Anything 3, owe their impressive multi-view pose and surface reconstruction capabilities to global all-to-all cross-view self-attention computed over pre-captured, unordered image sets. Adapting such foundation models to real-time streaming settings—where frames arrive sequentially without access to future observations—presents severe challenges. Existing streaming paradigms face an acute dilemma: recurrent state compression methods like CUT3R maintain constant memory footprints but suffer from catastrophic forgetting and severe long-range trajectory drift; causal attention caching methods such as StreamVGGT and Stream3R preserve nearly complete history, but their token sequences and computational burdens grow linearly with sequence length, quickly exhausting GPU memory; meanwhile, hybrid SLAM methods marry deep backends with classical keyframe graphs, but rely heavily on heuristic hand-crafted rules and non-linear bundle adjustment optimizations that compromise end-to-end learning and pure feed-forward real-time throughput.

The core tension in streaming 3D reconstruction lies in the trade-off between retaining rich geometric context for long-term consistency and maintaining a compact state representation for efficient per-frame inference. Prior streaming frameworks either over-compress and lose vital geometric priors, or store redundant image tokens without structural distinction. Classical SLAM systems successfully reconcile this balance through principled spatial context decomposition: maintaining a reference frame for coordinate grounding and metric scale establishment, a local sliding window for dense matching and relative motion tracking, and a global pose graph for bounding accumulated drift.

This paper's angle of attack is to internalize these classical SLAM geometric priors directly into an end-to-end differentiable Transformer attention mechanism, eliminating hand-crafted heuristics and iterative solvers. Core idea: introduce Geometric Context Attention (GCA), which explicitly partitions the streaming attention state into an anchor context for coordinate and scale grounding, a local sliding window for dense relative pose registration, and an evicted trajectory memory with video temporal positional encodings for drift suppression, delivering stable, high-fidelity streaming 3D reconstruction with near-constant incremental compute over unbounded sequences.

Method

Overall Architecture

GCT ingests an incoming causal video stream \(I = \{I_1, I_2, \dots\}\), predicting the camera pose \(\hat{P}_t\) and depth map \(\hat{D}_t\) of each arriving frame \(I_t\) using only current and past frames \(\{I_1, \dots, I_t\}\). Each frame is initially processed by a DINOv2-pretrained Vision Transformer (ViT) to extract \(M\) image patch tokens, which are augmented with 6 context tokens: 1 camera token \(c\), 4 register scratch tokens \(r_j\), and 1 learnable anchor token \(a\). The augmented token sequence is passed through alternating layers of Frame Attention and Geometric Context Attention (GCA). Frame Attention refines intra-frame representations independently, while GCA conducts structured cross-frame geometric message passing. Finally, task-specific prediction heads decode the updated representations: a camera head regresses the metric camera-to-world pose from the camera token, and a depth head decodes the dense depth map from the image patch tokens.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Continuous Monocular Video Stream<br/>Frame It arrives causally over time"] --> B["ViT Feature Extraction & Token Augmentation<br/>DINOv2 patch tokens + 6 context tokens"]
    B --> C["Intra-Frame Attention Layers<br/>Frame Attention refines per-frame features"]
    C --> D["Geometric Context Attention GCA<br/>Hierarchical cross-frame geometric reasoning"]
    D --> E["Anchor Context<br/>Fixes initial metric scale and global origin"]
    D --> F["Local Pose-Reference Window<br/>Retains dense image tokens for k recent frames"]
    D --> G["Trajectory Memory & Video RoPE<br/>Evicts image tokens, retains 6 context tokens with RoPE"]
    E & F & G --> H["Task-Specific Heads Decoding<br/>Camera Head predicts Pt, Depth Head predicts Dt"]
    H --> I["Streaming 3D Reconstruction Output<br/>Camera trajectory Pt and metric point cloud Dt"]

Key Designs

1. Anchor Context: Coordinate Grounding and Absolute Metric Scale Establishment

Monocular 3D reconstruction is fundamentally ill-posed due to scale ambiguity. Offline feed-forward models typically resolve this by normalizing predictions against the global reconstructed point cloud, an operation impossible in causal streaming where the future sequence remains unobserved. GCT resolves scale ambiguity at the sequence onset by designating the initial \(n\) frames (\(n \ll N\), e.g., \(n \in [2, 8]\)) as anchor frames. These anchor frames undergo full bidirectional cross-attention during initialization and are assigned distinct learned anchor token values to distinguish them from incoming streaming frames. Throughout the remainder of the streaming session, all image tokens and context tokens of these anchor frames remain permanently cached in the attention context as fixed coordinate anchors. During training, all ground-truth scene depth and translation labels are standardized by the mean distance of the ground-truth anchor point cloud from the origin, \(s = \frac{1}{|\bar{X}_{anchor}|} \sum_{x \in \bar{X}_{anchor}} \|x\|_2\), establishing a stable, consistent metric coordinate frame from the very first frame.

2. Local Pose-Reference Window: Dense Geometric Overlap and Relative Motion Registration

While anchor frames anchor the global coordinate system, they rapidly lose direct visual overlap as the camera traverses new areas, rendering them insufficient for registering incremental frame-to-frame motion. GCT maintains a dynamic sliding window of the \(k\) most recent frames (randomly sampled between 16 and 64 during training, set to 64 at inference), fully retaining all dense image patch tokens (\(M \approx 500\) tokens per frame). This dense local visual overlap provides rich fine-grained correspondence cues, enabling the model to accurately estimate relative motion and local surface geometry. To enforce rigorous local rigidity and prevent temporal jitter within this local window, the network is trained with a causal relative pose loss over all intra-window frame pairs, ensuring seamless, smooth trajectory propagation.

3. Trajectory Memory & Video RoPE: Lightweight State Eviction and Global Drift Mitigation

Over trajectories spanning thousands of frames, discarding intermediate frames outside the sliding window causes unbounded cumulative drift, whereas caching full image tokens triggers catastrophic memory explosion. GCT addresses this via a structured trajectory eviction policy: once a frame falls outside the active sliding window, its heavy image patch tokens (\(M\) tokens) are discarded from the KV cache, retaining only 6 lightweight context tokens (1 camera token, 1 anchor token, and 4 register tokens). Furthermore, video temporal rotary position encodings (Video RoPE) are injected into the retained tokens to preserve strict temporal order and relative temporal distances. Under GCA, the total token count for a \(T\)-frame sequence is bounded by \((n+k) \cdot M + 6T\), reducing the incremental per-frame token growth rate by approximately 80× compared to standard causal attention (\(M+6\) tokens per frame). This guarantees near-constant per-frame memory and computation for sequences exceeding 10,000 frames.

Loss & Training

GCT is optimized end-to-end using a composite multi-task objective comprising depth loss, absolute pose loss, and relative pose loss:

\[\mathcal{L} = \lambda_{\text{depth}} \mathcal{L}_{\text{depth}} + \lambda_{\text{abs-pose}} \mathcal{L}_{\text{abs-pose}} + \lambda_{\text{rel-pose}} \mathcal{L}_{\text{rel-pose}}\]

The depth loss \(\mathcal{L}_{\text{depth}}\) incorporates uncertainty-weighted L1 depth error, surface normal gradient smoothness, and an uncertainty regularization penalty. The absolute pose loss \(\mathcal{L}_{\text{abs-pose}}\) computes Huber error against ground-truth camera poses parameterized as Camera-to-World (C2W) transformations, decoupling translation estimation from rotation error amplification inherent in traditional World-to-Camera formulations. The relative pose loss \(\mathcal{L}_{\text{rel-pose}}\) penalizes geodesic rotation error and L1 translation error across all frame pairs \((i, j)\) in the local sliding window:

\[\mathcal{L}_{\text{rel-pose}} = \frac{1}{k(k-1)} \sum_{i \neq j, \, i, j \in \{1,\dots,k\}} \left( \mathcal{L}_{\text{rot}}(i, j) + \lambda_{\text{trans}} \mathcal{L}_{\text{trans}}(i, j) \right)\]

Training follows a two-stage curriculum: Stage 1 pretrains an offline base model with standard global attention on diverse video and static datasets (160K steps); Stage 2 introduces GCA and fine-tunes on long-sequence video datasets (160K steps) using progressive view scaling (linearly increasing sequence length from 24 to 320 frames). To handle GPU memory constraints during long-sequence training, Ulysses-based context parallelism (degree 16) distributes views across 64 GPUs. During inference, GCT utilizes FlashInfer with a Paged KV-cache layout, completely avoiding memory reallocation overhead during sliding window token evictions and achieving steady ~20 FPS inference at 518×378 resolution.

Key Experimental Results

Main Results

On the challenging Oxford Spires benchmark featuring extensive indoor-outdoor traversals and complex lighting shifts (sparse setting: 320 frames per scene, sampled every 12 frames), GCT outperforms prior offline foundation models (Depth Anything 3, VGGT), optimization-based SLAM methods (DROID-SLAM, VIPE), and all streaming baselines across trajectory accuracy metrics.

Methods	Type	AUC@15 ↑	AUC@30 ↑	ATE (m) ↓	RPE-trans ↓	RPE-rot (°) ↓
Fast3R	offline	1.20	2.99	34.80	8.21	59.51
VGGT	offline	23.84	35.09	24.78	8.87	22.79
Depth Anything 3 (DA3)	offline	49.84	56.68	12.87	3.22	16.17
DROID-SLAM	optim	8.58	21.41	21.84	1.02	6.90
VIPE	optim	45.35	51.88	10.52	0.43	5.98
CUT3R	online	5.98	14.95	18.16	1.17	7.18
TTT3R	online	13.92	25.90	19.35	2.28	13.30
Wint3R	online	11.61	23.42	21.10	1.62	6.27
Stream3R	online	9.67	15.21	29.58	6.67	16.90
GCT (Ours)	online	63.22	76.46	5.37	0.93	3.69

Across diverse standard 3D reconstruction and pose estimation benchmarks (ETH3D, 7-Scenes, Tanks and Temples, NRGBD), GCT establishes new state-of-the-art results for feed-forward streaming models:

Dataset	Metric	GCT (Ours)	Runner-up Online (Wint3R / Stream3R)	Relative Improvement
ETH3D	Trajectory ATE (m) ↓	0.43	0.86 (Wint3R)	50.0% error reduction
ETH3D	Reconstruction F1 (%) ↑	86.80	77.28 (Wint3R)	+9.52 points
7-Scenes	Trajectory ATE (m) ↓	0.08	0.11 (Stream3R)	27.3% error reduction
7-Scenes	Reconstruction F1 (%) ↑	82.38	78.81 (Wint3R)	+3.57 points
Tanks & Temples	AUC@30 ↑	92.34	71.39 (InfiniteVGGT)	+20.95 points
NRGBD	Reconstruction F1 (%) ↑	65.10	56.96 (Wint3R)	+8.14 points
KITTI Odometry	Trajectory ATE (m) ↓	24.12	31.89 (VGGT-Long)	24.4% error reduction

Ablation Study

Ablations on TartanGround validate the individual necessity of each component in Geometric Context Attention (GCA):

Configuration	Rel. Loss	A. Init.	Co. Tok.	V. RoPE	AUC@3 ↑	AUC@30 ↑	ATE (m) ↓	RPE-rot (°) ↓	Note
Baseline Sliding Window	✓				9.80	65.84	8.59	2.57	Local relative loss only
+ Anchor Initialization	✓	✓			13.63	68.71	7.88	2.90	Establishes scale and coordinate frame
w/o Relative Pose Loss		✓	✓		13.91	68.25	8.25	5.35	Severe rotation jitter without local constraint
+ Context Tokens	✓	✓	✓		15.75	69.92	7.46	2.26	Compact trajectory tokens aggregate history
Full Model	✓	✓	✓	✓	16.39	71.87	5.98	1.93	Video RoPE imposes temporal distance order

Hyper-parameter sensitivity analysis on Oxford Spires confirms that increasing the local window size from \(k=16\) to \(k=64\) improves ATE from 11.2m to 5.3m (saturating at \(k \ge 48\)), while performance remains virtually invariant to anchor count \(n\) across 2, 4, and 8 frames.

Key Findings

• Trajectory memory combined with Video RoPE is essential for mitigating long-term trajectory drift: introducing context tokens raises AUC@30 to 69.92, and adding Video RoPE drops ATE from 7.46m to 5.98m, showing that compact macro-trajectory tokens provide powerful global constraints.
• Relative pose supervision acts as an indispensable local stabilizer: removing relative loss causes relative rotation error (RPE-rot) to jump from 2.26° to 5.35°, demonstrating that global attention alone cannot prevent high-frequency local motion distortion.
• Remarkable drift resistance over long sequences: in dense 3,840-frame Oxford Spires sequences, baseline streaming models degrade severely as trajectory length expands (CUT3R: 18.16m → 32.47m; Wint3R: 21.10m → 32.90m), whereas GCT keeps error virtually flat (5.37m → 5.60m).

Highlights & Insights

• Internalizing classical SLAM decomposition into feed-forward Transformer attention: Mapping reference frames, local sliding windows, and global pose graphs into anchor context, dense window tokens, and compact trajectory tokens bridges classical geometry with modern foundation models.
• Selective image token eviction for constant-compute streaming: By evicting heavy patch tokens (\(M \approx 500\)) and retaining only 6 lightweight context tokens for retired frames, GCT reduces incremental token growth by ~80×, keeping memory and computation bounded over arbitrarily long streams.
• Engineered paged KV-cache with FlashInfer: Integrating Paged KV-cache avoids memory reallocation penalties during frequent sliding-window evictions, ensuring smooth 20+ FPS throughput on H800 and ~14 FPS on consumer-grade RTX 4090 GPUs.

Limitations & Future Work

• Sensitivity to initial anchor frame quality: Because global scale and world coordinates are grounded in the first few anchor frames, rapid motion blur, occlusion, or pure rotation during the sequence start can induce systemic scale drift across the entire run.
• Lack of explicit loop closure optimization: Although trajectory memory provides implicit drift suppression, the model lacks explicit loop detection and pose-graph relaxation when re-encountering long-separated previously mapped environments.
• Dynamic scene interference: While robust on Sintel dynamic benchmarks, prolonged occlusions caused by large moving objects can challenge the rigid-world assumption and degrade local surface accuracy.

Related Work & Insights

• vs CUT3R / TTT3R: CUT3R compresses history into an RNN hidden state, leading to catastrophic forgetting over long sequences; TTT3R attempts test-time training, which adds substantial runtime overhead. GCT uses structured GCA attention to retain explicit geometric anchors and history tokens, maintaining long-range consistency without iterative test-time updates.
• vs StreamVGGT / Stream3R: Existing causal variants cache all visual tokens for every frame, suffering from quadratic/linear explosion in compute and memory. GCT prunes non-window image tokens down to 6 context tokens per frame, drastically improving computational efficiency.
• vs DROID-SLAM / VIPE: Traditional hybrid learning SLAM systems rely on iterative bundle adjustment and hand-crafted keyframing. GCT achieves superior trajectory accuracy (ATE 5.37m vs. VIPE's 10.52m and DROID-SLAM's 21.84m on Oxford Spires) in a purely feed-forward pass.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Elegant mapping of classical SLAM spatial context principles into an end-to-end differentiable Transformer attention design.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations across Oxford Spires, ETH3D, 7-Scenes, Tanks & Temples, NRGBD, Sintel, and KITTI, supplemented by solid long-sequence ablations.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear exposition, intuitive motivation, and detailed mathematical and architectural formulations.
• Value: ⭐⭐⭐⭐⭐ High practical impact for real-time robotics, autonomous navigation, and spatial computing foundation models.

Related Papers

• [ECCV 2026] OVGGT: O(1) Constant-Cost Streaming Visual Geometry Transformer
• [ECCV 2026] LoGeR: Long-Context Geometric Reconstruction with Hybrid Memory
• [ECCV 2026] GRAFT: Geometric Refinement and Fitting Transformer for Human Scene Reconstruction
• [ICLR 2026] Streaming Visual Geometry Transformer
• [ECCV 2026] FILT3R: Latent State Adaptive Kalman Filter for Streaming 3D Reconstruction