OVGGT: O(1) Constant-Cost Streaming Visual Geometry Transformer¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/VAISR/OVGGT
Area: 3D Vision
Keywords: Streaming 3D Reconstruction, KV Cache Compression, Geometric Foundation Model, Visual Geometry Transformer, Constant-Cost Inference

TL;DR¶

OVGGT introduces a training-free streaming visual geometry transformer framework that bounds both memory footprint and per-step computation to an $O(1)$ constant budget via FFN-residual-based Self-Selective Caching with spatial smoothing and Dynamic Anchor Protection, eliminating out-of-memory failures while outperforming full-cache baselines.

Background & Motivation¶

Reconstructing dense, metrically consistent 3D scene geometry from continuous monocular video streams is a core enabling technology for autonomous navigation, augmented reality, and robotics. Geometric foundation models such as DUSt3R and VGGT have fundamentally shifted this landscape away from fragile, multi-stage Structure-from-Motion (SfM) pipelines toward direct, end-to-end regression of 3D pointmaps and camera poses. However, all-to-all attention architectures like VGGT suffer from quadratic computational and memory complexity, limiting them to short offline batches and exhausting GPU memory within a few hundred frames. While causal streaming reformulations such as StreamVGGT enable single-pass autoregressive processing with an external key-value (KV) cache, their linearly growing cache size quickly triggers out-of-memory (OOM) failures and degrades per-step throughput, rendering long-horizon streaming impossible.

The fundamental tension lies in maintaining constant memory and compute resources ($O(1)$ complexity) over unbounded sequences while preserving long-range geometric continuity and coordinate frame consistency. Existing token compression strategies developed for large language models (LLMs) cannot be directly transplanted into visual geometry: textual compression targets discrete linguistic semantics, whereas visual patch tokens possess rigid 2D/3D spatial topologies. Furthermore, state-of-the-art acceleration kernels like FlashAttention avoid materializing the full attention matrix, making attention-weight-based pruning computationally prohibitive. More critically, unconstrained token eviction across extended camera trajectories inevitably discards the origin reference frames, precipitating catastrophic coordinate drift.

To resolve these challenges, this paper leverages the forward-pass activation dynamics of geometric transformers to decouple saliency rating from coordinate preservation. Core idea: exploit the forward-pass FFN residual magnitude as a zero-overhead, FlashAttention-compatible metric of geometric salience combined with 2D spatial Gaussian smoothing, and establish Dynamic Anchor Protection to permanently shield initial coordinate and view-overlap keyframes from eviction, enabling drift-free streaming 3D reconstruction within a strict constant-cost envelope.

Method¶

Overall Architecture¶

OVGGT operates on top of a causal temporal attention pipeline, accepting video frames sequentially in real time. At time step $t$, the input frame $I_t$ is processed by a frozen DINOv2 visual backbone to generate $N_p$ patch tokens, which are concatenated with 1 camera token and 4 register tokens (yielding $M$ tokens in total). These tokens traverse 24 alternating Transformer blocks, each comprising intra-frame spatial self-attention (SA) and cross-frame temporal causal attention (CA) querying a bounded KV cache $\mathcal{C}_t$. Finally, dedicated camera, depth, and point cloud prediction heads output camera extrinsics, dense depth maps, and 3D point clouds. To guarantee constant resource usage, OVGGT enforces a per-layer fixed token budget $B^{(l)}$ (summing to $B$), capping per-step storage and compute to $O(M \cdot B)$. The framework consists of two complementary engines: Self-Selective Caching (SSC) and Dynamic Anchor Protection (DAP).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
    A["Input Video Stream I_t<br/>DINOv2 + Aux Tokens"] --> B["Intra-Frame Spatial Self-Attention<br/>Local geometry extraction"]
    B --> C["Cross-Frame Causal Attention<br/>Querying bounded KV cache"]
    C --> D["Self-Selective Caching SSC<br/>FFN residual scoring + smoothing"]
    D --> E["Hybrid Scoring Compression<br/>Current salience + historical diversity"]
    E --> F["Dynamic Anchor Protection DAP<br/>Global origin + view-overlap anchors"]
    F --> G["Fixed-Budget KV Cache C_t<br/>O(1) memory and compute update"]
    G --> H["3D Prediction Heads<br/>Camera poses + Depth + 3D points"]

Key Designs¶

1. Activation Value Rating & Spatial Smoothing: Zero-Overhead, FlashAttention-Compatible Saliency Selecting which tokens to preserve in a streaming cache is critical. Traditional attention-weight pruning requires materializing the full $N \times N$ attention matrix, which breaks the fused kernel of FlashAttention and doubles both memory and latency. OVGGT recognizes that in Pre-LN Transformer blocks, the non-linear update vector produced by the feed-forward network (FFN)—namely the FFN residual magnitude—directly quantifies the representational shift and geometric importance of each token. Shallow layers activate on high-frequency textures, middle layers highlight geometric structures, and deep layers isolate semantic boundaries. The activation score of token $i$ at layer $l$ is computed as: $$s_i^{(l)} = \big\|\lambda_2^{(l)} \cdot \text{FFN}(\text{LN}(\mathbf{h}_i^{(l)}))\big\|_2$$ where $\lambda_2^{(l)}$ is the LayerScale parameter. Because the FFN residual is naturally generated during the forward pass, this scoring incurs zero additional compute or memory. However, selecting tokens purely based on raw scores scatters retained patches across the 2D plane, breaking spatial continuity and degrading surface normals. OVGGT applies 2D Gaussian kernel smoothing over the activation map $\mathbf{S} \in \mathbb{R}^{H_p \times W_p}$: $$\tilde{\mathbf{S}} = \alpha \cdot (\mathbf{G} * \mathbf{S}) + (1 - \alpha) \cdot \mathbf{S}$$ where $\alpha$ modulates smoothing intensity (auxiliary tokens bypass smoothing). This enforces spatial clustering of retained tokens, preserving local neighborhood structures indispensable for high-fidelity 3D surfaces.

2. Hybrid Scoring Cache Compression: Balancing Instantaneous Salience and Historical Diversity When evicting tokens to maintain the layer budget $B^{(l)}$, only current-frame tokens pass through the FFN and possess fresh activation scores, while historical cached tokens lack up-to-date residual updates. OVGGT introduces a dual-metric hybrid scoring framework: for tokens in the historical evictable set $\mathcal{U}_{\text{hist}}$, their importance is measured by feature diversity relative to the key centroid, $d_i = 1 - \cos(\mathbf{k}_i, \bar{\mathbf{k}})$, preserving a diverse geometric basis. For current-frame tokens in $\mathcal{U}_{\text{new}}$, the smoothed activation score $s_i$ is used directly. After min-max normalization into $\hat{d}_i, \hat{s}_i \in [0, 1]$, a balancing coefficient $\beta$ unifies the ranking: $$r_i = \begin{cases} (1 - \beta) \cdot \hat{d}_i, & i \in \mathcal{U}_{\text{hist}} \\ \beta \cdot \hat{s}_i, & i \in \mathcal{U}_{\text{new}} \end{cases}$$ The lowest-scoring tokens beyond $B^{(l)} - |\mathcal{P}|$ are evicted. Furthermore, per-layer token budgets $B^{(l)}$ are dynamically allocated proportional to each layer's key diversity, assigning greater capacity to information-rich layers.

3. Dynamic Anchor Protection: Preserving Global Coordinate Origin and Long-Range Reference Repeated eviction cycles eventually purge all historical tokens, causing severe geometric drift because monocular reconstruction relies on early views to establish the metric world coordinate system. Dynamic Anchor Protection designates a protected set $\mathcal{P} = \mathcal{P}_{\text{init}} \cup \mathcal{P}_{\text{hist}}$ exempt from eviction. First, all $M$ tokens of the initial frame are permanently anchored as $\mathcal{P}_{\text{init}}$, locking the global reference frame. Second, as the camera explores new areas with zero overlap to the first frame, historical anchors $\mathcal{P}_{\text{hist}}$ are adaptively registered. By projecting the 3D points of the most recent anchor frame into the current view, the coverage ratio $\rho_t$ is tracked; whenever $\rho_t < \tau$ (default $\tau=0.2$) and at least 100 frames have elapsed, a new anchor is instantiated. For this frame, only the top-$\eta$ (default $\eta=0.05$) patch tokens with the highest point-cloud confidence scores are protected. An active anchor FIFO queue with capacity $K_{\max}=3$ caps total anchor overhead to $|\mathcal{P}| \le M + K_{\max} \cdot \lceil \eta N_p \rceil$, leaving the vast majority of the budget for dynamic cache renewal.

Loss & Training¶

OVGGT is entirely training-free, operating directly with pre-trained StreamVGGT weights without fine-tuning or parameter updates. In deployment, the default token budget is set to $B = 200\text{K}$ tokens (consuming $\sim$10 GB VRAM on an NVIDIA RTX 5090 GPU). Key inference hyperparameters include smoothing weight $\alpha = 0.5$, hybrid scoring balance $\beta = 0.5$, anchor overlap threshold $\tau = 0.2$, minimum anchor interval of 100 frames, confidence retention ratio $\eta = 0.05$, and maximum anchor capacity $K_{\max} = 3$.

Key Experimental Results¶

Main Results¶

OVGGT was evaluated across diverse benchmarks: indoor environments (7-Scenes, NRGBD), open outdoor driving (ETH3D), and ultra-long sequences containing up to 10,000 frames (Long3D). Performance metrics include reconstruction Accuracy (Acc, lower is better), Completeness (Comp, lower is better), and Normal Consistency (NC, higher is better). On 7-Scenes and NRGBD, the full-cache baseline StreamVGGT encountered OOM beyond 200 frames, whereas OVGGT maintained constant-resource inference and achieved superior geometric precision.

Table 1: Quantitative 3D reconstruction results on 7-Scenes and NRGBD datasets across sequence lengths (excerpted from original Table 1).

Dataset	Seq. Len.	Metric	Ours (OVGGT)	StreamVGGT (Full Cache)	Evict3R† (Matched Budget)	InfiniteVGGT
7-Scenes	200 Frames	Acc ↓ (Mean / Med)	0.024 / 0.008	0.038 / 0.014	0.037 / 0.013	0.046 / 0.016
7-Scenes	200 Frames	Comp ↓ (Mean / Med)	0.021 / 0.005	0.029 / 0.007	0.027 / 0.007	0.031 / 0.008
7-Scenes	200 Frames	NC ↑ (Mean / Med)	0.587 / 0.635	0.583 / 0.628	0.584 / 0.631	0.582 / 0.627
7-Scenes	500 Frames	Acc ↓ (Mean / Med)	0.031 / 0.011	OOM	0.042 / 0.016	0.040 / 0.015
7-Scenes	500 Frames	Comp ↓ (Mean / Med)	0.020 / 0.003	OOM	0.026 / 0.005	0.024 / 0.005
7-Scenes	500 Frames	NC ↑ (Mean / Med)	0.561 / 0.593	OOM	0.559 / 0.589	0.561 / 0.593
NRGBD	100 Frames	Acc ↓ (Mean / Med)	0.022 / 0.014	0.024 / 0.014	0.031 / 0.020	0.035 / 0.022
NRGBD	300 Frames	Acc ↓ (Mean / Med)	0.037 / 0.022	OOM	0.042 / 0.026	0.053 / 0.031
NRGBD	500 Frames	Acc ↓ (Mean / Med)	0.054 / 0.032	OOM	0.072 / 0.040	0.070 / 0.046

Table 2: Quantitative comparison on full outdoor ETH3D and ultra-long Long3D sequences (excerpted from original Table 2).

Dataset	Method / Config	Acc ↓ (Mean / Med)	Comp ↓ (Mean / Med)	NC ↑ (Mean / Med)	Operational Status
ETH3D (Outdoor Full)	StreamVGGT	0.601 / 0.369	0.442 / 0.169	0.791 / 0.933	Full cache reference
ETH3D (Outdoor Full)	InfiniteVGGT	0.603 / 0.371	0.444 / 0.169	0.792 / 0.933	Bounded cache
ETH3D (Outdoor Full)	Ours (200K Budget)	0.628 / 0.396	0.380 / 0.121	0.790 / 0.934	Constant budget
ETH3D (Outdoor Full)	Ours (400K Budget)	0.535 / 0.317	0.394 / 0.107	0.793 / 0.934	Constant budget (+1GB)
Long3D (10,000 Frames)	StreamVGGT	OOM	OOM	OOM	Memory exhausted
Long3D (10,000 Frames)	Evict3R†	4.928 / 2.710	0.715 / 0.204	0.507 / 0.504	Noticeable degradation
Long3D (10,000 Frames)	InfiniteVGGT	4.344 / 3.668	0.974 / 0.205	0.517 / 0.525	Accumulated drift
Long3D (10,000 Frames)	Ours (200K Budget)	2.453 / 1.794	0.390 / 0.060	0.507 / 0.509	Strong drift suppression
Long3D (10,000 Frames)	Ours (400K Budget)	2.449 / 1.675	0.542 / 0.151	0.507 / 0.509	Best median accuracy

Ablation Study¶

To isolate the efficacy of FFN-residual scoring, four eviction strategies were evaluated under an identical $B=200\text{K}$ budget with DAP disabled:

Table 3: Comparison of token eviction scoring strategies on 7-Scenes and NRGBD (from original Table 4).

Scoring Criterion	FlashAttention Compatible	Sequence Length	7-Scenes Acc ↓	7-Scenes CD ↓	NRGBD Acc ↓	NRGBD CD ↓
Attention Weight Matrix	No (Full Materialization)	100 Frames	0.024	0.035	0.020	0.025
Random Eviction	Yes	100 Frames	0.026	0.035	0.025	0.030
$Q \cdot K$ Dot Product	Yes	100 Frames	0.026	0.035	0.026	0.031
FFN Residual (Ours)	Yes (Fully Compatible)	100 Frames	0.023	0.033	0.022	0.027
Attention Weight Matrix	No (Full Materialization)	300 Frames	0.023	0.032	0.035	0.041
Random Eviction	Yes	300 Frames	0.027	0.034	0.044	0.049
$Q \cdot K$ Dot Product	Yes	300 Frames	0.032	0.037	0.044	0.049
FFN Residual (Ours)	Yes (Fully Compatible)	300 Frames	0.026	0.032	0.037	0.043

Table 4: Ablation of Dynamic Anchor Protection components on KITTI depth estimation at 500 frames (from original Table 5).

Initial Anchor $P_{\text{init}}$	Historical Anchors $P_{\text{hist}}$	Far (>35m) F1% ↑	Far F5% ↑	Far $\delta_{1.05}$ ↑	Near (15-35m) F1% ↑	Near F5% ↑
✕	✕	Baseline	Baseline	Baseline	Baseline	Baseline
✓	✕	+5.43%	+4.19%	+4.22%	+3.51%	+2.81%
✓	✓ (Full DAP)	+10.15%	+7.23%	+7.23%	+5.35%	+4.69%

Key Findings¶

Full cache is not an accuracy upper bound: Retaining every historical token in StreamVGGT diluted attention weights over uninformative noise, achieving an Acc of 0.038 on 7-Scenes (200 frames). OVGGT's selective caching improved this to 0.024 while eliminating OOM risks.
FFN residual provides a drop-in, zero-overhead scoring proxy: Table 4 demonstrates that FFN residuals closely match or beat materialized attention weights without sacrificing FlashAttention acceleration, outperforming heuristic dot-product and random eviction baselines.
Anchoring is indispensable for long-range metric depth: DAP boosted long-range depth metrics on KITTI by over 10% on F1%, demonstrating that locking early coordinate references prevents compounding drift.
Constant throughput and memory: Across 1,000+ frames, OVGGT stably maintains $\sim$14 FPS and $\sim$10 GB allocated VRAM on an RTX 5090 GPU, whereas StreamVGGT failed at 200 frames and Evict3R consumed significantly higher memory due to materialized attention maps.

Highlights & Insights¶

Repurposing FFN residuals as geometric salience proxies: Unlocks token pruning inside fused FlashAttention kernels without requiring explicit attention weights or retraining.
Spatial smoothing for visual token cohesion: Tailors token eviction to spatial physics by using 2D Gaussian kernels to retain continuous surface patches rather than fragmented points.
Decoupling coordinate stability from cache turnover: Dynamically shields the initial world coordinate frame and long-range view-overlap anchors, preventing trajectory drift over arbitrarily long sequences.

Limitations & Future Work¶

Single-pass drift under extreme conditions: As a purely causal frontend without loop closure or bundle adjustment, unavoidable drift can slowly accumulate over extended loops or textureless turns.
Dynamic scene susceptibility: While tested on scenes with moving objects (Bonn, KITTI), persistent dynamic foregrounds could be erroneously registered as historical anchors, perturbing static geometry.
Future directions: Integrating OVGGT as a constant-cost frontend with a decoupled backend global pose graph optimizer or staged streaming inference.

vs StreamVGGT: StreamVGGT established causal attention streaming for 3D transformers but exhausts GPU VRAM within 200 frames. OVGGT provides an $O(1)$ constant-resource plug-in that simultaneously solves memory scaling and improves reconstruction fidelity.
vs Evict3R: Evict3R requires explicit attention matrix materialization, breaking FlashAttention compatibility and increasing VRAM and latency. OVGGT retains native FlashAttention execution via FFN residual rating.
vs InfiniteVGGT: InfiniteVGGT compresses tokens purely by key diversity without spatial anchors, leading to substantial drift on ultra-long trajectories (4.344 Acc on Long3D). OVGGT's DAP cuts this error down to 2.453.

Rating¶

Novelty: ⭐⭐⭐⭐☆ Elegant insight connecting FFN residuals to FlashAttention-compatible geometric pruning, complemented by visually grounded spatial smoothing and coordinate anchoring.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across indoor, outdoor, video depth, and 10,000-frame benchmarks with exhaustive ablations.
Writing Quality: ⭐⭐⭐⭐⭐ Clear progression from architectural bottlenecks to visual-geometric principles, with thorough empirical validation.
Value: ⭐⭐⭐⭐⭐ Plug-and-play, training-free design directly deployable to causal 3D foundation models on consumer-grade hardware.