Stream3D-VLM: Online 3D Spatial Understanding with Incremental Geometry Priors

Conference: ECCV 2026
Paper: ECCV Full Paper
Code: https://stream3d-vlm.github.io/
Area: Multimodal VLM
Keywords: 3D Vision-Language Models, Online Video Understanding, Spatial Intelligence, Incremental Geometry Priors, Autoregressive Streaming Control

TL;DR

Stream3D-VLM presents the first online 3D vision-language model operating solely on streaming video, combining autoregressive next-token prediction for response timing, lightweight VSFI for incremental geometry prior injection, and GAVC for spatial token compression to enable real-time 3D spatial understanding.

Background & Motivation

The rapid advancement of Multimodal Large Language Models (MLLMs) in visual-language reasoning and cross-modal interaction has catalyzed growing interest in extending spatial intelligence to real-world physical environments, including autonomous robotics, smart wearable devices, and AR/VR glasses. To equip models with spatial comprehension in 3D physical spaces, early 3D multimodal models predominantly relied on explicit 3D sensor modalities such as point clouds, meshes, or depth maps aligned through instruction tuning. However, the acquisition and annotation costs of such high-precision 3D data are prohibitively high and scarce, severely restricting model scaling and preventing effective leverage of abundant unannotated 2D web-scale videos. Although recent feed-forward 3D reconstruction models allow extracting spatial cues purely from monocular RGB video, existing 3D vision-language models remain fundamentally confined to offline inference settings, requiring complete 3D scene scans or pre-segmented video clips prior to interaction.

In practical embodied agent and wearable scenarios, video frames arrive sequentially along an irreversible temporal stream, giving rise to two critical tensions. First, the model must autonomously decide not only what to respond, but also when to maintain silence and when to trigger an answer under continuous visual input. Existing online 2D video models predominantly target high-level temporal action detection and egocentric forecasting, lacking fundamental metric reasoning over 3D geometry and camera-object spatial relationships; even when fine-tuned on 3D data, their performance on metric spatial tasks remains deficient. Second, as online interaction extends over time, uncompressed visual tokens cause linear context growth, resulting in severe KV-cache memory bloat and unacceptable decoding latency. Meanwhile, existing 2D token pruning heuristics discard tokens based on semantic similarity or attention heatmaps, which inevitably damages the underlying 3D physical geometric topology. Furthermore, the community previously lacked large-scale, explicitly timestamped streaming 3D-language interaction datasets and rigorous benchmarks.

The angle of attack in this paper is to unify streaming interaction control within the LLM's native autoregressive next-token generation framework, while leveraging streaming feed-forward 3D reconstruction to provide incremental geometry priors from monocular video. Core idea: formulate streaming control as next-token prediction using decision tokens to learn response timing, incrementally inject feed-forward 3D geometric and camera priors via a lightweight VSFI module, and dynamically compress visual tokens via geometry-guided 3D voxel clustering in GAVC, realizing efficient and real-time 3D spatial understanding purely from streaming RGB video.

Method

Overall Architecture

Stream3D-VLM models continuous monocular RGB streaming video as a strictly time-ordered input sequence, autonomously determining when to answer upon receiving user queries at arbitrary timestamps. The overall pipeline operates across three primary stages: first, an autoregressive streaming control mechanism predicts special decision tokens at each frame to determine whether to remain silent and ingest inputs or trigger response generation; second, the Visual-Spatial Feature Integration (VSFI) module incrementally extracts and integrates latent 3D geometry and camera pose priors from StreamVGGT into 2D visual tokens; finally, the Geometry-Adaptive Voxel Compression (GAVC) module back-projects visual tokens into 3D coordinate space and executes spatial clustering to feed compressed tokens into the LLM backbone for real-time autoregressive text generation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Streaming RGB Frames & Text Query"] --> B["Autoregressive Streaming Control<br/>next-token prediction of &lt;SEP&gt; ingest or &lt;END&gt; trigger"]
    B --> C["Visual-Spatial Feature Integration<br/>VSFI incrementally injects StreamVGGT geometry & camera tokens"]
    C --> D["Geometry-Adaptive Voxel Compression<br/>GAVC 3D spatial K-Means clustering & dual-attention aggregation"]
    D --> E["Pre-trained LLM Autoregressive Generation"]

Key Designs

1. Autoregressive Streaming Control: Unifying Response Timing with Native Next-Token Prediction Prior offline 3D models deployed in streaming scenarios must reprocess the entire historical context at every incoming frame to evaluate whether to answer, causing excessive memory consumption and high decoding latency. To grant the model autonomous timing awareness, Stream3D-VLM reformulates streaming control into the LLM's native autoregressive token prediction objective. Specifically, two special decision tokens are introduced: <SEP> (streaming continuation token, instructing the model to remain silent and ingest the next video frame) and <END> (response trigger token, indicating that target conditions are met or historical evidence is complete, immediately terminating ingestion and initiating textual output). During training, the supervision mask is adjusted so that decision tokens and language generation tokens are jointly trained using a unified multi-task objective: $$ \mathcal{L} = \lambda \mathcal{L}{\text{stream}} + \mathcal{L} $$ where }\(\mathcal{L}_{\text{stream}}\) computes standard cross-entropy over the decision token set \(\mathcal{D} = \{ \texttt{<SEP>}, \texttt{<END>} \}\), and \(\lambda\) balances timing precision against response quality. At test time, the model determines its immediate behavior at each frame via a single token decoding step, eliminating external heuristics or separate classification heads.

2. Visual-Spatial Feature Integration: Incrementally Injecting Latent Geometric and Camera Priors Monocular 2D videos inherently suffer from depth ambiguity and lack metric physical scale, making direct spatial localization and distance estimation unreliable, while explicit point cloud inputs fail to scale. Stream3D-VLM designs a lightweight Visual-Spatial Feature Integration (VSFI) module that leverages the streaming feed-forward 3D reconstruction model StreamVGGT to extract temporally aligned geometry priors online. For each incoming frame at timestamp \(t\), the native vision encoder extracts 2D visual tokens \(\mathbf{H}^{\text{2D}}_t \in \mathbb{R}^{N \times D_{\text{vis}}}\). Simultaneously, StreamVGGT's spatial encoder yields latent geometry tokens \(\mathbf{G}_t \in \mathbb{R}^{K \times D_{\text{geo}}}\) and a global camera token \(\mathbf{c}_t \in \mathbb{R}^{1 \times D_{\text{geo}}}\) capturing camera kinematics and scene dimensions. These are concatenated and projected into the visual embedding space via a 2-layer MLP: $$ \mathbf{H}^{\text{3D}}t = \text{MLP}([\mathbf{c}_t ; \mathbf{G}_t]) \in \mathbb{R}^{(K+1) \times D $$ The module uses }}\(\mathbf{H}^{\text{2D}}_t\) as Queries and \(\mathbf{H}^{\text{3D}}_t\) as Keys and Values across stacked cross-attention blocks, with a residual connection \(\mathbf{H}^{\text{f}}_t \leftarrow \mathbf{H}^{\text{f}}_t + \mathbf{H}^{\text{2D}}_t\) ensuring semantic preservation while grounding each token with metric 3D structural cues.

3. Geometry-Adaptive Voxel Compression: 3D Coordinate Clustering to Mitigate Long-Context Overhead As online interaction continues, visual tokens accumulate rapidly, causing severe KV-cache explosion and unacceptable response latencies. Conventional 2D pruning techniques rely exclusively on semantic attention scores or regular grid downsampling, neglecting camera viewpoint shifts and fragmenting 3D object surfaces. Stream3D-VLM introduces a plug-and-play Geometry-Adaptive Voxel Compression (GAVC) module. Utilizing metric depth maps \(D_t\) and camera calibration parameters \((\mathbf{K}_t, \mathbf{E}_t)\) estimated by StreamVGGT, each 2D patch coordinate \((u_j, v_j)\) is unprojected into physical 3D world coordinates: $$ \mathbf{p}_{t,j} = \mathbf{E}_t^{-1} \left( D_t(u_j, v_j) \mathbf{K}_t^{-1} [u_j, v_j, 1]^\top \right) $$ Tokens are lifted into spatial voxels by adding sinusoidal 3D positional encodings \(\mathbf{v}_{t,j} = \mathbf{H}^{\text{f}}_{t,j} + \text{PE}(\mathbf{p}_{t,j})\), followed by GPU-accelerated spatial K-Means clustering that partitions the voxel set into \(K\) spatially proximal clusters \(\mathcal{C}_k\). Within each cluster, a dual-attention mechanism models both semantic feature similarity and 3D Euclidean proximity: $$ w_j = \alpha s^{\text{f}}_j + (1 - \alpha) s^{\text{p}}_j, \quad s^{\text{f}}_j = \text{cos}(\mathbf{v}_j, \bar{\mathbf{v}}_k), \quad s^{\text{p}}_j = \exp\left(-\frac{|\mathbf{p}_j - \bar{\mathbf{p}}_k|^2}{\sigma_k^2}\right) $$ The aggregated cluster feature is obtained as \(\mathbf{v}'_k = \sum_{j \in \mathcal{C}_k} w_j \mathbf{v}_j\). This enables high token reduction (50% to 75% retention ratios) while strictly preserving spatial object boundaries and structural integrity.

Loss & Training

The framework is built upon Qwen2.5-VL backbones (3B and 7B variants) integrated with a frozen StreamVGGT-1B spatial encoder. Both the vision encoder and StreamVGGT encoder remain frozen, while only the VSFI module and the LLM backbone are fully fine-tuned. The model undergoes a single-epoch instruction tuning on the curated 1M+ streaming 3D spatio-temporal QA dataset combined with VSI-590K using AdamW (weight decay 0.03). Online inference operates at a default streaming rate of 1 FPS, utilizing internal KV caching to implicitly reuse temporal history without explicit re-computation.

Key Experimental Results

Main Results

Stream3D-VLM is systematically evaluated on the online Stream3D-Bench benchmark (29 tasks, 518 real-world videos, 10,037 QA pairs across ScanNet, ScanNet++, and ARKitScenes). Evaluation incorporates Answer-Timing Accuracy (ATA) alongside standard task accuracies to evaluate streaming response punctuality.

Method	Type	Avg. ↑	Backward Tracing (NA / MCA / OEA)	Realtime Perception (NA / MCA / OEA)	Forward Response (NA / MCA / OEA)	Timing Acc. (ATA) ↑	TTFT ↓	End2End Latency ↓	Memory Usage ↓
GPT-4o	Proprietary API	28.0	10.5 / 36.3 / 24.9	31.1 / 34.8 / 51.1	10.2 / 29.9 / 23.0	55.9%	-	-	-
GPT-5	Proprietary API	35.0	18.7 / 44.1 / 33.6	38.0 / 46.2 / 52.8	12.9 / 30.6 / 37.7	61.7%	-	-	-
LLaVA-Video-7B	Open-source 2D	20.4	12.7 / 34.6 / 11.4	21.6 / 28.6 / 38.0	10.2 / 20.3 / 6.5	55.5%	650ms	2.84s	25.8G
InternVL3.5-8B	Open-source 2D	27.0	21.4 / 36.2 / 17.6	28.2 / 38.2 / 42.4	25.2 / 25.8 / 8.2	23.0%	1771ms	6.72s	43.1G
Qwen2.5-VL-7B	Open-source 2D	22.5	11.6 / 36.3 / 13.6	23.1 / 33.2 / 39.4	11.3 / 28.4 / 5.5	19.2%	325ms	3.16s	40.0G
VideoLLM-online-8B (FT)	Online 2D	34.6	28.8 / 40.9 / 27.8	31.3 / 37.9 / 44.6	38.2 / 32.7 / 29.2	70.2%	120ms	1.29s	28.4G
Qwen2.5-VL-7B (FT)	Online 2D	47.8	45.6 / 56.6 / 45.4	45.7 / 44.6 / 48.7	46.6 / 57.4 / 39.8	73.1%	106ms	0.53s	36.8G
Stream3D-VLM-4B (Ours)	Online 3D	54.6	58.1 / 61.2 / 45.2	58.8 / 57.3 / 52.8	55.3 / 60.6 / 41.9	75.4%	43ms	0.24s	20.7G
Stream3D-VLM-8B (Ours)	Online 3D	58.8	59.8 / 67.9 / 50.5	61.4 / 61.5 / 54.3	58.1 / 66.7 / 49.4	86.7%	62ms	0.39s	36.6G

To evaluate offline spatial generalizability, the model was tested on VSI-Bench and standard 3D tasks (ScanQA, ScanRefer, Scan2Cap). Stream3D-VLM-8B achieved an average accuracy of 65.9% on VSI-Bench, substantially outperforming specialized offline models such as VLM-3R-8B (60.9%) and proprietary Gemini-2.5 Pro (51.5%). In traditional tasks, it set new video-input records on ScanQA (CIDEr 104.5 vs SOTA 102.1) and ScanRefer ([email protected] 52.5 vs SOTA 51.7).

Ablation Study

The ablations confirm the individual contributions of streaming loss weighting, VSFI geometry injection, and GAVC compression.

VSFI Geometry Prior Modeling Ablation: | Settings | Stream3D-Bench (NA / MCA / OEA) | VSI-Bench (NA / MCA) | Note | |---|---|---|---| | Baseline (Visual Only) | 46.0 / 52.9 / 44.6 | 42.9 / 46.2 | Lacks explicit geometric grounding; metric reasoning drops sharply | | w/o Camera Tokens | 55.4 / 60.4 / 46.3 | 60.5 / 61.4 | Loses global pose context, degrading ego-motion understanding | | w/o Geometry Tokens | 52.9 / 58.6 / 50.2 | 56.1 / 55.7 | Deprived of dense 3D structure, severely harming metric size estimation | | Fusion: Addition | 57.6 / 65.8 / 49.2 | 63.5 / 63.4 | Element-wise addition introduces feature contamination | | Fusion: Concat. + MLP | 53.5 / 60.3 / 50.1 | 60.8 / 62.1 | Linear projection limits cross-modal capacity | | Full Model (Cross-Attn.) | 59.8 / 65.4 / 51.4 | 65.7 / 66.0 | Cross-attention enables flexible querying of geometry, yielding best scores |

Token Compression Comparison (50% Retention Ratio): | Compression Strategy | Geometry-Aware | Stream3D-Bench (NA / MCA / OEA) | ScanQA (B-4 / CIDEr / EM) | Note | |---|---|---|---|---| | Random Pruning | No | 35.6 / 40.2 / 36.8 | 8.5 / 72.4 / 18.8 | Arbitrary dropping destroys 3D spatial coherence | | Average Pooling | No | 47.8 / 52.9 / 45.2 | 11.4 / 94.5 / 24.0 | Fixed downsampling blurs geometric object boundaries | | VisionZip | No | 49.2 / 53.8 / 41.6 | 13.0 / 87.2 / 24.4 | 2D attention-based pruning drops critical geometric tokens | | GAVC (Ours) | Yes | 59.8 / 65.4 / 51.4 | 17.8 / 104.5 / 30.9 | 3D coordinate clustering preserves spatial topology with minimal loss |

Key Findings

• Geometry priors are essential for physical metric reasoning: Removing geometry tokens incurs a ~14-point plunge in numerical answer accuracy on Stream3D-Bench, proving that pure 2D visual models cannot implicitly resolve spatial metrics.
• Spatially guided compression achieves extreme efficiency without sacrificing accuracy: GAVC preserves competitive accuracy even at 25% token retention, slashing latency to 0.39s (and 0.24s for the 4B variant), demonstrating high feasibility for resource-constrained edge robotics.
• Streaming loss weighting exhibits a clear sweet spot: \(\lambda=2.0\) yields optimal timing accuracy (86.7%) and language generation fluency; setting \(\lambda=3.0\) degrades response generation, whereas \(\lambda=1.0\) leads to sluggish responses (ATA drops to 80.3%).

Highlights & Insights

• Unified formulation: Stream3D-VLM elegantly integrates temporal interaction decisions into the LLM's autoregressive sequence modeling via <SEP> and <END>, avoiding external detection heads.
• Topology-preserving compression: The method re-thinks token pruning from a 3D perspective by unprojecting tokens into world space coordinates and applying GPU-accelerated spatial clustering.
• Benchmark and scalable data pipeline: It establishes Stream3D-Bench with 29 tasks and ATA temporal evaluation, alongside a 1M+ timestamped QA generation pipeline that addresses the lack of streaming 3D training data.

Limitations & Future Work

• Dependency on reconstruction quality: VSFI and GAVC rely directly on StreamVGGT's depth and camera estimations; rapid camera shakes, severe motion blur, or textureless surfaces may cause geometric drift and downstream errors.
• Default sampling rate constraint (1 FPS): The current implementation operates at 1 FPS, which may be insufficient for ultra-fast reactive robotics requiring millisecond-level reaction times.
• Extension to embodied action closed-loops: While currently outputting multimodal language and spatial groundings, coupling the model with continuous action chunk outputs is an exciting avenue for end-to-end embodied agents.

Related Work & Insights

• vs Offline 3D LMMs (e.g., 3D-LLM, LEO, LLaVA-3D): Offline methods require complete pre-scanned point clouds or static clips; Stream3D-VLM enables arbitrary online streaming interaction purely from RGB video.
• vs Online 2D Video Models (e.g., VideoLLM-online, VideoChat-Online): Online 2D models lack 3D physical coordinate awareness; Stream3D-VLM bridges this gap via VSFI geometry prior injection.
• vs Monocular Spatial Models (e.g., Spatial-MLLM, VLM-3R): Prior models are limited to offline evaluation and suffer from heavy decoding latency on long videos; Stream3D-VLM achieves superior accuracy on VSI-Bench while enabling streaming control and real-time execution via GAVC.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Pioneering framework establishing online 3D vision-language understanding on monocular streaming video.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 29 streaming tasks, offline VSI-Bench, and three canonical 3D scene benchmarks.
• Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical formalization, articulate architectural narrative, and clean experimental presentation.
• Value: ⭐⭐⭐⭐⭐ Foundational contribution providing practical real-time spatial intelligence for next-generation embodied agents and AR/VR systems.