VoxAnchor: Explicit Voxel-Semantic Grounding for Spatial Understanding in Videos

Conference: ECCV 2026
Paper: ECCV 2026
Code: https://github.com/Embrace-Arch/VoxAnchor
Area: 3D Vision
Keywords: Video Spatial Reasoning, 3D Grounding, Voxel-Aware Attention, Multimodal LLM, Metric Awareness

TL;DR

To tackle spatial conflation and the lack of absolute metric scale caused by 2D projective collapse in Video MLLMs, VoxAnchor explicitly unprojects visual tokens and Depth Anything 3 (DA3) metric priors into a unified 3D voxel grid and constrains feature routing via Voxel-Aware Attention, establishing new SOTA results on VSI-Bench (66.8%) and VSTI-Bench (66.3%) with a compact 4B architecture.

Background & Motivation

Video spatial reasoning serves as a fundamental cornerstone of embodied spatial intelligence, enabling Multimodal Large Language Models (MLLMs) to actively perceive and reason within dynamic 3D physical environments from egocentric or mobile camera viewpoints. Beyond generic semantic perception such as recognizing objects and categorizing actions, this capability demands rigorous grounding of the physical world—including understanding complex 3D layouts, object metric scales, absolute distances, and continuous camera ego-motion trajectories. While single static images suffer from fundamental scale ambiguity due to projective geometry, continuous video sequences inherently provide temporal dynamics such as motion parallax and multi-view cues, serving as essential geometric constraints to construct persistent internal representations of 3D physical space.

However, standard multimodal architectures treat video input as a discrete sequence of isolated 2D frames, leaving cross-attention mechanisms bound to superficial 2D pixel-level adjacency rather than authentic 3D physical proximity. This projective collapse inevitably triggers severe "spatial conflation": models struggle to preserve object identity persistence under viewpoint shifts, frequently conflating distinct physical entities that share similar visual appearances at different 3D locations (e.g., misidentifying two different red cups as the same object), or entirely losing track of spatial coordinates across camera translations. Existing mitigation strategies either employ generative world models to imagine novel viewpoints—which introduce hallucinations and spatiotemporal inconsistencies via probabilistic sampling—or inject implicit 3D priors from foundation models like VGGT. Yet, these foundation models typically output relative unscaled reconstructions, and their features are easily overshadowed by the dominant semantic inductive biases of vision-language latent spaces.

Recognizing that a video stream is intrinsically a continuous projection of an evolving 3D environment onto a 2D temporal manifold, the authors argue that models must actively invert this projective process. Core idea: by establishing a decoupled dual-stream pipeline that extracts 2D semantic tokens and Depth Anything 3 metric geometric priors in parallel, VoxAnchor explicitly unprojects multimodal features into a unified 3D voxel grid via spatial hashing and enforces physical locality through Voxel-Aware Attention, eliminating misleading 2D planar adjacency to ground MLLM spatial reasoning in metric reality.

Method

Overall Architecture

The overall pipeline of VoxAnchor comprises four tightly integrated stages: dual-stream spatio-semantic encoding, a Geo-Semantic Adapter, 3D projection & voxelization, and Voxel-Aware Attention interaction. The framework takes a continuous monocular video sequence as input and generates accurate textual answers to metric-aware spatial queries.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    In["Input Video Sequence V (T Frames)"] --> DualStream

    subgraph DualStream["Dual-Stream Spatio-Semantic Encoding"]
        direction TB
        SemStream["Semantic Stream (Qwen3-VL Visual Encoder)<br/>Extract 2D semantic visual tokens Fsem"]
        GeoStream["Geometric Stream (Depth Anything 3)<br/>Regress metric depth D, intrinsics K, extrinsics E, geometric features Fgeo"]
    end

    DualStream --> Adapter["Geo-Semantic Adapter: Multi-Scale Refinement & Prototype Alignment<br/>Hierarchical fusion with instance prototype InfoNCE regularization"]

    SemStream --> Unproj["3D Projection & Voxelization: Inverse Projection to Metric World Space<br/>Analytical unprojection (d·K⁻¹[u,v,1]ᵀ) → s=0.5m voxelization & spatial hashing"]
    Adapter --> Unproj

    Unproj --> VAA["Voxel-Aware Attention: Physical Locality Masked Interaction<br/>Sparse mask M via Voxel ID matching (0 for co-location, -∞ otherwise)"]

    VAA --> LLM["MLLM Language Decoder (Qwen3-VL-4B LoRA)<br/>Output metric-aware spatial reasoning response A"]

The pipeline extracts semantic tokens and metric structural parameters concurrently, reconciles them across resolutions in the adapter, lifts all representations into world 3D coordinates, and feeds the voxel-constrained tokens to the language decoder.

Key Designs

1. Dual-Stream Spatio-Semantic Encoding: Decoupling Semantics from Absolute Metric Priors

Directly entangling geometric features with high-level visual tokens in early layers causes vision-language models to suppress fine-grained geometric coordinates under strong semantic inductive biases. VoxAnchor employs a decoupled dual-stream encoding strategy: the Semantic Stream utilizes a pre-trained Qwen3-VL visual encoder to extract 2D visual tokens \(F_{sem} \in \mathbb{R}^{T \times \frac{H}{16} \times \frac{W}{16} \times C_s}\); simultaneously, the Geometric Stream introduces Depth Anything 3 (DA3) as a deterministic inverse-rendering foundation model. In a single forward pass, DA3 directly regresses absolute metric depth maps \(D \in \mathbb{R}^{T \times H \times W}\), camera intrinsic matrices \(K\), and camera extrinsic ego-motion trajectories \(E\), while extracting intermediate multi-scale geometric representations \(F_{geo} \in \mathbb{R}^{T \times \frac{H}{14} \times \frac{W}{14} \times C_g}\) from Layers 9, 25, and 39. This decoupled design preserves clean metric physical hypotheses entirely uncorrupted by early semantic abstraction.

2. Geo-Semantic Adapter: Multi-Scale Structural Injection and Prototype Regularization

To bridge the representational gap between DA3 geometric features (rich in 3D geometry but semantically agnostic) and Qwen3-VL visual tokens (rich in categorical semantics but lacking metric depth), the Geo-Semantic Adapter first forms a bottleneck representation via a fusion block, then progressively modulates the semantic stream via gated skip connections across 1/7, 1/4, and 1/1 resolutions using RGB texture and depth cues. The resulting features are refined through a \(1 \times 1\) convolution and \(L_2\) normalization into a continuous metric spatial feature field. To ensure instance-level cross-frame persistence under rapid camera ego-motion, the adapter is contrastively pre-trained using instance masks to dynamically aggregate prototypes \(p_k\), supervised by a proxy-based InfoNCE objective:

\[ \mathcal{L}_{info} = -\frac{1}{N_{fg}} \sum_{i=1}^{N_{fg}} \log \frac{\exp(f_i \cdot p_{y_i} / \tau)}{\sum_{j=1}^{K} \exp(f_i \cdot p_j / \tau)} \]

where \(N_{fg}\) denotes foreground pixels, \(K\) is the dynamic instance count, and \(\tau\) is the temperature hyperparameter. Combined with intra-instance compactness and inter-instance margin penalties, this regularization guarantees that visual entities maintain geometric integrity across disparate temporal views.

3. 3D Projection and Voxelization: Inverse Projection to Metric World Space

To ground projective pixels into physical reality, VoxAnchor employs an analytical inverse projection operator using DA3 predicted depths and camera poses. For any token at pixel coordinate \((u, v)\) with predicted depth \(d\), camera intrinsics \(K\), and extrinsic matrix \(E\), its absolute world coordinate \(p_{world} \in \mathbb{R}^3\) is derived as:

\[ p_{world} = E^{-1} \left( d \cdot K^{-1} [u, v, 1]^T \right) \]

The continuous 3D world space is then discretized into a regular volumetric grid with a voxel resolution of \(s = 0.5\text{ m}\). A spatial hashing function assigns globally unique Voxel IDs (\(H\)) to both the unstructured visual tokens and the dual-stream GeoSem embeddings, anchoring multimodal features to precise physical addresses.

4. Voxel-Aware Attention: Physical Locality Masked Interaction

To fundamentally replace misleading 2D planar contiguity with 3D physical proximity, VoxAnchor restructures the cross-attention mechanism prior to language model reasoning. Unprojected 2D visual tokens serve as Queries (\(Q\)), while the 3D-consistent GeoSem features serve as Keys (\(K\)) and Values (\(V\)). Guided by matching Voxel IDs between Queries and Keys, a sparse voxel mask \(M\) is dynamically constructed:

\[ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}} + M\right) V \]

The spatial penalty term enforces a strict physical gating constraint: \(M_{i,j} = 0\) if \(H_i = H_j\) (indicating identical volumetric occupancy), and \(M_{i,j} = -\infty\) otherwise. Extended to multi-head attention across all heads, this mechanism prunes non-local interactions and forces the model to synthesize spatial intelligence strictly from geometrically valid 3D physical neighborhoods.

Loss & Training

The framework is optimized in two stages: 1. Stage 1 (Adapter Pre-training): The Geo-Semantic Adapter is contrastively pre-trained on InsScene-15K using DA3 features from Layers 9, 25, and 39 on a single NVIDIA L40 GPU, optimized via the InfoNCE loss with instance margin penalties. 2. Stage 2 (Multimodal Spatial Fine-Tuning): Fine-tuned on VSI590K and VSTI-Bench training splits for 2 epochs. The Qwen3-VL visual encoder, DA3 backbone, and pre-trained adapter are frozen; only the Voxel-Aware Attention module and language model LoRA parameters (rank \(r=128\), \(\alpha=256\)) are updated. Optimization is performed using AdamW with learning rate \(1 \times 10^{-4}\) and global batch size 64 across 4 NVIDIA A800 GPUs.

Key Experimental Results

Main Results

VoxAnchor establishes new state-of-the-art benchmarks on both VSTI-Bench (temporal dynamics and camera pose estimation) and VSI-Bench (comprehensive 3D spatial cognition and metric estimation), surpassing much larger models.

Table 1: Evaluations on VSTI-Bench for 3D spatiotemporal reasoning tasks

Category	Methods	Avg.	Cam-Obj Abs Dist	Camera Displace	Camera Mov Dir	Obj-Obj Rel Pos	Cam-Obj Rel Dist
Proprietary API	GPT-4o	38.2	29.5	23.4	37.3	58.1	42.5
Proprietary API	Gemini-1.5 Flash	32.1	28.5	20.9	24.4	52.6	33.9
Open-source VLM	InternVL2-40B	43.2	11.9	34.9	33.3	63.8	72.2
Open-source VLM	Qwen2.5-VL-72B	40.3	18.0	10.0	41.0	74.2	58.4
Open-source VLM	LLaVA-NeXT-Video-72B	44.0	32.3	10.5	48.1	78.3	50.9
Open-source VLM	Qwen3-VL-4B (Base)	39.7	29.5	32.9	46.0	68.6	25.2
Spatial-Enhanced	VLM-3R-7B	58.8	39.4	39.6	60.6	86.5	68.6
Ours	VoxAnchor-4B (Ours)	66.3	40.5	46.1	83.6	89.5	71.9

Table 2: Comparative results on VSI-Bench across numerical (NC) and multiple-choice (MCA) queries

Category	Methods	Avg.	Obj. Cnt.	Abs. Dist.	Obj. Size	Room Size	Rel. Dist.	Rel. Dir.	Route Plan	Appr. Order
Proprietary	GPT-4o	34.0	46.2	5.3	43.8	38.2	37.0	41.3	31.5	28.5
Proprietary	Gemini-1.5 Pro	45.4	56.2	30.9	64.1	43.6	51.3	46.3	36.0	34.6
Open-source	Qwen3-VL-4B	54.8	62.6	44.5	70.3	63.9	56.4	46.3	32.0	62.3
Spatial-Enhanced	VG-LLM-8B	50.7	67.9	37.7	58.6	62.0	46.6	40.7	32.4	59.2
Spatial-Enhanced	VLM-3R-7B	60.9	70.2	49.4	69.2	67.1	65.4	80.5	45.4	40.1
Spatial-Enhanced	GS-Reasoner-7B	64.7	69.1	61.9	70.0	65.7	65.4	88.9	44.3	52.3
Ours	VoxAnchor-4B (Ours)	66.8	66.3	52.6	72.5	71.0	67.3	80.0	46.4	78.0

Ablation Study

Ablation experiments on VSTI-Bench isolate the contributions of the 3D geometric foundation model and core structural modules.

Table 3: Ablation of 3D foundation models on VSTI-Bench

Config	Avg.	Cam-Obj Abs. Dist.	Cam. Displace	Cam. Mov. Dir.	Obj-Obj Rel. Pos.	Cam-Obj Rel. Dist.	Note
Baseline (Qwen3-VL-SFT)	54.4	36.0	32.7	53.6	79.1	70.6	Supervised fine-tuning without 3D priors
+ VGGT	59.5	38.6	40.5	62.3	86.4	70.0	Unscaled relative geometric representation
+ IGGT	60.7	38.0	41.1	65.8	87.3	71.3	Instance-grounded relative 3D geometry
+ DA3 (Depth Anything 3)	63.0	40.0	43.9	73.2	87.1	71.0	Deterministic metric depth & pose prior

Table 4: Ablation of structural components on VSTI-Bench (fixed DA3 backbone)

Voxel-Aware Attention	Geo-Semantic Adapter	Avg.	Cam-Obj Abs. Dist.	Camera Displace	Camera Mov. Dir.	Obj-Obj Rel. Pos.	Cam-Obj Rel. Dist.	Note
✗	✗	63.0	40.0	43.9	73.2	87.1	71.0	Standard cross-attention without voxel masking
✓	✗	64.5	39.4	43.0	79.2	89.9	73.0	Enforcing physical locality via voxel mask (+1.5%)
✓	✓	66.3	40.5	46.1	83.6	89.5	71.9	Full model with multi-scale prototype adapter (+1.8%)

Key Findings

• Crucial Role of Absolute Metric Priors: As shown in Table 3, DA3 with explicit metric scaling (63.0%) significantly outperforms scale-ambiguous models like VGGT (59.5%) and IGGT (60.7%). In Camera Movement Direction, DA3 delivers a leap from 53.6% to 73.2%, proving that absolute metric references are indispensable for stable physical coordinate frames.
• Voxel Masking Directly Mitigates Spatial Conflation: Table 4 demonstrates that replacing unconstrained cross-attention with Voxel-Aware Attention immediately improves Camera Movement Direction from 73.2% to 79.2% (+6.0%) and Object-Object Relative Position from 87.1% to 89.9% (+2.8%), validating that pruning false 2D planar contiguity is vital for spatial reasoning.
• Superiority of Geometric Grounding over Parameter Scaling: At only 4B parameters, VoxAnchor achieves 52.6% on Absolute Distance estimation, outperforming GPT-4o (5.3%) and InternVL2-40B (11.9%) by massive margins, proving that spatial capability is driven by explicit structural inductive bias rather than mere parameter scale.

Highlights & Insights

• Lifting Video Understanding into Explicit 3D Voxel Space: Bypasses the traditional limitation of flattening video frames by analytically unprojecting tokens into a metric-aware 3D grid with spatial hashing, providing stable physical anchor coordinates across temporal sequences.
• Hard Volumetric Constraints Replacing Soft Attention: Setting attention weights of non-overlapping voxels to \(-\infty\) provides an ultra-lightweight discrete spatial filter that strictly prevents spatial conflation while maintaining high inference efficiency.
• Harmonizing Metric Geometry with Multi-Scale Semantics: The contrastively pre-trained Geo-Semantic Adapter bridges the gap between metric depth and semantic vision-language representations, offering a plug-and-play foundation for embodied navigation and manipulation.

Limitations & Future Work

• Dependency on Monocular Geometric Estimation Quality: The accuracy of inverse projection remains bounded by DA3 depth and camera pose predictions; extreme illumination changes or textureless surfaces may introduce pose drift.
• Fixed Grid Resolution Trade-off: The current fixed \(s = 0.5\text{ m}\) voxel size excels at room-scale ego-motion and layout reasoning but is coarse for millimeter/centimeter-level tabletop object manipulation; hierarchical octree or adaptive sparse grids represent promising extensions.
• Long-Video Memory Scalability: As video duration extends, the global voxel hash map expands; local sliding-window attention or spatiotemporal topological graphs should be explored for extended embodied roaming.

Related Work & Insights

• vs VG-LLM / Spatial-MLLM: These models feed VGGT features as implicit tokens into LLMs, remaining subject to relative scale ambiguity; VoxAnchor leverages DA3 metric depth and explicit voxel inverse projection to dominate in absolute distance and size measurement tasks.
• vs GS-Reasoner: GS-Reasoner depends on real-time VGGT-SLAM to build point clouds and runs PTv3 for feature extraction, which is computationally heavier; VoxAnchor uses feed-forward dual-stream unprojection with hash masking, offering a cleaner and more efficient pipeline.
• vs Cambrian-S: While Cambrian-S pursues self-supervised next-latent-frame prediction without auxiliary encoders, VoxAnchor proves that explicit geometric foundation priors allow a compact 4B model to achieve superior spatial precision with far less training data.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Explicit voxel unprojection and voxel-aware attention elegantly eliminate 2D spatial conflation.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Sets new SOTAs on both VSTI-Bench and VSI-Bench with rigorous modular and foundation model ablations.
• Writing Quality: ⭐⭐⭐⭐⭐ Mathematically rigorous, clearly structured, and motivated by fundamental physical principles.
• Value: ⭐⭐⭐⭐⭐ Establishes a highly practical, parameter-efficient paradigm for 3D embodied spatial intelligence in MLLMs.