Make Geometry Matter for Spatial Reasoning¶

Conference: ECCV2026
Paper: ECCV Paper
Project: GeoSR
Authors: Shihua Zhang, Qiuhong Shen, Shizun Wang, Tianbo Pan, Xinchao Wang
Area: VLM Spatial Reasoning
Keywords: geometry priors, visual token masking, gated fusion, dynamic spatial reasoning

TL;DR¶

GeoSR weakens 2D appearance shortcuts during training and allocates geometry contributions per token and channel during fusion, making an existing geometry branch useful for spatial question answering and achieving official average scores of 51.9 on VSI-Bench and 66.1 on DSR-Bench.

Background & Motivation¶

A vision-language model (VLM) can recognize a vehicle without reliably determining how its direction relative to another object changes as the camera moves. Static spatial reasoning does not necessarily mean viewing a single photograph: the scene may remain rigid while viewpoints, visible regions, and occlusions change throughout a video. Dynamic spatial reasoning additionally allows object motion, so distance, orientation, speed, and their temporal changes require evidence across frames. The difficulty is not a lack of object names, but that semantically similar images can represent different 3D relations. Earlier approaches introduce depth, point clouds, or explicit reconstruction, but additional sensors and multi-stage processing can limit applicability to monocular videos.

Recent methods instead extract implicit geometry tokens from pretrained 3D models and introduce them alongside ordinary vision tokens. VG-LLM represents geometry injection for static scenes, while GSM retrieves question-relevant geometric evidence for dynamic videos. Adding an information pathway, however, does not guarantee that the downstream model will use it: a pretrained VLM can still answer training questions through familiar 2D appearance cues. The paper's counterintuitive observation is that models with naive fusion are insensitive to the geometry branch, and removing geometry can even slightly improve dynamic performance. The question is therefore not only whether geometric features are good enough, but whether training encourages the model to use them.

GeoSR retains the existing visual, text, and geometry encoders and concentrates its changes on training inputs and feature fusion. Masking some appearance evidence during training pushes the model toward alternative evidence; fusion then allows geometry weights to vary across locations and channels. The former addresses the option of continually ignoring geometry, while the latter addresses the assumption that geometry should be mixed equally everywhere. This is neither simply a larger model nor a requirement to damage inputs at inference time, but a training intervention intended to change information use under complete inputs. Core Idea: break appearance shortcuts with visual token masking, then route geometric evidence into the VLM through geometry-guided gated fusion instead of treating the presence of geometry tokens as evidence of geometric reasoning.

Method¶

Overall Architecture¶

Inputs are an image sequence or video frames together with a text question; outputs are numerical or multiple-choice spatial answers. A visual encoder extracts appearance tokens, a geometry encoder extracts implicit 3D features from the same frames, and a text branch encodes the question. The static configuration uses Qwen2.5-VL-7B with VGGT, while the dynamic configuration uses the same VLM with π3. These are separate task configurations, not two geometry models invoked together during a single inference pass. The static branch reshapes and projects geometric features to match the spatial resolution and channel dimension of vision tokens. The dynamic branch first extracts compact geometric evidence through a question-conditioned QFormer, then uses it for mask selection and local fusion.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Video frames + text question"] --> Encoders["Frozen encoders<br/>Visual, text, geometry"]
    Encoders --> Mask["Geometry-Unleashing Masking<br/>Static random / dynamic TopK"]
    Encoders -->|Geometry features and question| Fusion["Geometry-Guided Fusion<br/>Dynamic redistribution + gating"]
    Mask -->|Partly zeroed during training<br/>Full vision at inference| Fusion
    Fusion --> Output["VLM spatial answers<br/>Dynamic compact evidence appended"]

Geometry-Unleashing Masking changes only the visual features passed to fusion during training; it does not remove inputs to the geometry branch. Geometry-Guided Fusion operates during both training and inference, and the dynamic configuration additionally appends compact geometric evidence to the fused tokens. The method thus retains both location-specific evidence and a global geometry summary filtered by the question. This summary is not a separately generated natural-language chain-of-thought; the model still answers directly from fused representations.

Key Designs¶

1. Geometry-Unleashing Masking: make appearance shortcuts intermittently unavailable during training

Masking operates on vision-token feature values, rather than requiring pixel reconstruction or removing geometry tokens. When masking is enabled for a training input, selected visual features are zeroed while other positions remain visible. The enable probability is \(\beta\), and the masking ratio when enabled is \(\gamma\); these control masking frequency and strength separately. The default settings are \(\beta=0.5\) and \(\gamma=0.8\), which do not mean that every input always loses 80% of its visual information. For static tasks, positions are sampled uniformly at random on the visual grid, while the geometry branch can still provide structural evidence at masked positions. Retaining unmasked training inputs lets the model preserve appearance semantics while learning to consult geometry when appearance is missing.

The important difference for dynamic tasks is that mask locations depend on the question. Learnable bottleneck tokens of length \(L_B=32\) first cross-attend to text tokens as keys and values, producing a question-conditioned representation. That representation queries channel-projected geometry tokens, yielding compact geometric evidence \(Z_G\) and cross-attention weights. For each geometry position, the paper averages these weights over all attention heads and bottleneck tokens, then applies min-max normalization with a numerical stability term. The resulting score measures attention relevance to the current question, not geometric confidence supervised by human annotations. TopK selection then identifies the highest-scoring positions and masks the corresponding 2D vision tokens, encouraging the model to recover the required evidence through geometry. The method deliberately masks relevant positions rather than preserving the most relevant visual regions; it is not conventional salient-region selection.

Let \(N\) denote the token-grid size currently used to select mask positions. The count relationship in Eqs. (7) and (10) can be written as:

\[ K=\lceil\gamma N\rceil. \]

When geometry and vision grids differ, spatial interpolation and projection first align geometry representations to the visual resolution before computing the corresponding relevance. Otherwise, a high-scoring position in the geometry grid need not correspond to the same location in the visual features, and TopK masking would miss its intended target. The method therefore depends on aligned geometric representations, rather than directly applying a low-resolution mask to a different branch. Masking is disabled at inference and all vision tokens are retained; geometric retrieval still supports dynamic fusion but no longer suppresses visual inputs.

2. Geometry-Guided Fusion: control geometry contributions across positions and channels

Masking alone is insufficient because simple addition or token appending can still let the VLM ignore the new pathway. For static reasoning, geometry tokens are first reshaped and projected into position-aligned features before gated fusion. Dynamic reasoning additionally needs to turn the compact summary \(Z_G\) into evidence suitable for position-wise fusion, so it introduces a third cross-attention operation. Its queries come from the aligned full geometry grid, while keys and values come from \(Z_G\); the output retains the full grid's token count. This does not explicitly reconstruct a 3D scene; it lets each position retrieve suitable information from question-relevant geometric evidence. Compact retrieval selects question-relevant dynamics, while redistribution assigns that evidence back to local positions.

Before gating, visual and geometry features undergo separate LayerNorm operations so that feature scale does not directly dominate learned weighting. The normalized features are concatenated along channels and passed through a learnable linear mapping and sigmoid to obtain a per-token, per-channel gate. The gate is not a single scalar for an entire video, nor a fixed geometry retention ratio shared by all positions. Corresponding features are mixed with complementary weights, allowing some channels at a position to retain appearance semantics while others rely more heavily on geometry. The dynamic configuration still appends \(Z_G\) after local fusion, avoiding reliance on position-wise redistribution as the only outlet for compressed geometric evidence. The static configuration feeds fused tokens and text tokens directly into the VLM, without this dynamic-summary appending pathway. Eqs. (12) through (15) are damaged in the text extraction, so this explanation follows the prose rather than guessing missing symbols and presenting them as the authors' exact formulas.

A Worked Example¶

The question in Figure 2 asks how the direction of the camera relative to a specified car changes between 2s and 4s, from the camera's viewpoint. A bounding box identifies the car, and answering requires spatial relations over the interval rather than merely recognizing a vehicle. The dynamic geometry branch first processes the frames, after which the question-conditioned bottleneck retrieves evidence relevant to the car, interval, and direction. If masking is enabled during training, 2D features at highly relevant positions are zeroed while the geometric evidence remains available. Redistribution assigns the question-relevant geometry summary back to positions, and the gate combines it with still-visible visual semantics. The VLM receives fused tokens, compact geometric evidence, and text, then predicts a direction option; the source figure illustrates the answer Behind. At inference, the same process keeps all visual features and does not need to hide the car deliberately before answering. This example explains the data flow; it does not imply that the paper reports actual per-token gate values or attention scores for this sample.

Loss & Training¶

The paper describes spatial question-answering fine-tuning without introducing a separate geometric reconstruction loss or reinforcement learning reward. Masking should therefore not be interpreted as an MAE-style pixel reconstruction objective; it intervenes on available evidence during question-answering training. Visual, text, and geometry tokenizers remain frozen, while the fusion module is trained and the VLM backbone is fine-tuned. The static configuration follows VG-LLM's SPAR-7M and LLaVA-Hound data splits, training for 1 epoch with Adam and a batch size of 64. Its learning rate is \(1\times10^{-5}\), with 150 warmup steps followed by cosine decay to 0. The dynamic configuration uses the official DSR-Train protocol, training for 1 epoch with Adam and a batch size of 32. Its learning rate is \(2\times10^{-7}\) with 50 linear warmup steps; the main text does not further specify the same complete decay schedule as for static training. Both configurations use 4 H200 GPUs with 141GB each, taking approximately 14 hours for static training and 20 hours for dynamic training. The former uses DeepSpeed ZeRO-2 and the latter ZeRO-3 Offload; these training costs should not be confused with single-GPU inference latency.

Key Experimental Results¶

Main Results¶

The following selection comes from Table 1 on page 11 and Table 2 on page 12, reporting only each benchmark's official Avg. VSI-Bench contains over 5k question-answer pairs from 288 videos: multiple-choice questions use accuracy, while numerical questions use mean relative accuracy across multiple relative-error tolerances. DSR-Bench contains 1484 question-answer pairs from 575 videos and reports per-type accuracy and an overall average under its official protocol; the two benchmarks' averages do not represent equal task difficulty.

Model	VSI-Bench Avg., Table 1	DSR-Bench Avg., Table 2
Qwen2.5-VL-7B	33.0	23.5
VG-LLM	50.7	38.4
GSM	Not listed	58.9
GeoSR	51.9	66.1

The static average exceeds the main-table VG-LLM result by 1.2 points, while the dynamic average exceeds the main-table GSM result by 7.2 points. These are model-level main-table comparisons, not evidence that every difference comes from one particular component. Static performance does not improve in every category: object size, for example, changes from 58.6 for VG-LLM to 57.4 for GeoSR.

Ablation Study¶

The following results correspond to Tables 3 and 4, both on page 13, where components are changed during training; every value is the corresponding benchmark Avg. Original fusion denotes the task configuration's existing naive geometry fusion; masking without geometry distinguishes geometry use from ordinary regularization.

Config	Visual masking	Geometry fusion	Static Avg.	Dynamic Avg.
(a) Full GeoSR	Yes	Gated	51.9	66.1
(b) Replaced fusion	Yes	Original fusion	50.0	64.7
(c) Masking only	Yes	No geometry	49.6	58.1
(d) Gated fusion only	No	Gated	50.9	64.9
(e) Naive geometry baseline	No	Original fusion	50.2	62.8
(f) No-geometry baseline	No	No geometry	49.8	64.0

Removing masking while keeping gating reduces static and dynamic scores by 1.0 and 1.2 points; replacing gated fusion with original fusion under masking reduces them by 1.9 and 1.4 points. Without geometry, adding masking lowers the dynamic score from 64.0 to 58.1, showing that masking is not universally beneficial augmentation by itself. The authors' trained dynamic naive-geometry baseline scores 62.8, not the main-table GSM score of 58.9; controlled component comparisons should use the former.

The following selection comes from Table 5 on page 14: geometry is removed or zeroed only after training, with inference-time visual masking disabled in every configuration.

Model and setting	Full input	Geometry removed	Geometry zeroed
Static naive geometry baseline	50.2	49.9	49.7
Static GeoSR	51.9	49.2	49.6
Dynamic naive geometry baseline	62.8	63.1	63.0
Dynamic GeoSR	66.1	63.2	65.0

Key Findings¶

Removing geometry lowers dynamic GeoSR by 2.9 points, whereas the naive baseline gains 0.3 points, supporting a change in how the model depends on geometry.
In Table 6 on page 15, with the other parameter fixed, increasing the masking enable probability from 0.5 to 0.7 lowers the dynamic average from 66.1 to 64.5; stronger masking is not always better.
Table 7 on page 15 reports times of 0.40s and 0.41s and peak memory of 18.81GB and 18.95GB for the naive geometry baseline and GeoSR on one H200; these are costs under that evaluation setting, not a general real-time guarantee.

Highlights & Insights¶

The paper makes use of an additional modality testable through intervention. Removing or zeroing geometry probes information utilization more directly than reporting only the score after adding geometry.
Geometry relevance guides masking toward appearance shortcuts that can substitute for geometric evidence. The intervention is neither indiscriminate noise nor a demand that geometry dominate semantics for every question.
Compressing and then redistributing dynamic evidence connects question conditioning to position-wise fusion. A transferable idea is to design both a retrieval bottleneck and local evidence outlets instead of merely appending features.

Limitations & Future Work¶

Reader assessment: the main evaluation covers one static and one dynamic spatial benchmark, which is insufficient to establish universal gains on open-world videos, long-horizon tasks, or robot control.
Reader assessment: geometry comes from frozen pretrained models, and the main text does not systematically quantify how geometric distortion, occlusion failures, or out-of-distribution motion affect gating reliability.
Reader assessment: degradation after removing geometry establishes dependence, but cannot alone prove dependence on correct 3D relations rather than feature-distribution changes; controlled geometric perturbations would provide stronger complementary evidence.
Source boundary: the supplied cache contains the main paper and references, but not the appendix mentioned in the text or a separate limitations section; the reader assessments above are not presented as author statements.
Extraction boundary: some formulas have damaged symbols, so this note preserves mechanisms and numbers supported by prose without inventing the exact gate orientation, unspecified loss terms, or additional experimental conclusions.

Compared with VG-LLM: both use implicit 3D priors, but GeoSR focuses on whether the VLM uses injected geometry, adding training-time masking and adaptive fusion.
Compared with GSM: GeoSR retains question-conditioned dynamic geometry retrieval, then additionally uses relevance to choose training masks and redistributes compact evidence to local positions for gating.
Compared with MAE: both mask visible information, but GeoSR does not optimize pixel reconstruction; it encourages spatial question answering to depend on another evidence pathway.
Research direction: perturb geometry while holding appearance fixed, then change texture while preserving geometry, to test whether answers follow spatial relations; this is a suggested follow-up, not a result reported in the paper.

Rating¶

Novelty: 4/5. The components are simple, but form a coherent intervention for the specific failure mode of ignored geometry.
Experimental Thoroughness: 4/5. Training-time and inference-time ablations complement each other, but evidence on cross-domain generalization and robustness to geometry errors is limited.
Writing Quality: 4/5. The problem and experiments align; damaged formula extraction limits exact reproduction without implying the original typesetting has the same defects.
Value: 4/5. The method provides a low-incremental-overhead path for improving geometry-enhanced VLMs, with particularly clear gains on dynamic tasks.