ConsiSpace: Learning Geometric Consistency Matters for Video Spatial Reasoning

Conference: ECCV2026
Paper: https://eccv.ecva.net/virtual/2026/poster/4462
PDF: https://media.eventhosts.cc/Conferences/ECCV2026/pdfs/6027.pdf
Area: Vision-Language Reasoning / Video Spatial Reasoning
Keywords: Geometry-Consistent Memory, Cross-View Consistency, Hierarchical Retrieval, Self-Supervised Reinforcement Learning, Long Video Understanding

TL;DR

ConsiSpace controls the writing, fusion, and retrieval of video evidence using camera poses and depth, followed by post-SFT training with three types of cross-view consistency rewards (answer, metric, and topology). It achieves scores of 76.6, 53.0, and up to 58.1 on VSI-Bench, OSI-Bench, and MMSI-Video-Bench, respectively.

Background & Motivation

Video spatial reasoning requires not only recognizing what is in the scene, but also judging object orientation, distance, size, scene connectivity, and proximity order under camera movement, occlusion, and long time spans. General multimodal large models are typically centered on semantic recognition. Even when geometric foundation models such as DUSt3R or VGGT are introduced, common practices merely treat 3D features as additional inputs or supervision signals, without answering when evidence in long videos should be written, which duplicate observations can be merged, and which views should be retrieved when answering questions.

This leads to two interrelated problems. First, adjacent frames often experience only minimal displacement or rotation. Keeping every frame in the context increases GPU memory and inference time without necessarily bringing new evidence. Second, the same static spatial relationship should remain invariant across different observation windows, but redundant or mismatched-view evidence can cause the model to output inconsistent answers to the same question. Merely expanding the context cannot resolve the issue of evidence quality, and solely supervising the correct answer does not explicitly constrain cross-view stability.

ConsiSpace applies geometric invariance in static scenes to both evidence management and model learning: the former compresses and filters memory based on poses, depth, and orientations, while the latter constructs consistency rewards from two observations of the same question after SFT without needing additional human annotation. Core Idea: Instead of treating geometry merely as an auxiliary feature fed into the model, the principle that "cross-views should be consistent" is made the common foundation for managing the entire video evidence and optimizing the final answers.

Method

Overall Architecture

The input is a video \(V\) and a question \(q\), and the output is the answer generated by the language model. A frozen SigLIP2 extracts semantic visual tokens from each frame, and a frozen VGGT extracts spatial tokens, camera poses, and depth maps. These two types of implicit tokens and explicit geometric cues are temporally aligned and entered into the Geometry-Consistent Memory (GCM). Rather than storing all frames, GCM decides whether to write based on camera motion, fuses geometrically proximal and semantically similar observations, and performs hierarchical retrieval (retrieving clips first, then filtering frames) during question answering.

The retrieved local evidence, together with global topological and metric summary tokens, is fed into Qwen3-VL-8B-Instruct. The training consists of two stages: SFT first anchors task correctness, and then Unified Consistency Self-Supervised Reinforcement Learning (UC-SSRL) enforces consistency in answers, numerical metrics, and spatial relationships when predicting the same question across two different time windows or retrieval contexts.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
        Input["Video Frames + Question"] --> Enc["Dual-Encoded Geometric Evidence"]
        Enc --> Memory["Geometry-Gated Writing & Fusion"]
        Memory --> Retrieval["Hierarchical Geometrically-Filtered Retrieval"]
        Retrieval --> VLM["Qwen3-VL-8B Answer Generation"]
        VLM --> Train["Unified Cross-View Consistency Learning"]
        Train --> Output["Stable Spatial Answer"]

Key Designs

1. Dual-Encoded Geometric Evidence: Separating Answerable Content from Verifiable Geometry

Relying solely on visual semantic tokens makes it difficult for the model to judge if two views describe the same spatial location despite knowing "this is a bed and a TV." Conversely, relying solely on poses and depth is insufficient to answer natural language questions. Therefore, ConsiSpace uses SigLIP2 to generate visual semantic tokens, and VGGT to generate spatial tokens, camera poses \(T_t\), and depth maps \(D_t\). Visual and spatial tokens are concatenated and projected into fixed-length implicit evidence entries, while poses and depth serve as concurrent explicit geometric entries.

This dual memory is not two independent knowledge bases: implicit entries carry "what is in the frame" for semantic retrieval and language model consumption, while explicit entries carry "from where and toward where it is viewed" to control subsequent writing, fusion, and filtering. Decoupling content from control signals avoids directly converting geometric errors into linguistic answers, acting instead as a soft constraint for evidence organization.

2. Geometry-Gated Writing and Fusion: Expanding Memory Only When Views Truly Change

For the \(t\)-th frame, the system obtains the position \(p_t\) and viewing direction \(v_t\) from the camera pose, comparing the translation and rotation of adjacent frames. Writing occurs only when at least one change exceeds a threshold:

\[ \operatorname{write}(t)=\mathbb{I}\left[\lVert p_t-p_{t-1}\rVert_2>\tau_p\ \lor\ \angle(v_t,v_{t-1})>\tau_\theta\right]. \]

The default writing thresholds are set to \(0.15\) m and \(15^\circ\). This step skips redundant frames in smooth motions. However, "movement occurred" still does not mean a new entry must be added: the system also searches for existing observations within a geometric neighborhood defined by a spatial radius \(r\) and orientation difference \(\theta\). It then selects the entry with the highest semantic cosine similarity to the new evidence within that neighborhood for fusion; a new entry is appended only if the neighborhood is empty. The default fusion neighborhood is \(0.40\) m and \(30^\circ\), allowing repetitive observations of the same area to converge into compact evidence while retaining new information from cross-region transitions or large rotations.

3. Hierarchical Geometrically-Filtered Retrieval: Preserving Both Global Layout and Question-Relevant Details

Compact memory may still contain entries irrelevant to the current question or with conflicting orientations. ConsiSpace first retrieves the top-\(K\) candidate clips based on the semantic similarity between the question and clip keys, and then rates each frame within these candidates:

\[ s_f(i)=\alpha\cos(q_e,k_i^f)+\beta\cos(Q_{\mathrm{topo}},e_i^{\mathrm{dir}}). \]

The first term finds evidence semantically relevant to the question, while the second term checks the viewing orientation using a direction query derived from the question. Thus, a frame containing "television" will be downweighted or filtered out if its geometric view conflicts with the query. In practice, top-8 is used for both clip-level and frame-level retrieval, and the filtered implicit evidence along with its aligned geometric cues are sent to the language model.

Sparse top-\(K\) retrieval tends to lose overall room layouts. Therefore, the model also aggregates a topological summary and a metric summary from the memory: the former focuses on directions and connectivity, while the latter focuses on distance, size, and sequence. The final context is constructed by concatenating the question tokens, the two summaries, local evidence, and corresponding geometric cues. Ablations show that using only summaries lacks details, and using only top-\(K\) lacks global structure; combining both yields the most balanced results.

4. Unified Cross-View Consistency Learning: Mutually Supervising with Two Observations of the Same Scene

GCM can clean up evidence but cannot guarantee that the language model yields the same answer when facing two different time windows or retrieval results. UC-SSRL randomly samples two observations for the same \((V, q)\), obtaining prediction distributions \(\pi_i\) and \(\pi_j\), and continues optimizing three complementary rewards on the SFT model. Answer consistency penalizes the KL divergence between the two prediction distributions; metric consistency penalizes the discrepancy between the two distance estimates; and topological consistency constraints the two relationship distributions using bidirectional KL:

\[ R=-\alpha\operatorname{KL}(\pi_i\Vert\pi_j)-\eta|\hat d_i-\hat d_j|-\beta\left[\operatorname{KL}(P_i\Vert P_j)+\operatorname{KL}(P_j\Vert P_i)\right]. \]

Since consistency itself does not equal correctness, the authors first perform maximum-likelihood SFT and then use UC-SSRL to stabilize cross-view predictions. Only the LoRA parameters are updated to prevent the entire model from drifting due to self-supervised signals. The three rewards cover categorical answers, continuous metrics, and relational structures, respectively. In reward ablations, any single reward outperforms SFT alone, while using all three achieves the best performance across four summary metrics.

A Complete Example

Suppose the camera in a video moves slowly along a bedroom, and the question is "When standing by the desk facing the bed, which direction is the TV located?". SigLIP2 records the semantics of the bed, desk, and TV, while VGGT estimates the camera trajectory, depth, and orientation. Continuous small movements below \(0.15\) m or \(15^\circ\) are not written. New frames exceeding these thresholds are fused with the most similar entry if they still fall within the existing \(0.40\) m and \(30^\circ\) neighborhood; otherwise, a new entry is created.

When answering, the top-8 candidate clips are first retrieved from 4 memory clips, followed by selecting the top-8 frames within these clips by combining the semantics of "desk, bed, TV" and the query's implied orientation. The topological summary provides relative directions in the room, the metric summary supplements distance relationships, and local frames yield object details. In the training phase, the same question is answered again from another time window. If the two trials predict "front-right" and "back-left", both the answer and topological rewards will penalize this inconsistency, rather than treating them as unrelated samples.

Loss & Training

The model uses Qwen3-VL-8B-Instruct as the language backbone. SigLIP2, VGGT, the vision tower, and projectors are all frozen. Both SFT and UC-SSRL train only LoRA with rank 16 and \(\alpha=32\). The SFT data includes VSI-590K, along with nuScenes-10K filtered from 22,518 candidates based on geometric validity, answer parseability, consistency, MLLM verification, and deduplication. Manual inspection of 1,000 random samples from this dataset showed an accuracy of 91.6%.

Training is conducted in bfloat16 with DeepSpeed ZeRO-3 on 8 A100 80GB GPUs for 200 steps, with a learning rate of \(1\times10^{-4}\), a per-device batch size of 1, gradient accumulation of 2, and a maximum sequence length of 5,120. UC-SSRL is initialized from the SFT checkpoint with an overall reward weight of \(10^{-2}\) and reuses the same training pool without adding manual consistency annotations.

Key Experimental Results

Main Results

Benchmark / Setting	Qwen3-VL-8B	Strong Specialized Baseline	ConsiSpace SFT	ConsiSpace UC-SSRL	Relative Gain over Strong Baseline
VSI-Bench Avg.	57.4	SpaceMind 69.6	71.2	76.6	+7.0
OSI-Bench Avg.	31.2	VLM-3R 40.3	45.8	53.0	+12.7
MMSI Sufficient-Coverage Avg.	29.1	VLM-3R 42.6	48.4	57.5	+14.9
MMSI Uniform-50 Avg.	27.6	VLM-3R 43.1	48.9	58.1	+15.0

GCM Ablation

Configuration	VSI	OSI	MMSI-Video	Normalized Memory Entries
Full ConsiSpace SFT	71.2	45.8	52.3	1.0×
Without Gated Writing	69.8	44.5	47.2	2.3×
Without Consistent Fusion	70.3	45.0	47.9	1.8×
Without Geometric Filtering	69.5	43.9	46.0	1.0×

UC-SSRL Reward Ablation

Answer / Metric / Topological Reward	VSI	OSI	MMSI-SC	MMSI-U50
No / No / No (SFT)	71.2	45.8	48.4	48.9
Yes / No / No	73.5	48.2	52.1	52.6
No / Yes / No	72.8	49.0	51.4	51.9
No / No / Yes	72.9	47.1	51.7	52.2
Yes / Yes / Yes	76.6	53.0	57.5	58.1

Key Findings

• Under the same architecture, UC-SSRL improves by 5.4, 7.2, 9.1, and 9.2 points on VSI, OSI, MMSI-SC, and MMSI-U50 respectively compared to SFT. Tasks with stronger view variations and metric relationships, such as OSI and MMSI, benefit more.
• The three lifecycle steps of GCM serve different purposes: gating and fusion primarily reduce memory entries, while geometric filtering does not change the number of entries but contributes 6.3 points on MMSI-Video, showing that both "storing less" and "retrieving accurately" are indispensable.
• With 200 frames of input, ConsiSpace uses 35.0GB GPU memory and requires 3.10s inference time, whereas VLM-3R uses 78.5GB and requires 10.80s. Under this setup, ConsiSpace scores 72.4/46.8/54.4/55.1 on VSI/OSI/MMSI-SC/MMSI-U50, outperforming VLM-3R's 69.5/43.5/51.1/52.3.
• After adding 30% noise to the VGGT geometry, ConsiSpace still achieves 73.1/49.6/53.4/54.3, though this is a noticeable drop compared to clean inputs (76.6/53.0/57.5/58.1), indicating that geometric quality remains an important factor limiting performance.

Highlights & Insights

• The most ingenious aspect of the paper is integrating the "geometric consistency" principle throughout two levels of the system: on the input side to decide how evidence enters memory, and on the output side to ensure answers from two views can explain each other. This prevents the geometric module from operating independently of the training targets.
• Gated writing and fusion shift long-video memory from "retaining more frames" to "only retaining mutations in the spatial state." This strategy can be transferred to robotic inspection, first-person life logging, and autonomous driving history caching, by simply replacing the pose and orientation thresholds with task-relevant state change criteria.
• The readout design combining summaries and top-\(K\) retrieval is highly worth reusing: summaries capture the global layout, while sparse retrieval preserves local evidence. Its ablation results demonstrate better than merely increasing the retrieval count that long-context systems need to distinguish global structure from local details.
• UC-SSRL does not rely on additional manual consistency labels, but it must be built upon the correctness anchor of SFT. This serves as a reminder that self-consistency is suitable for reducing prediction variance, yet cannot guarantee that two consistent answers are both correct on its own.

Limitations & Future Work

• The main text of the paper does not have a separate limitation section. Based on its setup, GCM depends on the poses, depth, and orientations output by the frozen VGGT. Although noise experiments indicate graceful degradation rather than sudden collapse, a 3.5 to 4.1 point drop in the four metrics under 30% noise suggests that wrong writing, incorrect fusion, or filtering out correct evidence can still occur in dynamic scenes, strong occlusions, or when geometric estimation fails.
• The method primarily assumes static 3D relationships under smooth view changes. When objects move on their own, the assumption that "similar observations near the same location should be fused" may become obsolete, and the explicit memory lacks object state expiration, motion trajectory updating, and conflict resolution mechanisms. Dynamic object tracking and temporal decay could be incorporated in future work.
• The UC-SSRL reward optimizes cross-view consistency rather than direct correctness. Although SFT provides an anchor, two views can still consistently produce the same incorrect answer. Verifiable geometric constraints, answer confidence, or a small proportion of calibration rewards with ground truth could be further introduced.
• The training cost remains high (8×A100 80GB), and only Qwen3-VL-8B-Instruct is reported as the backbone. Transferability across model scales, different geometric encoders, and longer real-world videos has not been covered by current experiments.
• The 91.6% accuracy rate in the 1,000 sampled items of nuScenes-10K indicates that automatically constructed data still contains noise. A more detailed audit of error types, as well as a potential similarity analysis between the training data and OSI-Bench at the scene-category level, would make the generalization conclusions more solid.

Related Work & Insights

• vs VLM-3R / Spatial-MLLM: These methods inject reconstruction or 3D representations into MLLMs, focusing on "providing geometric information." ConsiSpace further uses geometry to control the complete lifecycle of long-term evidence, yielding particularly noticeable memory and latency advantages with 200 frames.
• vs SpaceMind / GeoThinker: These focus on camera-guided fusion or active geometric integration. ConsiSpace differs by explicitly determining when to write, with what to fuse, and which views to filter during retrieval, while constraining final predictions with cross-view rewards.
• vs Spatial-SSRL: Spatial-SSRL relies on spatial intrinsic pretext tasks, while UC-SSRL directly generates answer, metric, and topological consistency rewards from two observations of the same video question, which is closer to view stability in video spatial QA.
• Insights for Memory-Augmented VLMs: External memory should not merely be a repository of tokens. If verifiable invariants exist within the domain (such as spatial geometry, object identity, or temporal causality), they can be leveraged to simultaneously constrain writing, merging, retrieval, and training.

Rating

• Novelty: ⭐⭐⭐⭐☆ Unifying geometric consistency for both memory lifecycle and post-SFT reinforcement learning. The combination perspective is clear and systematic.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Three main benchmarks, navigation verification, efficiency scaling, component and reward ablations, as well as threshold and noise sensitivity are all covered.
• Writing Quality: ⭐⭐⭐⭐☆ Completeness in main threads and formulas, well-organized experiments. However, the main text lacks a dedicated discussion on limitations, and some implementation details are left to the supplementary materials.
• Value: ⭐⭐⭐⭐⭐ Simultaneously improves the accuracy, cross-view stability, and resource efficiency of long-video spatial reasoning, offering strong reusability for memory-augmented VLMs.

Related Papers

• [ECCV 2026] Towards Spatial Trace with Reasoning in Vision-Language Models for Robotics
• [ECCV 2026] MonoSR: Open-Vocabulary Spatial Reasoning on Monocular Images
• [ECCV 2026] RoadBench: Benchmarking MLLMs on Fine-Grained Spatial Understanding and Reasoning under Urban Road Scenarios
• [ECCV 2026] EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoning
• [ECCV 2026] HyLaR: Hybrid Latent Reasoning with Decoupled Policy Optimization