ConsiSpace: Learning Geometric Consistency Matters for Video Spatial Reasoning¶

Conference: ECCV2026
Paper: https://eccv.ecva.net/virtual/2026/poster/4462
PDF: https://media.eventhosts.cc/Conferences/ECCV2026/pdfs/6027.pdf
Area: Visual-Language Reasoning / Video Spatial Reasoning
Keywords: Geometry-Consistent Memory, Cross-View Consistency, Hierarchical Retrieval, Self-Supervised Reinforcement Learning, Long Video Understanding

TL;DR¶

ConsiSpace utilizes camera poses and depth to control the writing, fusion, and retrieval of video evidence, followed by post-SFT training using answer, metric, and topological cross-view consistency rewards. It achieves scores of 76.6, 53.0, and up to 58.1 on VSI-Bench, OSI-Bench, and MMSI-Video-Bench, respectively.

Background & Motivation¶

Video spatial reasoning requires not only recognizing what is in the scene, but also reasoning about object orientation, distance, size, scene connectivity, and proximity order under camera movement, occlusion, and long time horizons. General multimodal large language models (MLLMs) are typically semantic-centric. Even when geometric foundation models like DUSt3R or VGGT are introduced, the common practice is merely to treat 3D features as extra inputs or supervision signals. This fails to address when evidence in long videos should be written, which redundant observations can be fused, and which views should be retrieved when answering a question.

This leads to two interconnected problems. First, adjacent frames often possess minor displacements or rotation angles; keeping every frame in the context increases GPU memory usage and inference overhead without necessarily introducing new evidence. Second, static spatial relations should inherently remain invariant across different observation windows, but redundant or view-mismatched evidence can cause the model to yield inconsistent answers to the same query. Simply expanding the context length does not address the issue of evidence quality, and solely supervising correct answers does not explicitly constrain cross-view stability.

ConsiSpace utilizes geometric invariance in static scenes for both evidence management and model learning: the former compresses and filters memories based on camera pose, depth, and orientation, while the latter constructs consistency rewards from two different observations of the same question after SFT, without requiring human annotations. Core Idea: Instead of treating geometry merely as an auxiliary feature fed into the model, "cross-view consistency" is formalized as a unified principle for both managing the entire video evidence and optimizing the final predictions.

Method¶

Overall Architecture¶

The inputs are a video \(V\) and a question \(q\), and the output is the answer generated by the language model. A frozen SigLIP2 extracts semantic visual tokens from each frame, while a frozen VGGT extracts spatial tokens, camera poses, and depth maps. These two types of implicit tokens and explicit geometric cues are temporally aligned and stored in the Geometry-Consistent Memory (GCM). GCM does not retain all frames; instead, it determines whether to write based on camera motion, fuses geometrically proximal and semantically similar observations, and retrieves segments before filtering individual frames during question answering.

The retrieved local evidence is fed into Qwen3-VL-8B-Instruct alongside global topological and metric summary tokens. The training process consists of two stages: Supervised Fine-Tuning (SFT) to anchor task accuracy, and Unified Consistency Self-Supervised Reinforcement Learning (UC-SSRL) to enforce consistency in answers, numerical metrics, and spatial relationships of the same question across two different time windows or retrieval contexts.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
        Input["Video Frames + Question"] --> Enc["Dual-Encoded Geometric Evidence"]
        Enc --> Memory["Gated Geometric Writing & Fusion"]
        Memory --> Retrieval["Hierarchical Geometrically Filtered Retrieval"]
        Retrieval --> VLM["Qwen3-VL-8B Answer Generation"]
        VLM --> Train["Unified Cross-View Consistency Learning"]
        Train --> Output["Consistent Spatial Answers"]

Key Designs¶

1. Dual-Encoded Geometric Evidence: Decoupling Queryable Content from Verifiable Geometry

Relying solely on visual semantic tokens enables the model to identify "this is a bed and a TV" but makes it difficult to determine whether two views describe the same spatial location. Relying solely on poses and depth is insufficient to answer natural language questions. ConsiSpace thus uses SigLIP2 to produce visual semantic tokens, and VGGT to generate spatial tokens, camera poses \(T_t\), and depth maps \(D_t\). Visual tokens and spatial tokens are concatenated and projected to form fixed-length implicit evidence entries, while camera poses and depths serve as explicit geometric entries with the same timestamps.

This dual memory structure does not function as two independent knowledge bases: implicit entries carry "what is in the scene" for semantic retrieval and LLM consumption, while explicit entries represent "where and in which direction it is viewed from" to control subsequent writing, fusion, and filtering. Decoupling the content from the control signals ensures that geometric errors are not directly translated into linguistic errors, but rather serve as soft constraints for organizing evidence.

2. Gated Geometric Writing & Fusion: Memory Expansion Only under Physical Perspective Shifts

For the \(t\)-th frame, the system obtains the position \(p_t\) and viewing direction \(v_t\) from the camera pose and compares the translation and rotation vector against the adjacent frame. Writing occurs only when at least one displacement exceeds its threshold:

\[ \operatorname{write}(t)=\mathbb{I}\left[\lVert p_t-p_{t-1}\rVert_2>\tau_p\ \lor\ \angle(v_t,v_{t-1})>\tau_\theta\right]. \]

The default writing thresholds are set to \(0.15\) m and \(15^\circ\). This step skips redundant frames during smooth camera movements. However, "movement detection" does not automatically dictate adding a new entry: the system searches for existing observations within a geometric neighborhood defined by a position radius \(r\) and angle difference \(\theta\). It then selects the entry with the highest semantic cosine similarity to the new evidence for fusion. A new entry is appended only if the neighborhood is empty. The default fusion neighborhood is set to \(0.40\) m and \(30^\circ\), condensing duplicate observations of the same area into compact evidence while retaining new information brought by cross-region transitions or large angular rotations.

3. Hierarchical Geometrically Filtered Retrieval: Retaining Global Layout and Query-Specific Details

Compact memory may still contain entries that are irrelevant or orientationally conflicting with the current query. ConsiSpace first retrieves top-\(K\) candidate segments based on semantic similarity between the question and the segment keys, and then scores each frame within the candidates:

\[ s_f(i)=\alpha\cos(q_e,k_i^f)+\beta\cos(Q_{\mathrm{topo}},e_i^{\mathrm{dir}}). \]

The first term evaluates the semantic relevance to the question, while the second term checks the observation direction using a direction query derived from the question; hence, even if a frame contains "television", it is down-weighted or filtered out if its geometric perspective conflicts with the query. In practice, the top-8 candidates are selected at both the segment and frame levels, and the filtered implicit evidence along with their aligned geometric cues are fed into the language model.

Since sparse top-\(K\) retrieval tends to lose holistic spatial relationships, the model separately aggregates topological and metric summaries from the memory: the former focuses on directions and connectivity, while the latter captures distances, sizes, and sequences. The final context is concatenated from the question tokens, both summaries, local evidence, and aligned geometric cues. Ablation studies demonstrate that relying solely on summaries lacks fine-grained details, while relying solely on top-\(K\) ignores global structure; combining both yields the most optimal performance.

4. Unified Cross-View Consistency Learning: Mutually Supervising Two Views of the Same Scene

While GCM refines the evidence, it cannot guarantee that the language model yields consistent answers when facing two different temporal windows or retrieval contexts. UC-SSRL randomly samples two observations for the same pair \((V, q)\), obtaining prediction distributions \(\pi_i\) and \(\pi_j\), and optimizes three complementary rewards on top of the SFT model. Answer consistency penalizes the KL divergence between the two prediction distributions; metric consistency penalizes the difference between two distance estimations; and topological consistency enforces symmetry using bidirectional KL constraints over the relation distributions:

\[ R=-\alpha\operatorname{KL}(\pi_i\Vert\pi_j)-\eta|\hat d_i-\hat d_j|-\beta\left[\operatorname{KL}(P_i\Vert P_j)+\operatorname{KL}(P_j\Vert P_i)\right]. \]

Since consistency alone does not guarantee correctness, the authors first perform maximum likelihood SFT, and then apply UC-SSRL to stabilize cross-view predictions. Only the LoRA parameters are updated to prevent representation drift from self-supervised training signals. The three rewards target categorical answers, continuous metrics, and relational structures, respectively. Ablation of consistency rewards shows that using any single reward outperforms SFT, while combining all three yields the best results across all four summary metrics.

A Detailed Example¶

Suppose a camera moves slowly through a bedroom in a video, and the question is "When standing by the desk facing the bed, which direction is the TV?" SigLIP2 captures the semantics of the bed, desk, and TV, while VGGT estimates the camera trajectory, depth, and orientation. Small, continuous movements that do not exceed \(0.15\) m or \(15^\circ\) are not written. A new frame exceeding these thresholds is fused with the most similar entry if it falls within the existing \(0.40\) m and \(30^\circ\) neighborhood; otherwise, a new entry is created.

During inference, the model retrieves the top-8 candidates from 4 memory segments, and then selects the top-8 frames within these segments by combining the semantics of "desk, bed, TV" and the query's implicit orientation. The topological summary provides relative directions in the room, the metric summary supplements distance relations, and the local frames present fine-grained object details. During training, the same question is answered again from another temporal window. If the two model outputs predict "front-right" and "back-left" respectively, both the answer and topological rewards will penalize this inconsistency, rather than treating the two predictions as mutually independent samples.

Loss & Training¶

The model adopts Qwen3-VL-8B-Instruct as its language backbone, with SigLIP2, VGGT, the vision tower, and projectors frozen. Both SFT and UC-SSRL train only LoRA parameters with rank 16 and \(\alpha=32\). The SFT data includes VSI-590K, and nuScenes-10K filtered from 22,518 candidates through geometric validity, answer parseability, consistency, MLLM verification, and deduplication. Manual inspection of 1,000 randomly selected samples reveals an accuracy rate of 91.6%.

Training is conducted in bfloat16 using DeepSpeed ZeRO-3 across 8x A100 80GB GPUs for 200 steps. The learning rate is set to \(1\times10^{-4}\), the per-GPU batch size is 1, gradient accumulation is 2, and the maximum sequence length is 5,120. UC-SSRL is initialized from the SFT checkpoint with an overall reward weight of \(10^{-2}\) and reuses the same training pool without requiring additional human consistency labels.

Key Experimental Results¶

Main Results¶

Benchmark / Setup	Qwen3-VL-8B	Strong Specialized Baseline	ConsiSpace SFT	ConsiSpace UC-SSRL	Gain over Strong Baseline
VSI-Bench Avg.	57.4	SpaceMind 69.6	71.2	76.6	+7.0
OSI-Bench Avg.	31.2	VLM-3R 40.3	45.8	53.0	+12.7
MMSI Sufficient-Coverage Avg.	29.1	VLM-3R 42.6	48.4	57.5	+14.9
MMSI Uniform-50 Avg.	27.6	VLM-3R 43.1	48.9	58.1	+15.0

Ablation Study of GCM¶

Configuration	VSI	OSI	MMSI-Video	Normalized Memory Entries
Full ConsiSpace SFT	71.2	45.8	52.3	1.0×
w/o Gated Writing	69.8	44.5	47.2	2.3×
w/o Consistent Fusion	70.3	45.0	47.9	1.8×
w/o Geometric Filtering	69.5	43.9	46.0	1.0×

Ablation Study of UC-SSRL Rewards¶

Answer / Metric / Topological Reward	VSI	OSI	MMSI-SC	MMSI-U50
None / None / None (SFT)	71.2	45.8	48.4	48.9
w/ / None / None	73.5	48.2	52.1	52.6
None / w/ / None	72.8	49.0	51.4	51.9
None / None / w/	72.9	47.1	51.7	52.2
w/ / w/ / w/	76.6	53.0	57.5	58.1

Key Findings¶

UC-SSRL improves over SFT with the same architecture by 5.4, 7.2, 9.1, and 9.2 points on VSI, OSI, MMSI-SC, and MMSI-U50 respectively. The benchmarks requiring perspective change and metric reasoning (OSI and MMSI) show the most significant gains.
The three lifecycle stages of GCM serve distinct functions: gated writing and consistency fusion primarily reduce the number of memory entries, while geometric filtering, despite not altering the entry count, contributes a 6.3-point gain on MMSI-Video. This indicates that both "storing efficiently" and "retrieving accurately" are vital.
With 200 input frames, ConsiSpace uses 35.0GB of memory and 3.10s of inference time, while VLM-3R consumes 78.5GB and requires 10.80s. At this capability, ConsiSpace achieves 72.4/46.8/54.4/55.1 on VSI/OSI/MMSI-SC/MMSI-U50, outperforming VLM-3R's 69.5/43.5/51.1/52.3.
Injecting 30% noise into the VGGT-generated geometry still yields 73.1/49.6/53.4/54.3 for ConsiSpace, but shows a clear drop compared to clean inputs (76.6/53.0/57.5/58.1), highlighting that geometric quality remains critical to the performance upper bound.

Highlights & Insights¶

The most elegant aspect of the paper is unifying "geometric consistency" across two levels: on the input side, it governs how evidence is admitted to memory; on the output side, it determines whether predictions from two perspectives can align with each other. This prevents the geometric modules and training objectives from operating in isolation.
Gated writing and fusion shift long video memory management from "keeping more frames" to "retaining changes in spatial states." Such post-processing strategies can be transferred to robotic patrolling, first-person egocentric logs, and autonomous driving history caching by simply replacing the pose and orientation thresholds with task-relevant state-change criteria.
The readout design of combining summaries with top-\(K\) retrieval is highly reusable: summaries capture the global layout while sparse retrieval preserves local details. Its ablation results are more convincing than simply increasing retrieval counts, demonstrating that long-context systems must distinguish between global structure and local granularity.
UC-SSRL bypasses the need for manual consistency labels, but it must be anchored on the correctness provided by SFT. This serves as a reminder that self-consistency is effective for reducing prediction variance, but cannot inherently guarantee that two mutually consistent answers are indeed correct.

Limitations & Future Work¶

The main text does not include a dedicated limitation section. Based on its design, GCM relies heavily on the pose, depth, and orientation estimated by a frozen VGGT. Although noise sensitivity experiments show gradual degradation instead of a catastrophic breakdown, the four metrics drop by 3.5 to 4.1 points under 30% noise. In dynamic scenes, under severe occlusions, or when geometric estimations fail, the system may still incorrectly write, fuse, or filter out essential evidence.
The proposed method assumes a nearly static 3D relationship under smooth perspective changes. When objects move, the rule "similar observations near the same location should be fused" might become invalid. The explicit memory also lacks mechanisms for handling state expiration, trajectory updates, and conflict resolution; future work could integrate dynamic object tracking and temporal decay.
UC-SSRL optimizes consistency across views rather than direct accuracy. Although SFT anchors correctness, the model could still stably predict the same incorrect answer from two different views. Future investigations can introduce verifiable geometric constraints, prediction confidence, or a small portion of ground-truth calibration rewards.
The training cost remains high (8x A100 80GB), and the paper only reports results on a single backbone, Qwen3-VL-8B-Instruct. Generalizability across varying model scales, different geometric encoders, and longer real-world videos has not yet been covered by the current experiments.
The 91.6% accuracy in the manual audit of 1,000 samples from nuScenes-10K suggests that automatically generated data still contains noise. A more granular error analysis, as well as a similarity analysis between the training data and OSI-Bench at the scene-category level, would strengthen the generalization claims.

vs VLM-3R / Spatial-MLLM: These approaches inject reconstructions or 3D representations into MLLMs, focusing on "providing geometric inputs." In contrast, ConsiSpace leverages geometry to control the entire lifecycle of long-term evidence, yielding clear memory and latency advantages particularly under 200 input frames.
vs SpaceMind / GeoThinker: While both focus on camera-guided fusion or active geometric integration, ConsiSpace differs by explicitly determining when to write, what to fuse, and which view to filter during retrieval, while further constraining final predictions via cross-view rewards.
vs Spatial-SSRL: While Spatial-SSRL relies on intrinsic spatial pretext tasks, UC-SSRL directly derives answer, metric, and topological consistency rewards from two different observations of the same question on a video, which more closely aligns with view stability in video spatial QA.
Insights for Memory-Augmented VLMs: External memories should not merely serve as passive token repositories. If a domain possesses verifiable invariants (e.g., spatial geometry, object identity, or temporal causality), these invariants can be utilized to concurrently constrain memory writing, merging, retrieval, and training.

Rating¶

Novelty: ⭐⭐⭐⭐☆ Unifying geometric consistency in memory lifecycles and post-SFT reinforcement learning offers a clear and systemic architecture.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ The paper thoroughly evaluates three primary benchmarks, and covers navigation validation, efficiency scaling, component and reward ablations, as well as threshold and noise sensitivity analyses.
Writing Quality: ⭐⭐⭐⭐☆ The core logic and mathematical formulas are complete, and the experiments are clearly structured. However, the main text does not centralize the discussion on limitations, leaving some implementation details in the supplementary material.
Value: ⭐⭐⭐⭐⭐ Simultaneously improving accuracy, cross-view stability, and resource efficiency in long video spatial reasoning holds strong reusable value for memory-augmented VLMs.