CVSBench: A Comprehensive Benchmark for Cross-view Spatial Reasoning and Dreaming¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://huggingface.co/datasets/zlyzlyzly/CVSBench
Area: LLM Reasoning
Keywords: Cross-view Spatial Reasoning, Spatial Imagination, Satellite-Street Multi-view, Vision-Language Models, Spatial Cognition Benchmark

TL;DR¶

Addressing the severe deficiency of current vision-language models in spatial cognition under extreme real-world viewpoint transformations, this paper introduces CVSBench, a benchmark comprising 3,297 image groups and 40,679 QA pairs across satellite-street pairs, demonstrating that explicit 3D visual imagination brings substantially greater spatial reasoning gains than text-only Chain-of-Thought prompting.

Background & Motivation¶

Spatial cognitive ability forms the core foundation of embodied navigation, autonomous driving, and large-scale urban scene understanding. Humans can effortlessly construct internal representations across drastic viewpoint changes, mentally projecting an overhead satellite perspective from a single street-level snapshot to infer roof silhouettes, building setbacks, or integrating disparate first-person observations into a coherent, map-centric layout. In sharp contrast, modern Vision-Language Models (VLMs) suffer from severe egocentric biases and spatial hallucinations when confronted with substantial viewpoint shifts, failing to preserve cross-view object persistence and 3D geometric consistency.

Existing spatial benchmarks exhibit two critical bottlenecks. First, most current datasets are confined to indoor synthetic rooms or tabletop object manipulations, lacking the complex geometry, varied scale, and realistic occlusions inherent to large-scale outdoor urban environments. Second, existing multi-view benchmarks primarily restrict camera motion to small rotations orbiting a single object or slight agent displacements, offering limited viewpoint disparity. Satellite-street image pairs naturally present extreme viewpoint variations, drastic scale discrepancies, and asymmetric occlusions, serving as a rigorous testbed for evaluating deep cross-view geometric reasoning and mental imagination. However, prior research in remote sensing VQA and cross-view geo-localization has developed in isolation, lacking a unified cognitive benchmark that jointly assesses cross-view alignment, camera pose estimation, and visual question answering.

To address these limitations, this work focuses on a systematic evaluation that bridges cross-view visual question answering, cross-view grounding, and viewpoint localization. Core idea: construct CVSBench as a large-scale cross-view spatial reasoning benchmark based on satellite-street pairs, enforcing spatial imagination via an asymmetric single-view inference protocol and demonstrating that explicit 3D visual imagination provides significantly stronger cognitive gains than text-only chain-of-thought reasoning.

Method¶

Overall Architecture¶

CVSBench is designed to establish a challenging, multi-scale, and asymmetric evaluation suite for satellite-street spatial reasoning while exploring viable pathways to augment VLM spatial intelligence. The overall pipeline is organized into three principal components: first, a semi-automatic annotation and human verification workflow produces a multi-task dataset spanning cross-view VQA, cross-view grounding, and viewpoint localization; second, under a single-view input protocol, structured scene decomposition and spatial imagination Chain-of-Thought (CoT) prompting are combined with Supervised Fine-Tuning (SFT) and Reinforcement Learning (GRPO); third, to overcome the representational bottleneck of pure text reasoning, metric depth priors and 3D cognitive map miniature generation are introduced as explicit visual mental imagery for the VLM.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Satellite-Street Paired Data<br/>CVUSA and University1652"] --> B["Semi-automatic Multi-task Construction<br/>VQA / Grounding / Pose Localization"]
    B --> C["Single-view Constrained Protocol<br/>Asymmetric Cues Force Mental Inference"]
    C --> D["Dual Textual CoT Exploration<br/>Structured Scene and Spatial Imagination CoT"]
    D --> E["GRPO Reinforcement Learning<br/>Accuracy and Formatting Dual Reward"]
    C --> F["Explicit Visual Spatial Imagination<br/>Metric Depth Prior and 3D Cognitive Map"]
    E --> G["Cross-view Spatial Reasoning Evaluation"]
    F --> G

Key Designs¶

1. Semi-automatic Multi-task Pipeline with Asymmetric Spatial Inference: Enforcing Mental Projection via Restricted Input

Conventional cross-view matching benchmarks typically supply fully paired images simultaneously, allowing models to exploit superficial texture correlations or color shortcuts rather than exercising spatial geometric reasoning. CVSBench curates image pairs from CVUSA and University-1652, establishing three core task families: cross-view Visual Question Answering (VQA), cross-view Grounding, and Viewpoint Localization. In the VQA setting, two reciprocal directions are defined: Ground-to-Satellite (G2S) and Satellite-to-Ground (S2G). While paired images and Gemini-generated proposals are leveraged during annotation to guarantee ground-truth factual correctness across 3,297 image groups and 9,468 verified bounding boxes, models during evaluation are restricted to a single-view observation. The model is forced to deduce implicit, occluded, or geometrically ambiguous attributes of the unobserved target viewpoint (e.g., inferring whether a facade seen from the street corresponds to an indentation on the satellite footprint, or deducing street-level facade color from an aerial roofline). This asymmetric protocol eliminates direct image-matching shortcuts, compelling models to perform mental coordinate transformations.

2. Dual Textual CoT Formulations Optimized with GRPO: Structured Scene Decomposition and Perspective Projection

To evaluate whether modern VLMs can master cross-view projective geometry through textual reasoning alone, two complementary CoT strategies are developed. Structured Scene CoT prompts the model to parse the input image into structured entities, explicitly enumerating candidate object categories and bounding box coordinates before performing reasoning on the spatial topology. In contrast, Spatial Imagination CoT emulates human mental perspective-taking: the model sequentially details current observations, deduces cross-view geometric mappings, mentally projects target-view layouts, and infers the final answer. Models undergo Supervised Fine-Tuning (SFT) on curated reasoning traces followed by reinforcement learning via Group Relative Policy Optimization (GRPO). The optimization objective maximizes token-level advantage while regularizing drift via KL divergence against the reference policy:

\[J_{\mathrm{GRPO}}(\theta) = \mathbb{E}_{q \sim P(Q), \{o_i\}_{i=1}^G \sim \pi_{\theta_{\mathrm{old}}}(O|q)} \left[ \frac{1}{G} \sum_{i=1}^G \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \left( \hat{A}_{i,t} - \gamma D_{\mathrm{KL}}(\pi_\theta \parallel \pi_{\mathrm{ref}}) \right) \right]\]

where \(G\) denotes the sampled trajectory count per query, \(\hat{A}_{i,t}\) represents the normalized advantage, and the reward function assigns a 0.9 weight to multiple-choice option matching accuracy and 0.1 to structural format compliance.

3. Explicit Visual Spatial Imagination: Incorporating Depth Priors and 3D Cognitive Map Generation

Human cognitive psychology indicates that mental perspective shifts rely heavily on visual-spatial cognitive maps rather than symbolic text manipulation. Because general-purpose VLMs lack native mechanisms to synthesize intermediate 3D latent representations, external generative foundation models are deployed as cognitive prosthetics. Focus is placed on the FOV-subset, where narrow viewpoints induce severe spatial ambiguity. Two explicit visual modalities are explored: first, DepthAnything-v2 produces metric depth maps to clarify foreground-background geometric ordering; second, an instruction-following diffusion generator (nanobanana) synthesizes 3D miniature scenes combining oblique perspective with top-down layouts. This generated 3D miniature acts as an externalized mental sketch, allowing the VLM to visually inspect global footprint geometry, facade boundaries, and spatial arrangements prior to generating its response.

Loss & Training¶

The textual reasoning models are trained through a two-stage regimen on Qwen3-VL-4B. The first stage applies Supervised Fine-Tuning (SFT) on high-quality CoT demonstrations using a learning rate of \(3 \times 10^{-5}\) and a sampling temperature of 0.01. The second stage applies GRPO on the remaining training instances using a reduced learning rate of \(1 \times 10^{-6}\) and an exploration temperature of 0.7. The policy is optimized using normalized advantage estimates regularized by the KL divergence constraint.

Key Experimental Results¶

Main Results¶

Evaluations span frontier proprietary models, open-source multimodal baselines, and spatial-specialized models. Accuracy (Acc, %) is reported for VQA and viewpoint localization tasks, while mean Intersection over Union (mIoU, %) evaluates cross-view grounding.

Model Category	Method	CVUSA G2S	CVUSA S2G	CVUSA Grounding	FOV G2S	FOV S2G	FOV Grounding	FOV View-Arrow	FOV View-Image	Overall Acc	Overall mIoU
Human Baseline	Human	88.70	87.53	92.30	85.68	87.20	93.33	98.78	98.34	89.52	93.69
Proprietary	gpt-5-chat	69.70	59.50	10.72	53.50	54.10	13.65	40.10	31.70	53.76	12.30
Proprietary	gpt-4o	51.80	62.10	10.80	36.00	46.80	13.61	30.00	26.60	45.54	12.30
Open-source	InternVL3.5-8B	65.20	61.90	2.59	52.30	41.50	4.69	40.10	26.80	49.89	3.71
Open-source	Qwen3-VL-8B-Instruct	69.90	61.90	0.60	24.00	46.70	8.10	42.30	27.50	45.88	4.59
Open-source	Gemma-3-4b-it	71.00	52.88	0.38	40.51	47.99	6.94	30.13	28.68	46.43	3.87
Open-source	Qwen3-VL-4B-Instruct	67.90	62.40	3.68	26.60	44.20	13.15	40.80	27.10	45.48	8.72
Open-source	Deepseekv3.2	63.39	47.03	3.29	56.69	38.68	10.98	26.13	23.92	44.43	7.39
Spatial-oriented	SpaceThinker-Qwen2.5VL-3B	21.80	34.10	5.46	19.60	25.30	8.45	32.40	23.40	26.66	7.05
Spatial-oriented	SpaceQwen2.5VL-3B	22.70	40.20	2.22	3.40	29.40	8.50	26.40	26.70	25.45	5.56
Spatial-oriented	ViLaSR-7B	68.20	85.20	1.28	35.14	41.79	12.01	25.41	27.49	44.05	6.39

Ablation Study¶

The following table evaluates fine-grained attribute accuracy (%) for Qwen3-VL-4B across reasoning configurations and auxiliary visual representations on FOV-subset G2S and S2G tasks.

Training / Strategy	CoT / View	Facade	Roof	Symmetry	Color	Material	Height	Occlusion	Position	Vegetation Sector
Baseline (Qwen3-VL-4B)	None	18.4	9.9	79.8	24.5	41.5	73.6	51.9	56.1	49.0
SFT	Structured Scene	41.3	14.8	72.9	32.0	46.2	65.5	48.1	56.1	51.6
SFT + RL	Structured Scene	25.5	11.2	56.0	29.0	45.4	64.2	42.4	58.7	51.7
SFT	Spatial Imagination	51.0	14.1	57.1	27.7	43.8	66.7	54.8	55.8	49.5
SFT + RL	Spatial Imagination	43.3	8.4	67.5	27.0	43.9	69.6	42.4	55.9	51.0
Auxiliary View	Depth	18.5	10.2	79.8	24.4	42.2	74.3	54.8	56.1	48.8
Auxiliary View	3D View	17.1	18.7	80.5	36.9	47.0	75.0	51.2	58.3	50.0

Key Findings¶

Substantial Gap between Frontier VLMs and Human Cognition: While human annotators achieve 89.52% overall accuracy and 93.69% mIoU on cross-view grounding, the best-performing model (gpt-5-chat) only attains 53.76% accuracy and 12.30% mIoU. Most open-source models score below 10% mIoU, revealing that cross-view object correspondence remains an unsolved challenge.
Catastrophic Failure of Indoor-Tuned Spatial Models: Spatial-oriented models fine-tuned on indoor synthetic data (such as SpaceQwen and SpaceThinker) degrade severely to roughly 25% overall accuracy, trailing general-purpose VLMs. Single-view geometric priors do not translate into multi-view outdoor topological correspondence.
Plateau of Text-Only CoT and Vulnerability to Reward Hacking: Language-based reasoning yields marginal average improvements (1.75% on FOV-subset, 0.4% on CVUSA-subset). Furthermore, on fine-grained attributes lacking direct visual cues (e.g., facade color and roof structure), RL fine-tuning induces reward hacking and option bias, degrading post-RL performance relative to SFT.
Superiority of Explicit 3D Visual Imagination: Introducing a generated 3D miniature scene boosts facade color recognition by 12.4% (from 24.5% to 36.9%) and nearly doubles roof structure accuracy (from 9.9% to 18.7%), delivering an overall gain of 3.34% that surpasses both depth augmentation (1.23%) and textual CoT.

Highlights & Insights¶

Asymmetric Single-View Protocol: Prevents shortcut learning based on direct visual correlation, isolating whether the model has established an internal 3D spatial representation.
Visual Imagery Outperforms Symbolic Text: Confirms that high-level projective spatial reasoning is fundamentally grounded in visual cortex representations, indicating that future multimodal architectures should unify generation and perception into an "imagine-then-reason" paradigm.
Asymmetry in Viewpoint Localization: Identifying satellite arrows from a single street view achieves around 40% accuracy, whereas selecting the correct street-view image among four candidates given a satellite arrow drops to 27%, revealing severe information bottlenecking when cross-referencing multiple visual candidates.

Limitations & Future Work¶

Generation Artifacts and Latency: Reliance on external diffusion generators introduces geometric distortions, hallucinations, and inference latency, struggling especially with panoramic inputs.
Non-End-to-End Pipeline: Visual imagination currently operates as a disconnected offline preprocessing step rather than allowing the VLM to iteratively query imagined perspectives conditioned on task uncertainty.
Future Directions: Developing unified spatial foundation models with native 3D representations (such as latent 3D Gaussian Splatting) to perform closed-loop mental rendering and multi-view geometric verification.

vs SIBench / OmniSpatial / MIND-CUBE: Prior spatial benchmarks focus on indoor scenes, static 2D relations, or minor egocentric rotations; CVSBench introduces urban-scale orthogonal viewpoint transitions (satellite-to-street).
vs RSVQA / VRSBench: Conventional remote sensing VQA centers on aerial object classification and counting; CVSBench unifies aerial observation with ground-level perspective, object-level grounding, and camera pose localization.
vs Perspective-Aware Reasoning: Prior methods emphasize symbolic coordinate transformations; CVSBench proves that supplying explicit 3D cognitive maps provides a more effective geometric scaffold for entity correspondence.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Establishes the first systematic cross-view spatial reasoning and dreaming benchmark bridging satellite and street perspectives.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous coverage of proprietary and open-source models, fine-grained attribute breakdowns, and systematic ablations across text CoT, RL, and visual imagination.
Writing Quality: ⭐⭐⭐⭐⭐ Highly coherent narrative tracing the failure of symbolic text reasoning to the necessity of visual spatial imagination.
Value: ⭐⭐⭐⭐⭐ Provides a benchmark and methodological insight for embodied intelligence, autonomous systems, and 3D-aware vision-language foundation models.