VIEW2SPACE: Studying Multi-View Visual Reasoning from Sparse Observations¶

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/pokerme7777/VIEW2SPACE
Area: VLM Reasoning
Keywords: Sparse multi-view observations, spatial reasoning, visual grounding, geometric supervision, chain-of-thought

TL;DR¶

VIEW2SPACE generates cross-view questions and geometrically verifiable reasoning traces from controlled 3D scenes, training Qwen3-VL-4B to output intermediate object boxes and reach 64.93% MCQ accuracy and 69.34% grounding mIoU, while increased training scale still struggles with difficult compositional reasoning.

Background & Motivation¶

Understanding one scene from isolated drone, surveillance, and ground-level images differs from processing a continuous video. Smooth camera motion is unavailable for tracking, and the same object may change appearance, become occluded, or appear in only one view. A model must establish cross-view correspondences before composing object-relative or camera-relative spatial relations; single-image localization does not automatically provide these skills.

Real data rarely offers accurate camera parameters, object identities, occlusion measurements, and ground truth for every reasoning step at once. Existing evaluations also often emphasize human viewpoints and shallow relations, so selecting the correct answer need not mean locating the relevant objects. Controlled simulation lets the authors verify the geometric basis of answers and vary reasoning depth separately from visibility instead of attributing every failure to insufficient model size.

Core Idea: use the same 3D ground truth to generate questions, intermediate visual evidence, and difficulty labels, so models must identify which object in which view supports their reasoning rather than merely answer spatial questions.

Method¶

Overall Architecture¶

The input consists of sparse RGB views of the same static scene and a question; the output is an option, a count, or target boxes. The work connects a data engine, graded evaluation, and Grounded CoT supervision. It is not a new inference architecture that additionally consumes point clouds, depth, or a complete 3D scene graph.

The 3D information primarily supports data production: known object identities, positions, and camera parameters provide reliable intermediate supervision. The trained vision-language model (VLM) must still learn cross-view integration from RGB observations. The evaluation and training branches below use disjoint scenes.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Assets["3D assets and theme configurations"] --> Engine["Geometric Ground-Truth<br/>Data Engine"]
    Engine --> Tasks["Evidence Constraints<br/>and Difficulty Grading"]
    Tasks -->|Disjoint training scenes| Grounded["Chain-of-Thought<br/>with Visual Evidence"]
    Grounded --> Model["Full-parameter Qwen3-VL fine-tuning"]
    Tasks -->|Disjoint test scenes| Eval["MCQ, counting, and grounding evaluation"]
    Model --> Eval

Key Designs¶

1. Geometric Ground-Truth Data Engine: establish reliable spatial relations before generating language supervision

The asset library uses high-quality 3D objects, prioritizing scanned assets and calibrating their scale to real-world proportions. OSD-Tag does not ask GPT-4o to describe every object independently. It presents an aggregated overview of assets within a category, asks the model to identify visual and functional differences and construct hierarchical semantic tags, and then assigns those tags to individual assets with manual verification. Similar-looking objects can consequently be distinguished through consistent descriptions rather than arbitrary names in later referring questions.

Designers specify object categories, placement constraints, and inter-category relations for each theme, and Blender samples layouts within these constraints. Cameras cover drone, bird's-eye, human-like egocentric, and fixed-surveillance configurations, with adjustable positions, orientations, coverage, and fields of view. The engine records per-view 2D boxes and occlusion metadata alongside 3D object poses and camera parameters. This distinguishes an apparent image-right relation from a rightward relation defined in a specified coordinate frame.

Question templates select combinations of object relations, and geometric rules determine the answers rather than another VLM guessing them. The engine computes relations in object-centric and camera-centric coordinate frames and produces instance grounding, cross-view correspondences, and reasoning traces. Determinism here is relative to engine ground truth and rules: it removes stochastic inconsistency in model-generated traces, but does not establish that synthetic assets, tags, or templates are unbiased.

2. Evidence Constraints and Difficulty Grading: separate selecting, counting, and locating correctly

The benchmark does more than change output formatting. MCQ permits informed guessing from partial cues; counting requires identifying and aggregating relevant instances; grounding requires target boxes, so plausible language cannot substitute for correct locations. MCQ uses accuracy, counting uses accuracy and mean absolute error, and grounding uses mean intersection over union (mIoU) and F1 with a matching threshold of \(\mathrm{IoU}>0.5\). The available paper text does not specify the complete multi-object matching and averaging procedure, so the single-box definition below should not be mistaken for the full evaluation implementation.

\[ \operatorname{IoU}(B_p,B_g)=\frac{|B_p\cap B_g|}{|B_p\cup B_g|}. \]

Here \(B_p\) and \(B_g\) are the predicted and ground-truth boxes. This is the standard IoU definition, not a new loss introduced by the paper. Requiring box overlap tightens the requirement from a plausible answer to correctly located evidence. Percentages across the different tasks nevertheless measure different quantities and cannot be directly subtracted to quantify difficulty.

Reasoning difficulty depends on the minimum relational hops required in a per-view object-relation graph: objects are nodes, spatial relations are edges, and a single perceptual grounding step also counts as one hop. It additionally accounts for the logarithm of the number of key objects. The exact combined formula is deferred to an appendix absent from the cache, so only the main-text definition is retained here. Key-object visibility difficulty measures occlusion: the engine casts rays from object surface points toward the camera and estimates invisibility from the fraction blocked before reaching it, aggregating over task-relevant objects and views. Higher values mean more difficult observation.

VIEW2SPACE-v1 contains 3,591 questions across difficulty levels: 1,400 MCQ, 591 counting, and 1,600 grounding questions, involving 40 themes and 1,096 primary assets. The engine can generate 3 million QA pairs from 2,000 scenes, whereas the main training uses 300K samples with scenes disjoint from evaluation. Generation capacity, actual training size, and evaluation size are distinct quantities.

3. Chain-of-Thought with Visual Evidence: bind intermediate entities to boxes in their corresponding views

Instruction tuning supervises final answers, allowing a model to learn question formats without necessarily learning cross-view relational processes. Standard chain-of-thought (CoT) adds intermediate language, but the model may continue reasoning from an incorrect description without having located its referent. Grounded CoT changes the supervision by explicitly outputting the bounding box of each key object in its corresponding view during reasoning. Intermediate entities thereby obtain checkable visual referents before their spatial relations support subsequent conclusions.

These traces are generated deterministically by a symbolic pipeline from geometric metadata, rather than treating a model's self-generated long responses as teacher signals. Cross-view identity, relations, and positions share the same scene ground truth, giving verbal reasoning and box coordinates a consistent source. This is structured output supervision for a standard VLM, not an added external detector, cropping tool, or inference-time 3D reconstruction module. The available main text does not introduce a separate box-regression loss or reinforcement learning reward.

Such training does not automatically compress the reasoning process into direct-answer capability. When the Grounded CoT-trained model skips its chain at test time, grounding mIoU falls from 69.34% to 16.35%. Explicit traces remain a condition of effective operation for this model; having seen visual reasoning evidence during training does not imply that producing it during inference is unnecessary.

Loss & Training¶

The main experiments use Qwen3-VL-4B-Instruct with full-parameter supervised fine-tuning on 300K-scale engine-generated data, comparing final-answer supervision, standard CoT, and Grounded CoT. The main text does not provide a verifiable learning rate, epoch count, optimizer, or special loss formula. These details are deferred to the appendix, while the available cache contains only the main paper and references, so no hyperparameters are inferred here.

Separate scaling experiments use Qwen3-VL-2B/4B, 1K–300K training examples, and an independent balanced 3K analysis set. To partially isolate the factors, reasoning analysis restricts visibility difficulty to below 0.5, while visibility analysis restricts reasoning difficulty to below 6. These are not universal filtering conditions for the main 3,591-question test set.

Key Experimental Results¶

Main Results¶

Original Table 1, VIEW2SPACE-v1. Percentages retain the original metric definitions; lower counting MAE is better, while higher is better for the remaining columns. The proposed model receives dedicated training, whereas GPT-5 and the original Qwen3-VL-4B are comparison models; their training data and compute budgets are not matched.

Model / Setting	MCQ ACC (%)	Counting MAE	Counting ACC (%)	Grounding mIoU (%)	Grounding F1 (%)
Random (frequency)	28.59	2.46	13.05	2.01	0.27
Qwen3-VL-4B	35.19	2.26	21.15	16.53	18.72
GPT-5	59.86	1.31	38.10	8.18	3.43
Ours, Grounded CoT, 4B	64.93	0.58	54.99	69.34	70.92

Ablation Study¶

These results also come from original Table 1 and use Qwen3-VL-4B as the base model. The final row changes test-time output after Grounded CoT training; it is not a separate training objective.

Training / Test Setting	MCQ ACC (%)	Counting MAE	Grounding mIoU (%)	Grounding F1 (%)
Instruction tuning	36.50	0.71	47.58	50.07
Standard CoT	59.57	0.68	49.77	52.34
Grounded CoT	64.93	0.58	69.34	70.92
Grounded CoT training, direct answer at test	33.95	10.75	16.35	11.50

Standard CoT mainly improves MCQ over instruction tuning, with grounding mIoU increasing only from 47.58% to 49.77%. Explicit visual evidence then adds 19.57 percentage points in grounding. Relative to the original Qwen3-VL-4B, Grounded CoT gains 52.81 percentage points, or 319.48% relatively, in grounding mIoU. The main text's claim of an improvement exceeding 300% must not be read as a gain over standard CoT or instruction tuning.

Real-data transfer follows original Table 2 and the official MINDCUBE-Tiny protocol. It evaluates MCQ only, not grounding in real scenes.

Model	Fine-tuned on MINDCUBE	Overall ACC (%)
MINDCUBE-Plain	Yes	60.76
Ours, Grounded CoT, 2B	No	59.92
Ours, Grounded CoT, 4B	No	70.00

Key Findings¶

The 4B model exceeds MINDCUBE-Plain by 9.24 percentage points on real-data transfer, supporting transfer under this MCQ protocol only.
The controlled analysis in Figure 4 shows a sharp decline beyond visibility difficulty of approximately 0.7. This does not mean that more than 70% of the object is visible: higher difficulty indicates stronger occlusion.
Data scaling provides diminishing gains as reasoning difficulty rises, particularly beyond 50K examples. This is an empirical trend for the evaluated models and analysis set, not a theoretical scaling limit for all VLMs.

Highlights & Insights¶

The same geometric engine supports both answers and intermediate evidence, preventing teacher explanations and target locations from having unrelated sources. A transferable design principle is to give every key entity in a training trace a verifiable visual referent.
Separating correct option selection from successful localization is diagnostically useful. Strong MCQ performance does not establish that a model can provide precise evidence for similar spatial judgments.
Controlling visibility and reasoning difficulty separately distinguishes missing observations from failures to compose observed relations, rather than merging both into one aggregate score.

Limitations & Future Work¶

The authors suggest that linear CoT accumulates early errors and struggles to explore alternative relational hypotheses. Tree search and structured reasoning are proposed directions, not mechanisms implemented and validated in this paper.
Essential evidence may genuinely be unobservable in sparse views; more training cannot recover evidence that is absent. Actively acquiring views or allowing abstention are possible extensions, not demonstrated capabilities.
Real-data transfer is tested only on MINDCUBE-Tiny MCQ, which includes language descriptions of inter-view relations and relatively simple relational structures. The result does not establish grounding success in open-ended real scenes.
Scene-disjoint splits do not establish that assets, themes, and question templates are all disjoint. The main text does not demonstrate complete separation of these factors, leaving possible dependence on shared templates and synthetic distributions.
The cache lacks the appendix, preventing full reproduction of the difficulty formula, box-matching details, and training hyperparameters. Key comparisons without reported confidence intervals should not be interpreted as statistical significance tests.

vs MINDCUBE: Both study spatial understanding from limited views, but VIEW2SPACE emphasizes heterogeneous viewpoints, deeper relations, and explicit grounding evidence. Transfer is informative, yet difficulty is not directly equivalent across the two evaluations.
vs SpatialThinker / 3D-R1: Related methods strengthen reasoning through explicit 3D information, spatial rewards, or reinforcement learning. This work centers on geometrically generated supervision for RGB inputs rather than adding 3D inputs at inference time.
vs standard CoT: The goal is not simply longer answers, but intermediate steps tied to observable entities. Future evaluations should check those boxes as well as final answers rather than merely count reasoning tokens.

Rating¶

Novelty: 4/5. Sparse heterogeneous views, evidence constraints, and deterministic visual reasoning supervision form a unified research setting.
Experimental Thoroughness: 4/5. Training strategies, test-time output, transfer, and scaling are evaluated, but real-scene grounding remains untested.
Writing Quality: 4/5. The narrative is clear; some improvement claims require Table 1 to identify their reference baseline.
Value: 4/5. Useful for spatial reasoning supervision and diagnostic evaluation, without implying that higher aggregate scores solve deep multi-view reasoning.