ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Models

Conference: ECCV2026
Paper: ECCV Paper
Area: Multimodal VLM
Keywords: spatial reasoning, perspective taking, 3D annotation, spatial localization, supervised fine-tuning

TL;DR

ViewSpatial-Bench diagnoses perspective-taking failures through five camera- and person-centered spatial localization tasks, while approximately 43K targeted supervision samples raise Qwen2.5-VL-3B accuracy from 35.85% to 82.09%, a gain of 46.24 percentage points rather than a 46.24% relative improvement.

Background & Motivation

A vision-language model (VLM) can recognize people, furniture, and objects without understanding whose reference frame a spatial instruction uses. Passing a cup on a speaker's right requires adopting that person's position and orientation, rather than selecting a cup on the right side of the image. Correct recognition therefore leaves several problems unresolved: choosing the reference frame, estimating orientation, and transforming relative positions. Image-plane cues that work from the camera's perspective do not automatically solve these problems.

Existing spatial benchmarks cover object relationships, 3D properties, and some cross-perspective tasks, but differ in query targets, direction categories, and perspective transformations. The authors seek to diagnose spatial perception and perspective adoption within one task system. ScanNet supplies reconstructed scene coordinates, while MS-COCO supplies person boxes and keypoints. These complementary sources support geometric object relationships and head-versus-body orientation analysis, reducing the need to rely on language-model-generated spatial answers.

The paper also tests whether some spatial weaknesses arise from missing supervision rather than an inability to learn the relevant relationships. Applying the annotation procedure to source training splits and checking gains across backbones and answer formats makes this hypothesis testable. Accuracy improvements alone, however, cannot establish that a model has acquired a complete 3D world model. Core Idea: make the reference subject explicit in spatial questions and geometric labels, compare camera and person perspectives within one benchmark, and test whether targeted supervision can repair perspective-taking weaknesses.

Method

Overall Architecture

This is a benchmark and data-driven fine-tuning study, not a new visual encoder or explicit 3D reasoning network. Evaluation inputs are images or scene frames required by the task, a question specifying the reference frame, and candidate directions; the output is a direction option. Coordinates and human orientation estimates primarily support offline label construction. They should not be mistaken for ground-truth 3D inputs supplied to every evaluated VLM at inference time.

The procedure defines five perspective-dependent tasks, processes ScanNet and MS-COCO separately, derives directional labels using geometry and necessary manual annotation, and constructs multiple-choice questions with ambiguity filtering. For training, the same annotation approach operates on source training splits. Mixed camera- and person-perspective image-question pairs provide supervised fine-tuning data; the resulting models are collectively called Multi-View Spatial Models (MVSM).

Here, multiple perspectives primarily means multiple reference frames in the questions, not multiple camera inputs for every example. Person-orientation questions can use a single MS-COCO image, whereas scene-simulation questions draw on ScanNet scenes and frame information. The task, label, and training relationships are explained directly below, without a diagram that could be mistaken for a proposed network architecture.

Key Designs

1. Dual-reference-frame tasks: separate object positions, human orientation, and scene simulation

The camera-centered tasks are Cam-Rel. Dir. and Cam-Obj. Ori. The former asks for the relative direction between objects in the camera reference frame; the latter asks which way a person is facing relative to the camera. These are distinct problems: an object appearing on the image's left need not face left, and a person's head orientation cannot be inferred from image location alone. Measuring them separately distinguishes position errors from orientation errors.

The person-centered tasks are Per-Obj. Ori., Per-Rel. Dir., and Per-Sce. Sim. Per-Obj. Ori. measures head orientation relative to the person's body. Per-Rel. Dir. adopts a depicted person's viewpoint to locate another person or object. Per-Sce. Sim. asks the model to imagine standing at one object, facing a second, and locating a third. Specifying position and forward direction separately prevents perspective taking from collapsing into a simple left-right label swap.

The benchmark contains 18 directional labels in the union of its five task types, not 18 answer options per question. Camera and person tasks account for 48.4% and 51.6% of the data, respectively. This gives both reference frameworks substantial coverage, but their task mixtures differ. Near-balanced sample proportions therefore do not make every aggregate accuracy difference a pure measure of reference-frame difficulty.

2. Geometric and human annotation: ground direction labels in traceable visual evidence

The ScanNet branch supports Cam-Rel. Dir. and Per-Sce. Sim. The authors obtain scene voxels and apply maximum-coverage frame sampling: each iteration selects the frame adding the most previously uncovered voxels, subject to a frame budget or stopping condition. This reduces repeated observations of the same location and preserves more varied spatial layouts. Frame metadata records visible objects, visibility rates, and 3D coordinates in the camera frame.

Camera-relative directions follow from object coordinates and camera parameters. For scene simulation, the pipeline selects objects occurring only once in a scene, forms object triples, and finds frames containing all three. It then places the simulated observer at the first object, points them toward the second, and computes the third object's relative direction. Uniqueness and joint visibility matter: multiple identically named chairs or an invisible target would introduce reference ambiguity rather than isolate spatial reasoning.

The MS-COCO branch supports both human-orientation tasks and person-relative direction. Images are filtered to retain subjects occupying at least 20% of the image area. Person boxes define crops, and shoulder keypoints divide head and body regions. Orient-Anything-Large estimates their azimuth angles separately. For person-relative orientation, the important quantity is not the head's absolute orientation but its rotation relative to the body.

Algorithm 2 uses the following wrapped angular difference to avoid spuriously large differences across the full-turn boundary:

\[ \Delta=(az_{\mathrm{head}}-az_{\mathrm{body}}+540)\bmod 360-180. \]

Thresholds then map the offset to directions such as left, front-left, front, front-right, and right. Samples with invisible shoulders are rejected. For camera-relative orientation, the procedure compares confidence in head and body estimates and selects the more confident calculation. It also requires gaze to approximately align with head direction, so this is not fine-grained eye-gaze estimation. Complex multi-person cases use manual annotation, and all 864 Per-Rel. Dir. instances are manually annotated. The full dataset is therefore not entirely automatic.

3. Direction discretization and quality control: reduce option ambiguity while retaining explicit assumptions

Raw coordinates and angles must become standardized language answers. The authors map angle intervals to directions: for example, 22.5 to 67.5 degrees maps to front-right, and 67.5 to 112.5 degrees maps to right. Three-object questions first establish where the observer stands and what defines forward, then locate the target. The same target can consequently receive different correct directions in camera and simulated-observer frames; the label must follow the question's frame.

Distractors are constrained as well. If the answer is a single direction such as front, distractors exclude compound directions containing it. If the answer is front-left, distractors exclude its components front and left. This reduces ambiguity between descriptions at different granularities. However, option construction may itself offer statistical cues. The subsequent comparison of multiple-choice and direct-answer training addresses this concern instead of treating a high multiple-choice score as sufficient proof of spatial understanding.

Filtering addresses both invalid geometry and visually indeterminate questions. It removes incorrect or excessively large orientation angles, objects too close together, blurry images, difficult-to-identify subjects, and cases where gaze differs substantially from head orientation. Manual verification then checks target visibility and directional labels. Table 1 reports the final scale as 5,712 samples across 1,338 scenes, which is more precise than the rounded counts elsewhere in the paper.

4. Perspective-mixed supervision: train spatial judgments using consistent task language

The authors construct approximately 43K training examples from ScanNet and MS-COCO training splits, covering all five tasks. Because person-relative direction is difficult to annotate automatically, Spatial-MM supplies additional training data for that task. Consistent natural-language templates express the questions, standardized direction classes express the answers, and batches mix camera and person perspectives. The model therefore cannot answer every question by retaining the camera frame.

MVSM names existing VLMs after spatial fine-tuning rather than a separate architecture. The main backbone is Qwen2.5-VL-3B; Qwen2.5-VL-7B and InternVL2.5-2B provide additional checks. Benchmark imagery primarily comes from validation splits, while supervision comes from training splits. This separates source splits, but shared domains, templates, and annotation rules may still make training and evaluation closely matched, motivating separate transfer tests.

Loss & Training

The main text describes supervised fine-tuning without introducing a new loss term. No specialized objective or implementation is reconstructed here. Multiple-choice and direct-answer training use the same spatial data to compare response formats; the latter expresses the training target without candidate answer lists.

Detailed training and evaluation settings are deferred to Appendices C.1 and C.2. The available official-PDF text cache contains the main paper and references but not those appendices. Learning rate, epochs, effective batch size, freezing policy, and decoding parameters therefore cannot be verified from this cache. Common SFT defaults should not be substituted for reported settings.

Key Experimental Results

Main Results

Accuracy is the evaluation metric. The following rows come from Table 2, with all values in %. Cam denotes the camera frame and Per the person frame; Rel. Dir. means relative direction, Obj. Ori. human orientation, and Sce. Sim. scene simulation.

Model	Cam-Rel. Dir.	Cam-Obj. Ori.	Per-Obj. Ori.	Per-Rel. Dir.	Per-Sce. Sim.	Overall
GPT-4o	41.46	19.58	42.97	40.86	26.79	34.98
Doubao-Seed-2.0	65.60	44.78	47.19	72.09	33.57	53.52
Qwen3.5-Plus (397B)	62.21	38.65	50.20	68.17	38.37	52.28
Qwen2.5-VL (3B)	43.43	33.33	39.16	28.62	28.51	35.85
Qwen2.5-VL (3B) + SFT / MVSM	83.59	87.65	90.16	71.14	75.75	82.09

MVSM gains 46.24 percentage points overall. Cam-Obj. Ori. gains 54.32 percentage points, the largest improvement among the five tasks. Overall is copied from the source table rather than recomputed as an unweighted mean of the five columns. Although Table 2 is labeled zero-shot performance, MVSM receives specialized supervision for these task types. Its comparison with general models is not a ranking under identical training resources.

Ablation Study

The paper does not remove annotation components one at a time. Instead, Table 3 analyzes answer formats and backbones. Improvements below are accuracy percentage points; the format comparison does not exhaust all possible shortcuts.

Config (Table 3)	Original accuracy	Fine-tuned accuracy	Gain
Qwen2.5-VL (3B), multiple choice MC	35.85	82.09	46.24
Qwen2.5-VL (3B), direct answer DA	35.85	79.34	43.49
Qwen2.5-VL (7B)	36.85	83.01	46.16
InternVL2.5 (2B)	34.98	76.45	41.47

Direct-answer training scores 2.75 percentage points below multiple-choice training but remains far above the original model. This weakens the explanation that option structure accounts for the entire gain. Improvements across backbones also show that the effect is not unique to Qwen2.5-VL-3B, without establishing equal gains for every architecture.

Transfer to different tasks or scenes provides another necessary check. The following values come from Table 4 and retain individual tasks rather than mixing aggregation conventions. Only two VSI-Bench spatial subtasks are evaluated, not the complete benchmark.

Transfer evaluation (Table 4)	Qwen2.5-VL (3B)	MVSM	Gain (percentage points)
VSI-Bench relative direction	46.00	46.93	0.93
VSI-Bench route planning	21.90	31.44	9.54
VSI-App indoor	18.00	41.00	23.00
VSI-App outdoor	27.00	36.00	9.00
VSI-App average	22.50	38.50	16.00

Key Findings

• Route planning gains 9.54 percentage points without explicit route-planning supervision, supporting some cross-task transfer. Relative direction gains only 0.93 percentage points, so transfer is not uniform.
• VSI-App contains just 50 scenes, split into 25 indoor and 25 outdoor scenes. Table 4 gives indoor and outdoor gains of 23.00 and 9.00 percentage points, whereas another passage reports 20.00 and 4.00. This internal inconsistency is retained as a caveat; the note follows the directly checkable subtraction in Table 4.
• Camera-versus-person averages require attention to task composition. Some models do not follow the pattern that camera tasks are easier, so an average trend should not be recast as a universal rule.

Highlights & Insights

• Reference frames become supervision content: the question specifies whose spatial judgment is required, rather than merely asking for recognized objects. This exposes failures where object recognition is correct but the reference frame is wrong.
• Geometry and manual verification have complementary roles: reliable coordinate-based relations are automated, while difficult human relationships retain manual labels. Scalability need not exclude valuable cases that resist automation.
• Format and transfer checks constrain interpretation: direct-answer and route-planning results examine option dependence and task transfer, respectively. They offer stronger evidence than a single benchmark score, but do not establish a general 3D spatial representation.

Limitations & Future Work

• Reproducibility gaps: the cache lacks the cited appendices, preventing verification of training hyperparameters, detailed evaluation protocols, and annotation agreement. Application gains also contain a numerical inconsistency in the main text.
• Labels depend on estimators and filtering assumptions: head and body orientations come from Orient-Anything-Large, and gaze is approximated by head direction. Clearly visible filtered examples may not represent heavy occlusion or mismatched eye and head directions in real interactions.
• Controls remain incomplete: training and evaluation share source domains and construction rules; format comparison rules out only one shortcut explanation. Stricter separation of scenes, templates, and direction combinations, plus image-free controls, would better distinguish spatial learning from statistical pattern fitting.
• Application evidence is limited: 50 VSI-App scenes and selected VSI-Bench subtasks cannot replace closed-loop robot evaluation. Continuous directions, localization error, dynamic occlusion, and execution success would be useful additional tests.

Related Work & Insights

• vs SpatialRGPT-Bench / EmbSpatial-Bench: Table 1 characterizes these as primarily camera-centered object-relation evaluations. ViewSpatial-Bench adds human orientation, person-centered relations, and scene simulation, while introducing orientation-estimation and manual-label uncertainty.
• vs VSI-Bench / SPHERE: these already include some person-centered or cross-perspective evaluation, so the paper should not be described as the first study of perspective taking. Its contribution is a more explicit five-task localization suite and an annotation procedure extended to supervised training.
• vs Orient-Anything: this is an offline orientation-labeling tool in the pipeline, not a replacement VLM backbone. Reusing specialized perception models to produce supervision is a transferable approach for other geometric question-answering tasks.

Rating

• Novelty: 4/5. The contribution lies in multi-reference-frame task organization and trainable data construction, not a new model architecture.
• Experimental Thoroughness: 4/5. Broad model coverage, format comparisons, and transfer tests are useful, but component ablations and larger application studies are missing.
• Writing Quality: 3/5. Task definitions are clear, while percentage wording, application gains, and appendix availability complicate precise verification.
• Value: 4/5. A useful reference for diagnosing spatial perspective taking and designing targeted supervision data.