Visual Spatial Tuning¶
Conference: ECCV2026
Paper: ECCV Official Page
Code: https://github.com/Yangr116/VST
Area: Multimodal VLM
Keywords: spatial perception, spatial reasoning, textual scene reconstruction, bird's-eye-view supervision, layout reward
TL;DR¶
Without adding a specialized 3D encoder, VST first teaches a VLM spatial perception through multimodal geometric data, then develops spatial reasoning with BEV-assisted textual reconstruction traces and reinforcement learning, raising Qwen2.5-VL-7B from 25.9 to 34.8 on MMSI-Bench and from 38.9 to 61.2 on VSI-Bench.
Background & Motivation¶
Vision-language models (VLMs) can recognize objects and predict their 2D positions, but recognizing a chair and locating it after a viewpoint change require different abilities. The former relies primarily on semantics and pixel localization, whereas the latter requires depth, cross-view correspondence, and temporal information. Data-driven methods such as SpatialVLM add spatial knowledge, but many cover only single images or supervised fine-tuning. Other approaches inject geometry through additional 3D encoders, increasing architectural and deployment complexity.
VST separates the problem into interdependent spatial perception and spatial reasoning: perception establishes object locations and cross-view relations, while reasoning builds and manipulates an internal scene representation. Asking a model to produce a longer chain-of-thought cannot repair a layout it perceived incorrectly in the first place. Conversely, training it to output many 3D boxes does not guarantee that it can compose those relations in language. The paper therefore addresses geometric consistency in training data, the intermediate reasoning representation, and the reliability of teacher-generated traces together.
The approach teaches a general VLM to describe a layout reliably before answering questions grounded in that layout, instead of making numerical reconstruction a mandatory interface for every reasoning task. Core Idea: establish spatial perception with unified geometric supervision, use BEV-assisted teachers to generate reconstruction-before-reasoning traces, and reinforce answer and layout quality through task-specific rewards.
Method¶
Overall Architecture¶
The base model follows Qwen2.5-VL's ViT-MLP-LLM architecture and accepts a single RGB image, multiple views, or a video together with a question. Training proceeds through perception-oriented supervised fine-tuning on VST-P, a chain-of-thought cold start on VST-R, and GRPO reinforcement learning. Outputs depend on the task and include spatial answers, textual layouts, or 3D detection boxes.
The four key components are Unified Geometric Perception, BEV Teacher Prompting, RT-CoT Textual Reconstruction, and the 3D Layout Reward. BEV images and ground-truth annotations in the diagram construct training supervision; they are not additional maps supplied to the student at test time. The trained VLM still receives ordinary visual inputs.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Single images, multiple views, videos<br/>and geometric annotations"] --> B["Unified Geometric Perception<br/>VST-P supervised fine-tuning"]
A --> C["Training only<br/>ground-truth boxes and QA"]
C --> D["BEV Teacher Prompting<br/>generate VST-R traces"]
B --> E["RT-CoT Textual Reconstruction<br/>chain-of-thought cold start"]
D -->|Trace supervision| E
E --> F["3D Layout Reward<br/>task routing and GRPO"]
F -->|Produces trained model| G["VST model"]
H["Test RGB and question<br/>no ground-truth BEV"] --> G
G --> I["Textual layout and reasoned answer<br/>or 3D detection boxes"]
VST-P contains 4.1M samples spanning 19 tasks; single-image, multi-image, and video data account for 64.8%, 33.1%, and 2.1%, respectively. VST-R contains 135K samples, with 77.8% providing cold-start reasoning traces and 22.2% providing rule-checkable data for online RL. It also includes general reasoning tasks involving mathematics, OCR, and knowledge, rather than consisting exclusively of spatial questions.
Key Designs¶
1. Unified Geometric Perception: make heterogeneous visual supervision describe a consistent physical space
VST-P does more than scale up left-versus-right questions. Single-image data covers depth comparison, distance and size estimation, 3D detection, and grounding. Its data engine transforms depth and 3D boxes into a unified camera coordinate system before generating instructions. Detection boxes encode the center, dimensions along three axes, and three rotation angles, giving 9 degrees of freedom. Multi-image samples further transform objects from all views into the first image's camera frame and use point clouds and depth to establish correspondences, avoiding inconsistent supervision for the same object across views. Video data extends the engine with object appearance times for ordering, counting, and spatiotemporal relations; approximately two-thirds of the video data comes from VLM-3R and is reorganized into a multi-turn format. Loose 2D boxes obtained by projecting 3D annotations can refer to the wrong object, so expert models verify object presence and generate masks to reduce referring noise. This data preparation is distinct from loading an additional 3D encoder at inference time.
A shared coordinate system is not sufficient: focal-length differences change the relationship between pixels and spatial rays, potentially teaching the model to mistake camera differences for size differences. FoV Unification applies a geometry-consistent image transformation only to metric-sensitive data, normalizing inputs to a virtual camera with a predefined focal length. Training also mixes single-turn and multi-turn detection so that later boxes can use the layout context provided by earlier ones. Scene captions convert boxes and scene-graph relations into natural-language spatial descriptions, discouraging memorization of numerical output templates alone. The former improves geometric comparability, while the latter connects geometric knowledge to language; together they prepare the model for textual reasoning.
2. BEV Teacher Prompting: improve the teacher's spatial evidence before distilling its reasoning traces
A teacher receiving only multi-view RGB can also misidentify object correspondences or directions, producing fluent but incorrect chains-of-thought. VST samples question-answer pairs from VST-P and supplies the Seed1.5-VL teacher with RGB images, a bird's-eye view (BEV) rendered from ground-truth 3D boxes, compact textual box metadata, the question, and its answer. The BEV is a deterministic top-down projection with object categories and orientations, placing relative positions that are difficult to compare in perspective views onto a common plane. The teacher still generates a textual trace, but now geometric annotations support its organization of the scene across views instead of leaving it dependent on unreliable spatial inference.
The privileged-information boundary matters: ground-truth boxes and answers are available to the teacher during data synthesis to produce accurate, coherent explanations. The student learns to generate traces conditioned on RGB and does not require a ground-truth BEV at test time. This is therefore not an inference pipeline that first invokes an external mapping system and then asks a VLM to query its map. Table 8 compares ordinary RT-CoT and BEV-assisted traces on the same object-relation subset to test supervision quality, rather than attributing the benefit of extra test inputs to the model itself.
3. RT-CoT Textual Reconstruction: establish a referenceable layout before changing frames to answer the question
Reconstruction with Text in Chain-of-Thought (RT-CoT) organizes an output into RECON, REASON, and ANSWER: describe objects, shared landmarks, and relative layout across views; derive the requested relationship from that representation; then give the answer. Supervised cold-start training teaches the perception model this trace format. Its value is not simply a longer response. It separates observations from deductions, allowing directional judgments to refer to spatial relationships already established in the reconstruction. The paper shows that a textual layout can be drawn as an SVG to illustrate its interpretability; this does not mean the model must generate or render an SVG whenever it answers.
The Num-CoT comparator represents scenes directly with 3D boxes and is vulnerable to numerical errors when camera poses across viewpoints are estimated poorly. RT-CoT does not require precise absolute coordinates before every answer, fitting the VLM's existing language modeling abilities and supporting descriptions of objects outside the current view. Metric tasks remain intact: detection still produces 3D boxes, while relational reasoning uses a textual layout. The trade-off is that natural-language layouts have no automatic guarantee of geometric consistency. Incorrect object correspondences can still propagate into the answer, so readable reasoning should not be equated with verifiable reconstruction.
4. 3D Layout Reward: detect accurate boxes without inflating recall through excessive predictions
The cold-start model initializes Group Relative Policy Optimization (GRPO). Responses sampled for the same question receive group-relative advantages, eliminating the need for a separate value model. Rewards are hard-routed by task: multiple-choice questions, open-ended questions, and OCR use their respective accuracy scores, whereas 3D detection uses a layout score; the selected score is then multiplied by a format reward. Not every spatial question is scored with IoU, and answer and detection rewards are not added together for every sample. The task determines what constitutes useful feedback.
The layout score first establishes a bipartite matching between predicted and ground-truth boxes, then equally combines the mean 3D IoU of successfully matched pairs with detection F1, using a default mixing coefficient of 0.5. A true positive for F1 requires a matched pair with 3D IoU exceeding 0.25; precision and recall jointly constrain false positives and missed objects. Rewarding IoU plus recall alone can encourage excessive box predictions to improve coverage, ultimately reducing detection quality. F1 makes overprediction costly as well. The cached equation is damaged, so this explanation follows the surrounding prose rather than presenting a repaired expression as the authors' exact formula. The layout-reward threshold should also not be confused with the threshold used for the reported AP15 metric.
A Worked Example¶
Figure 4 presents two bedroom images: the first contains a crib with a towel on it, while the second shows a glass balcony door and a nearby curtain. The question asks the model to imagine standing at the door in the second image, with the towel in the first image defining forward, and determine the curtain's direction relative to that position.
During RECON, the model describes both views and connects the crib, door, and curtain within the shared scene; the right-hand side of the second image cannot simply be treated as the imagined observer's right. During REASON, it places the observer at the door, defines a new directional frame by facing the towel, and then gives B, front-right, in ANSWER. Qwen2.5-VL answers A, back, in the figure. This is a qualitative example, not an independent success-rate estimate.
When constructing this kind of training sample, the teacher can use the ground-truth BEV to resolve the door and crib locations. At test time, the student must reconstruct the layout from the supplied images alone. The example shows why multi-view fusion and reference-frame transformation should be distinguished, and why a teacher's access to BEV does not imply that the student has a complete map at inference time.
Loss & Training¶
Perception SFT trains autoregressive text prediction conditioned on visual tokens and mixes in general multimodal data to mitigate forgetting. RT-CoT cold-start training uses the same supervised objective before switching to GRPO. The prose mentions per-token loss weights, but Equation 2 is corrupted in the cache, so no exact mathematical expression is reconstructed here. Learning rates, batch sizes, GPU configurations, and complete inference budgets are deferred to an appendix that is absent from this cache, which contains only the main paper and references; these settings have not been reproduced.
For downstream validation, the authors follow OpenVLA by discretizing the action space into 256 bins, representing actions with special tokens, and autoregressively predicting single-step actions from an image and instruction. They then fine-tune the model and action embeddings on LIBERO. No additional robot pretraining data is used in this experiment, but the VLM is already pretrained and VST has undergone spatial training. Starting action learning from scratch does not mean initializing the entire network randomly.
Key Experimental Results¶
Main Results¶
The following values are selected from the paper's Table 2 and evaluated with VLMEvalKit. Higher scores are better, and gains are absolute percentage-point differences from the same-size Qwen2.5-VL-7B baseline. CV-Bench and 3DSRBench evaluate single images, MMSI-Bench and BLINK evaluate multiple images, and VSI-Bench evaluates videos. The main text does not list individual split names or a uniform test-token budget, so these are not established as strictly compute-matched comparisons.
| Model / Gain | CV-Bench | 3DSRBench | MMSI-Bench | BLINK | VSI-Bench | MMMU |
|---|---|---|---|---|---|---|
| Qwen2.5-VL-7B | 80.6 | 53.2 | 25.9 | 56.4 | 38.9 | 51.2 |
| VST-7B-SFT | 85.5 | 54.6 | 32.0 | 62.1 | 60.6 | 50.6 |
| VST-7B-RL | 86.5 | 60.1 | 34.8 | 62.6 | 61.2 | 49.4 |
| RL relative to the base model | +5.9 | +6.9 | +8.9 | +6.2 | +22.3 | -1.8 |
The main table reports Gemini-2.5-Pro at 36.9 on MMSI-Bench, above VST's 34.8, so the results do not establish universal leadership across all models. The introduction gives 87.8 for CV-Bench, conflicting with Table 2's 86.5; this note uses the table value. Table 3 reports 44.2 AP15 for VST-7B-RL on SUN RGB-D, but also lists Gemini Robotics-ER at 48.3, which does not support the prose's unqualified first-place claim.
Ablation Study¶
The MMSI-Bench experiment in Table 8 uses a separately constructed Qwen2.5-VL-7B baseline: one-third of the VST-P single-image data, multi-view correspondence, multi-image 3D detection, and 800K general samples. This differs from the main training recipe, so its final 35.3 must not replace 34.8 in the main table. OO denotes object-object relation data, while Mix includes all task types.
| Config (paper Table 8) | MMSI-Bench | Gain over preceding row | Note |
|---|---|---|---|
| Perception baseline | 26.4 | Not applicable | Before reasoning cold-start training |
| Num-CoT, OO, cold start | 29.2 | +2.8 | Numerical 3D boxes represent the layout |
| RT-CoT, OO, cold start | 30.0 | +0.8 | Switch to a textual layout |
| RT-CoT + BEV, OO, cold start | 31.1 | +1.1 | Improve teacher traces |
| RT-CoT + BEV, Mix, cold start | 31.7 | +0.6 | Expand the task mixture |
| Mix, RL | 35.3 | +3.6 | Initialize GRPO from the preceding row |
Table 10 separately evaluates the layout reward on SUN RGB-D. AP is the paper's reported aggregate average precision, and AP15 is average precision at a 3D IoU threshold of 0.15; higher is better for both. The main text does not explicitly identify the model size for this table or enumerate the full threshold set underlying aggregate AP, so its results are not combined directly with the main 7B results.
| Config (paper Table 10) | AP | AP15 | AP relative to baseline |
|---|---|---|---|
| Baseline | 20.2 | 30.3 | 0.0 |
| 3D IoU + Recall | 13.8 | 20.9 | -6.4 |
| 3D IoU + F1 | 24.4 | 36.0 | +4.2 |
Thus, +4.2 AP is relative to the baseline, not to IoU + Recall; the latter two rows differ by 10.6 AP. The corresponding AP15 gain over baseline is 5.7 and should not be confused with the main VST-7B-SFT-to-RL improvement.
Key Findings¶
- In Table 5's progressive 3B data experiment, adding VST video supervision after general video data raises VSI-Bench from 38.1 to 54.7, a gain of 16.6 percentage points. General video QA does not substitute for explicit spatiotemporal supervision.
- Table 7 fixes training at 500M tokens on Qwen2.5-VL-3B: VST-P obtains an S-AVG of 56.2 versus SPAR's 50.2, with 200K randomly sampled instances for each. This supports the role of data quality more directly than comparing full datasets of unequal size.
- Table 11 reports average LIBERO success rates increasing from 53.1 to 61.7 at 2.5K steps and from 67.9 to 70.8 at 10K steps. In the former setting, the Goal column rises from 53.8 to 67.8, a difference of 14.0, although the table labels it +4.0. This discrepancy is retained rather than using the incorrect annotation to draw a conclusion.
Highlights & Insights¶
- Converting geometric ground truth into BEV improves reasoning-trace supervision rather than adding test inputs. This use of privileged training information is transferable to multi-view relational tasks.
- Numerical supervision and linguistic intermediate representations are complementary: one teaches spatial foundations, and the other organizes relational deductions. VST connects them rather than claiming that text replaces every form of precise reconstruction.
- The layout reward reveals a concrete RL failure mode: prioritizing coverage can induce false positives. Reward design must penalize both missed objects and unsupported extra predictions.
Limitations & Future Work¶
- The authors explicitly identify cross-view pose estimation as a difficulty for numerical layouts and excessive false positives as a failure of IoU-plus-recall rewards. Textual reconstruction and F1 address these issues, respectively.
- This note's assessment: the paper has no dedicated limitations section and provides no empirical guarantee that textual layouts always agree with true 3D geometry. Verifiable relational constraints could help audit intermediate reconstructions.
- This note's assessment: preserving general capabilities should mean retaining overall competitiveness, not avoiding every regression. In Table 2, the 7B model's MMMU score falls from 51.2 to 49.4 and MMBench from 83.5 to 83.0.
- Data construction requires ground-truth geometry, expert models, and a teacher. Avoiding extra architectural encoders does not establish low training cost. LIBERO evaluates simulated manipulation, not autonomous driving or real-robot deployment.
Related Work & Insights¶
- vs SpatialVLM / SAT: VST covers single images, multiple images, and videos while connecting perception SFT with reasoning RL. Its advantage combines data coverage, cleaning, and the training sequence.
- vs SPAR: Beyond differences in data sources, VST verifies and corrects projected 2D references. The matched-token comparison supports the overall data recipe but does not isolate every cleaning operation's contribution.
- vs VLM-3R: VST adds no specialized 3D encoder to the student, but reuses some of its video training data. Architectural independence should be distinguished from reliance on a data source.
- vs OpenVLA: VST retains the discrete action-token interface and improves VLA initialization through spatial pretraining; it does not introduce a new action-decoding architecture.
Rating¶
- Novelty: 4/5. BEV-assisted textual reconstruction supervision and the layout reward form a distinct methodological combination.
- Experimental Thoroughness: 4/5. Evaluations span modalities, training stages, data sources, and robotic transfer, but lack complete budgets and real-world deployment validation.
- Writing Quality: 3/5. The central argument is clear, but several numerical and ranking claims conflict with the tables.
- Value: 4/5. The work offers a reusable data and training pathway for adding spatial abilities to general VLMs.