SpatialBoost: Enhancing Visual Representation through Language-Guided Reasoning¶
Conference: ECCV 2026
Paper: ECCV Original
Area: LLM Reasoning
Keywords: visual representation learning, 3D spatial reasoning, large language models, chain-of-thought (CoT), dual-channel attention
TL;DR¶
SpatialBoost converts dense 3D spatial geometry and relations from 2D/multi-view images into hierarchical multi-turn Chain-of-Thought language descriptions (pixel-level โ object-level โ scene-level), fine-tuning vision encoders through a frozen LLM via dual-channel attention to empower general vision backbones with superior 3D spatial perception without catastrophic forgetting.
Background & Motivation¶
Pre-trained vision representation models (such as DINOv2, DINOv3, SigLIPv2, and CLIP) have achieved remarkable breakthroughs across diverse downstream tasks including image classification, semantic segmentation, and vision-language understanding. However, these foundational vision encoders are predominantly pre-trained on 2D planar image datasets. Consequently, they fundamentally struggle to perceive real-world 3D spatial structures and geometric relationships between objects and background scenes. This deficit causes multimodal foundation models to suffer severe spatial hallucinations when inferring 3D relative orientations or physical distances, while also yielding suboptimal control policies in vision-based robot manipulation compared to models equipped with explicit 3D inputs.
Directly training vision models on multi-view images with masked prediction or contrastive learning offers an intuitive path toward spatial awareness. Nonetheless, such paradigms heavily rely on carefully curated real multi-view datasets or synthetic simulation environments, posing formidable scalability bottlenecks. In contrast, specialized geometric vision models (such as monocular metric depth estimators and 3D surface reconstruction networks) can reliably infer pixel depths and point clouds from standard 2D or multi-view imagery. However, systematic and scalable supervisory signals to unify and inject these disparate geometric cues into general vision encoders have remained elusive.
Natural language provides an intrinsically sequential, highly structured medium capable of composing dense geometric relationships and 3D topologies into cohesive reasoning narratives. The angle of attack in this paper is to utilize the structured reasoning and generative capabilities of Large Language Models (LLMs) to bridge the gap between continuous geometric measurements and visual representations. Core idea: convert dense 3D spatial cues into hierarchical multi-turn Chain-of-Thought linguistic reasoning trajectories (spanning pixel coordinates, object bounding volumes, and scene-level metric distances), and backpropagate text cross-entropy loss through a frozen LLM into a newly introduced dual-channel attention module in the vision encoder, thereby injecting rich 3D spatial awareness while preserving pre-trained 2D semantic capabilities.
Method¶
Overall Architecture¶
The objective of SpatialBoost is to augment arbitrary pre-trained vision encoders \(f_V\) (e.g., DINOv3, SigLIPv2, OpenCLIP) with dense 3D spatial reasoning without sacrificing their baseline visual representations. The training framework comprises the vision encoder \(f_V\), a trainable projection layer \(g_P\), and an LLM decoder \(f_L\) (instantiated with Qwen-2.0-7B). Training proceeds across three structured stages: Stage 1 aligns image features with the LLM token space, Stage 2 performs multi-view visual instruction tuning, and Stage 3 injects hierarchical spatial knowledge by fine-tuning the vision encoder equipped with dual-channel attention layers while keeping the LLM parameters frozen.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
A["Single-view / Multi-view Image Inputs"] --> B["Stage 1 & 2: Feature Alignment & Multi-view Tuning"]
B --> C["Hierarchical Spatial Reasoning Data Construction<br/>Pixel-level Depth โ Object-level Bounds โ Scene Distances"]
C --> D["Dual-channel Attention Fine-tuning<br/>Freeze LLM and original encoder weights, update only Attn+ and ฮฑ"]
D --> E["Spatially-Enhanced Vision Encoder"]
Key Designs¶
1. Cross-Modal Alignment and Multi-View Instruction Tuning: Establishing Gradient Flow Without rigorous feature alignment between the visual encoder and the LLM embedding space, cross-entropy training gradients from text tokens cannot effectively propagate back to guide visual representations. SpatialBoost addresses this via a progressive two-stage alignment strategy. In Stage 1 (feature alignment), both the vision encoder \(f_V\) and the language model \(f_L\) are kept completely frozen, optimizing only a 2-layer MLP projector \(g_P\) on image-text pairs to stably project visual features \(z_v = f_V(x)\) into the text token space. In Stage 2 (visual instruction tuning), the vision encoder remains frozen while both the projector \(g_P\) and the LLM \(f_L\) are tuned jointly on single-view instructions alongside multi-view VQA pairs synthesized via LPIPS pair filtering and GPT-4o prompting. This establishes the necessary capacity for processing multi-view visual tokens and complex spatial instructions.
2. Hierarchical Spatial Reasoning Trajectories: Structuring Pixel-to-Scene 3D Cues Existing spatial visual question answering datasets typically consist of flat, single-turn binary questions with minimal information density, failing to teach coherent global 3D geometry. SpatialBoost leverages specialized vision models (Depth Pro for metric depth, SAM 2 for instance segmentation, and VGGT for 3D point cloud reconstruction) to construct 12-turn Chain-of-Thought (CoT) dialogues organized hierarchically: - Pixel-level Reasoning: Queries absolute or relative 3D point coordinates and metric depth (e.g., "What is the depth value at \((x, y)\)? Which point is closer to the camera?"), establishing fine-grained geometric foundations; - Object-level Reasoning: Uses pixel-level geometric rationales to infer 3D bounding cubes, orientations, and semantic spatial relations (e.g., "Is the man wearing glasses to the left of the man in the orange vest?"); - Scene-level Reasoning: Aggregates 3D object centroids to compute exact physical Euclidean distances across multiple entities (e.g., "The distance between the two workers is 50 centimeters"); - General Scene Captions: Appends 2 final turns of holistic scene descriptions to retain high-level semantic generalization alongside granular spatial reasoning.
3. Dual-Channel Attention Fine-Tuning: Preventing Catastrophic Semantic Forgetting Direct full fine-tuning of all vision encoder parameters during Stage 3 risks severe degradation of pre-trained 2D discriminative features. To preserve established representations, SpatialBoost attaches a parallel attention branch \(\text{Attn}^+(\cdot)\) to each existing attention layer \(\text{Attn}(\cdot)\) within \(f_V\), initialized with identical weights. Outputs from both channels are combined using an element-wise learnable gating vector \(\alpha \in (0, 1)^d\):
where \(\alpha = \text{sigmoid}(a)\), with parameter vector \(a \in \mathbb{R}^d\) initialized to zero (yielding \(\alpha = 0.5\)). During Stage 3, the original backbone parameters and the LLM remain strictly frozen, while only \(\text{Attn}^+\), \(\alpha\), and the projector \(g_P\) receive gradient updates. This ensures that the encoder retains its pre-trained visual knowledge from the start, smoothly integrating language-guided 3D geometric awareness over the course of training.
A Worked Example¶
Consider a warehouse surveillance image featuring multiple personnel and equipment: 1. Pixel-level: The LLM evaluates "Which point is closer to the viewer? Point 1: (0.49, 0.60), Point 2: (0.23, 0.85)". The rationale checks the underlying depth field and concludes Point 2 is closer than Point 1. 2. Object-level: The prompt asks "Is the man wearing glasses on the left side of the man wearing an orange vest?". Grounded by predicted 3D bounding boxes \([0.47, 0.25, 0.59, 0.55, 0.38, 0.77]\) and \([0.54, 0.42, 0.64, 0.61, 0.53, 0.91]\), the model resolves horizontal relative coordinates and answers "Yes". 3. Scene-level: The query assesses "How far is the man wearing glasses from the man wearing an orange vest?". Leveraging the 3D centroids, the model computes that the physical distance is 50 centimeters. 4. Scene Caption: The model provides a broad summary: "Workers wearing vests gather in a warehouse as a manager demonstrates a metal component from a crate." Backpropagation from this multi-turn trajectory directly supervises \(\text{Attn}^+\) in the visual encoder.
Loss & Training¶
All three stages optimize standard next-token autoregressive cross-entropy loss:
During Stage 3, \(f_L\) is frozen, so backpropagated gradients update only \(\text{Attn}^+\) and \(g_P\) via AdamW. The dataset consists of 100K single-view SA1B images and 200K multi-view samples (drawn from Ego4D, ScanNet, and Mip-NeRF 360), trained for a single epoch.
Key Experimental Results¶
Main Results¶
SpatialBoost was validated across diverse foundational vision encoders (OpenCLIP ViT-G/14, SigLIPv2 ViT-g/16, DINOv2 ViT-g/14, and DINOv3 ViT-7B/16). Downstream evaluations include monocular depth estimation on NYUd, semantic scene parsing on ADE20K, unified 3D scene understanding on Lexicon3D (ScanQA, SQA3D, ScanRefer, and ScanNet 3D semantic segmentation), and vision-based robotic manipulation on CortexBench. In all downstream benchmarks, the visual backbone remains frozen.
| Task / Benchmark | Metric | Baseline Encoder | Baseline Score | +SpatialBoost (Ours) | Gain / Improvement |
|---|---|---|---|---|---|
| NYUd Monocular Depth (Linear) | RMSE โ | DINOv3 | 0.31 | 0.25 | -0.06 (19.4% error reduction) |
| NYUd Monocular Depth (Linear) | RMSE โ | SigLIPv2 | 0.51 | 0.39 | -0.12 (23.5% error reduction) |
| ADE20K Semantic Segmentation (Linear) | mIoU (%) โ | DINOv3 | 55.9 | 59.7 | +3.8%p (New SOTA) |
| ADE20K Semantic Segmentation (+ms) | mIoU (%) โ | DINOv3 | 60.3 | 63.1 | +2.8%p |
| ScanQA 3D Visual-Language Reasoning | BLEU-1 โ | DINOv3 | 40.6 | 43.3 | +2.7%p |
| SQA3D Situated 3D Scene QA | BLEU-1 โ | DINOv3 | 51.4 | 54.9 | +3.5%p |
| ScanRefer 3D Visual Grounding | Acc (%) โ | DINOv3 | 56.2 | 61.1 | +4.9%p |
| ScanNet 3D Semantic Segmentation (3D SU) | mIoU (%) โ | OpenCLIP | 6.9 | 54.9 | +48.0%p (Massive leap) |
| CortexBench Robot Manipulation | 4-Domain Avg. โ | DINOv3 | 72.8 | 80.8 | +8.0%p |
| ImageNet-1K Classification (Linear) | Top-1 Acc (%) โ | DINOv3 | 88.4 | 90.2 | +1.8%p (Surpasses 90%) |
Ablation Study¶
Ablation experiments on DINOv2-ViT-L/14 isolate the impact of the LLM-based supervision, conversational ordering, and attention architectures:
| Configuration / Variant | ImageNet Top-1 (%) โ | ADE20K mIoU (%) โ | NYUd Depth RMSE โ | ScanQA BLEU-1 โ | Note |
|---|---|---|---|---|---|
| DINOv2 Baseline | 86.3 | 47.7 | 0.38 | 39.2 | Unmodified pre-trained backbone |
| + Linear (Depth supervision only) | 85.7 (-0.6) | 47.9 (+0.2) | 0.35 (-0.03) | 36.9 (-2.3) | Suffers semantic degradation |
| + SAM Decoder (Segmentation only) | 86.3 (+0.0) | 50.1 (+2.4) | 0.42 (+0.04) | 37.6 (-1.6) | Severely degrades depth perception |
| + VGGT Decoder (3D Reconstruction) | 84.8 (-1.5) | 45.6 (-2.1) | 0.35 (-0.03) | 37.3 (-1.9) | Pixel-level 3D loss harms 2D classification |
| + LLM Decoder (SpatialBoost) | 88.3 (+2.0) | 51.5 (+3.8) | 0.32 (-0.06) | 40.0 (+0.8) | Superior across all downstream tasks |
| Multi-turn Order: Reversed (Scene โ Pixel) | 87.4 | 48.4 | 0.35 | - | Inverted logic hurts lower-level reasoning |
| Multi-turn Order: Shuffled Random | 87.4 | 48.5 | 0.36 | - | Unstructured flow impairs representations |
| Multi-turn Order: Forward (Pixel โ Scene) | 87.6 | 48.9 | 0.34 | - | Canonical micro-to-macro progression |
Key Findings¶
- Language Supervision Outperforms Dedicated Pixel Decoders: As demonstrated in Table 6, replacing the LLM decoder with dedicated task heads (such as SAM for segmentation or VGGT for 3D reconstruction) results in task-specific overfitting and harms general visual features (e.g., ImageNet accuracy drops by 1.74% under VGGT supervision). In contrast, LLM-guided language reasoning simultaneously reduces NYUd depth RMSE from 0.38 to 0.32 while boosting ImageNet linear probing accuracy from 86.3% to 88.3%.
- Forward Hierarchical CoT is Essential: Ablation results in Table 7 verify that reversing or shuffling the multi-turn QA trajectory causes consistent performance degradation. Anchoring global scene distances upon prerequisite pixel-level depths and object-level bounding boxes mirrors physical geometric induction.
- Dual-Channel Attention Prevents Catastrophic Forgetting: While full fine-tuning degrades baseline image classification accuracy, the dual-channel attention module shields pre-trained weights behind the frozen \(\text{Attn}(\cdot)\) stream and smoothly integrates 3D spatial cues through \(\text{Attn}^+(\cdot)\).
Highlights & Insights¶
- Geometric Information Expressed through Language: SpatialBoost demonstrates that continuous geometric quantities (depth coordinates, bounding cubes, metric distances) can be effectively represented through structured natural language, allowing visual encoders to tap into the broad contextual capacity of LLMs.
- Mutual Reinforcement of 3D Geometry and 2D Semantics: Rather than trading off general visual representation quality for 3D awareness, spatial reasoning actually sharpens object boundaries and scene understanding, producing non-trivial gains on ImageNet classification (+1.8%p on DINOv3) and instance image retrieval.
- Plug-and-Play Post-Training Paradigm: By incorporating dual-channel attention without altering the underlying ViT architecture, SpatialBoost serves as an efficient post-training pipeline for pre-trained vision backbones using modest dataset sizes (~300K samples).
Limitations & Future Work¶
- Reliance on Pseudo-Ground-Truth Geometric Estimators: Synthetic spatial QAs rely on existing perception models (Depth Pro, SAM 2, VGGT). Any systematic biases or artifacts produced by these upstream models in challenging conditions (e.g., severe specular reflections or extreme occlusions) will propagate into the linguistic reasoning targets.
- Multi-Stage Training Overhead: The three-stage pipeline (projector alignment, multi-view tuning, and dual-channel encoder fine-tuning) requires alternating parameter freezes and dataset switching, which is more complex than unified single-stage end-to-end training.
- Future Directions: Exploring direct reinforcement learning with physical environment feedback to self-refine spatial reasoning, eliminating dependence on pre-computed geometric pseudo-labels.
Related Work & Insights¶
- vs DINOv2 / DINOv3: While DINO models excel at semantic clustering and dense feature correspondence, they lack explicit awareness of metric depth and 3D object relationships. SpatialBoost introduces a modular dual-channel attention layer that elevates DINOv3 to 59.7 mIoU on ADE20K and unlocks strong robotic control capabilities.
- vs TIPS / PE-Core: Existing spatially-aware encoders like TIPS and Perception Encoder integrate spatial sensitivity via multi-task pre-training from scratch on huge multimodal corpora. SpatialBoost instead operates as an efficient post-training adapter on top of frozen LLM decoders, achieving comparable or superior spatial capabilities with only 300K training instances.
Rating¶
- Novelty: โญโญโญโญโญ Elegant framework translating dense continuous 3D geometry into hierarchical language CoT trajectories to supervise vision encoders.
- Experimental Thoroughness: โญโญโญโญโญ Comprehensive evaluations across dense depth/segmentation, 3D scene understanding, robot manipulation, and 2D classification.
- Writing Quality: โญโญโญโญโญ Clear exposition, detailed figures, and rigorous self-consistent ablation studies.
- Value: โญโญโญโญโญ Provides a scalable, generalizable solution to the spatial awareness limitations of vision foundation models.