Empowering Large Language Models with 3D Situation Awareness

Conference: CVPR 2025
arXiv: 2503.23024
Code: None
Area: 3D Vision
Keywords: 3D Scene Understanding, Situation Awareness, LLM, Point Cloud, Ego-centric Perspective

TL;DR

This paper proposes to automatically generate a situation-aware dataset, View2Cap (over 200k descriptions, 550k+ QAs), utilizing the camera trajectories of RGB-D videos. It designs a Situation Grounding (SG) module that converts pose estimation into an anchor classification task, enabling 3D LLMs to understand ego-centric spatial relationship descriptions (e.g., how "left" and "right" change depending on the viewpoint), achieving 54.0% EM@1 on SQA3D.

Background & Motivation

Background: Applying LLMs to 3D scene understanding is an emerging trend. Existing methods like 3D-LLM, LL3DA, and LEO align point clouds with text for 3D captioning, VQA, and visual grounding. However, the fundamental difference between 3D scenes and 2D images is that the observer's position and orientation (situation) change spatial descriptions—for instance, the same sofa can be "on the left" or "on the right" depending on different perspectives.

Limitations of Prior Work: (1) Existing 3D-text datasets are mostly based on global perspectives (scene graphs), ignoring first-person situational context; (2) Data generation based on scene graphs relies on manually annotated 3D instance labels, which are costly and cover incomplete categories (especially small objects and rare classes); (3) Relations between objects are described using fixed templates, failing to handle open-vocabulary scenarios; (4) Situational descriptions in SQA3D rely on manual writing, making it difficult to scale up to large-scale training.

Key Challenge: 3D LLMs require a large amount of situation-aware data to understand first-person perspectives, but manual annotation costs are extremely high and existing datasets lack situational information.

Goal: (1) How to automatically generate situational 3D text data at low cost? (2) How to enable LLMs to explicitly ground textual descriptions to positions and orientations in 3D space?

Key Insight: 3D reconstruction scans are often reconstructed from RGB-D videos, where camera trajectories naturally represent the first-person perspective of human explorers. Utilizing camera extrinsics of these video frames, combined with 2D VLMs to generate descriptions, can yield situation-aware point cloud-text data.

Core Idea: Use camera trajectories of RGB-D videos as the source of situation context, employ 2D VLMs to generate descriptions and QA, and design an anchor mechanism to convert situation pose estimation into a classification task, empowering 3D LLMs with first-person spatial understanding capabilities.

Method

Overall Architecture

The method consists of two parts: (1) Data pipeline—frames are extracted from RGB-D videos in ScanNet/3RScan/Matterport3D, with LLaVA-OneVision generating simple/detailed descriptions and four types of QA, and GPT-4 validating and ranking them to obtain the View2Cap dataset (231K descriptions + 553K QA spanning 2841 scenes); (2) Model architecture—a point cloud encoder extracts instance features, a connector fuses spatial and semantic information, an LLM (LLaMA 3.1) processes visual and textual tokens, and a Situation Grounding (SG) module predicts the observer's position and orientation. Training is conducted in three stages: region-text alignment \(\rightarrow\) situation grounding \(\rightarrow\) instruction tuning.

Key Designs

1. View2Cap Automated Data Generation Pipeline:

   • Function: Automatically generate large-scale situation-aware 3D text data at low cost.
   • Mechanism: For each frame of RGB-D videos: (a) Obtain precise position and orientation from camera extrinsics as the situation; (b) Extract the region point cloud visible from this view using depth information and camera parameters; (c) Utilise LLaVA-OneVision to generate two types of descriptions from 2D images (simple: main objects and relationships; detailed: background and environment) and four types of QA (object identification, spatial relationships, visual features, overall layout). Then, GPT-4 is used to validate the quality of descriptions based on 3D labels (scores 0-5, View2Cap average score is 3.09, 3.31 after refinement), and rank the QA to filter out low-quality items. The final dataset size is over 10 times larger than SQA3D, with an average description length of 54.73 words vs 17.49 words for SQA3D.
   • Design Motivation: Camera trajectories are "free" sources of situational information, and 2D VLMs already possess strong image understanding capabilities. Distilling 2D knowledge to 3D avoids expensive 3D annotation, and the free-form text descriptions from VLMs are much richer than templated scene-graph descriptions.

2. Situation Grounding (SG) Module:

   • Function: Explicitly predict the observer's position and orientation in the 3D scene.
   • Mechanism: Treat each object in the scene as an anchor point, utilizing its center coordinate \(\mathbf{a}_k^{pos}\) and orientation \(\mathbf{a}_k^{rot}\) (uniformly set to face the center of the room) as reference points. The LLM outputs a special [GRD] token, whose hidden state \(\mathbf{h}_{GRD}\) is concatenated with the hidden state of each object \(\mathbf{h}_k\). An MLP then predicts three quantities: confidence \(c_k \in [0,1]\), position offset \(\Delta\mathbf{p}_k \in \mathbb{R}^3\), and rotation angle bin \(\hat{b}_k\). The rotation angle is discretized into \(B\) bins (\([-pi, pi]\)) to transform it into a classification problem. During inference, the anchor with the highest confidence is selected: \(k^* = \arg\max_k c_k\), and the predicted pose is \(\hat{\mathbf{s}}^{pos} = \mathbf{a}_{k^*}^{pos} + \Delta\mathbf{p}_{k^*}\).
   • Design Motivation: Directly predicting absolute poses is extremely difficult. Using anchors decomposes the problem into: (1) selecting the closest object as reference; (2) predicting relative offsets and angle differences, which reduces the learning difficulty. Discretizing rotation into classification further simplifies the otherwise highly challenging continuous angle prediction.

3. Three-stage Training Strategy:

   • Function: Step-by-step construction of capabilities from feature alignment to situation understanding, and finally downstream reasoning.
   • Mechanism: Stage 1 (Region-Text Alignment): Train the connector using the region point-cloud-description pairs from View2Cap to map point cloud features into the LLM embedding space. Occluded objects are filtered using depth information, leaving only visible instances to reduce ambiguity. Stage 2 (Situation Grounding): Train the SG module, where losses include L2 position loss \(\mathcal{L}_{pos}\) (supervising only anchors within a distance \(D\) of the ground truth), rotation cross-entropy loss \(\mathcal{L}_{rot}\), and confidence loss \(\mathcal{L}_{conf}\) (targeting a distance-decaying objective). Stage 3 (Instruction Tuning): Fine-tune the entire model on downstream datasets such as 3D VQA. LoRA is applied to the LLM throughout the process.
   • Design Motivation: Direct end-to-end training easily diverges on large-scale data. Progressive three-stage training allows the model to learn low-level alignment first, followed by situation understanding, and finally adaptation to specific tasks.

Loss & Training

Situation grounding stage: \(\mathcal{L} = \mathcal{L}_{pos} + \mathcal{L}_{rot} + \mathcal{L}_{conf}\). Instruction tuning stage: standard autoregressive language model cross-entropy loss \(\mathcal{L}_{ans}\). LoRA is used for fine-tuning LLaMA 3.1 throughout.

Key Experimental Results

Main Results

3D Scene Understanding Tasks:

Model	Scan2Cap CIDEr	ScanQA EM@1	SQA3D EM@1
LEO	72.4	24.5 (47.6)	50.0 (52.4)
LL3DA	65.2	-	-
3D-VisTA	66.9	22.4	48.5
Ours	75.2	22.9 (40.2)	54.0 (56.0)

Situation Grounding Performance:

Model	[email protected]	[email protected]	Acc@15°	Acc@30°
Random	7.2	25.8	8.4	16.9
SQA3D	9.5	29.6	8.7	16.5
3D-VisTA	11.7	34.5	16.9	24.2
Ours	17.4	36.9	24.1	28.5

Ablation Study

Ablation of Situation Grounding Module Design:

Configuration	[email protected]	[email protected]	Acc@15°	Acc@30°
LEO + SG (no anchor)	8.3	30.4	10.9	19.5
+ Anchor Mechanism	13.7	32.2	16.9	21.8
+ Discretized Rotation Bins	13.6	32.3	21.6	25.0
+ View2Cap Pre-training	17.4	36.9	24.1	28.5

Downstream Tasks Ablation:

Configuration	ViewQA EM	SQA3D EM	ScanRefer [email protected]
LEO	39.3	52.4	36.1
+ SG module	40.2	53.2	38.3
+ View2Cap	42.0	56.0	42.8

Key Findings

• The anchor mechanism improves [email protected] from 30.4% to 32.2% (+5.9%), demonstrating the effectiveness of decomposing absolute pose prediction into anchors + offsets.
• Rotation discretization improves Acc@15° from 16.9% to 21.6% (+27.8%), indicating classification is better suited for angle prediction than regression.
• View2Cap pre-training significantly boosts all metrics, SQA3D EM@1 rises from 52.4% to 56.0%, and ScanRefer rises from 36.1% to 42.8%.
• Scan2Cap CIDEr reaches 75.2, exceeding LEO by 2.8 points; SQA3D EM@1 reaches 54.0%, exceeding LEO by 4%.
• View2Cap descriptions are more detailed and accurate than SceneVerse (e.g., capturing a glass vase on a table or an open book that SceneVerse missed).

Highlights & Insights

• Clever utilization of "free" data: Camera trajectories from RGB-D videos are existing but underutilized sources of situation information. Distilling 2D VLM knowledge to 3D avoids expensive 3D labeling. This data generation paradigm can be generalized to any 3D task requiring viewpoint information.
• Elegant conversion from pose estimation to classification: Decomposing continuous pose prediction into "anchor selection + offset prediction + angle classification" significantly lowers the difficulty of learning. This design concept can be migrated to other tasks requiring localization in 3D.
• Fundamental meaning of situation awareness: It explicitly points out that the core difference between 3D and 2D understanding lies in the observer's perspective, an essential difference that most prior 3D LLM works ignored.

Limitations & Future Work

• Situation grounding [email protected] is only 17.4%, making precise localization still challenging.
• It relies on the quality of a pre-trained instance segmentation model (Mask3D); segmentation errors can cascade.
• Setting anchor rotations uniformly to face the room center is a simplified assumption; actual object orientations might contain useful information.
• VLM-generated descriptions in View2Cap inevitably suffer from hallucination (average verification score from GPT-4 is only 3.09/5).
• Testing the practical value of situation awareness for embodied AI by integrating it with navigation tasks has not yet been conducted.

Related Work & Insights

• vs LEO: LEO is a general-purpose 3D LLM but lacks situation awareness, scoring only 50.0% on SQA3D. In contrast, incorporating situational data and the SG module in this work achieves 54.0%, demonstrating that situational information is crucial for spatial reasoning.
• vs SQA3D: The situation descriptions in the SQA3D dataset are manually written, yielding only 20K descriptions with a short average length (17.49). View2Cap automatically generates 231K descriptions with an average length of 54.73, achieving a comprehensive improvement in both scale and quality.
• vs SceneVerse: SceneVerse uses scene graphs + templates to generate descriptions, which relies on 3D labels and has fixed relations. View2Cap uses free-text generated by VLMs without requiring 3D labels, covering much more detail.

Rating

• Novelty: ⭐⭐⭐⭐ The perspective of situation awareness is novel, and the anchor-based localization design is clever, though the core remains data generation + module addition.
• Experimental Thoroughness: ⭐⭐⭐⭐ Covers 3 baselines + situation grounding + captioning + VQA + ablation studies, which is comprehensive.
• Writing Quality: ⭐⭐⭐⭐ Clear problem definition and systematic method description, though some parts contain heavy notations.
• Value: ⭐⭐⭐⭐ Situation awareness is an important addition to 3D LLMs, and the dataset holds valuable contribution to the community.

Related Papers

• [ECCV 2024] PointLLM: Empowering Large Language Models to Understand Point Clouds
• [ICCV 2025] 3DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understanding
• [CVPR 2025] Perception Tokens Enhance Visual Reasoning in Multimodal Language Models
• [CVPR 2025] DepthCues: Evaluating Monocular Depth Perception in Large Vision Models
• [CVPR 2025] Grounding 3D Object Affordance with Language Instructions, Visual Observations and Interactions