Open-World 3D Scene Graph Generation for Retrieval-Augmented Reasoning¶

Conference: AAAI 2026
arXiv: 2511.05894
Code: None
Area: 3D Vision
Keywords: 3D Scene Graph, Open-World, Retrieval-Augmented Reasoning, Vision-Language Models, Embodied Interaction

TL;DR¶

Proposes a unified framework, OSU-3DSG, which integrates vision-language models for open-world 3D scene graph generation and supports four interactive tasks (scene question answering, visual grounding, instance retrieval, and task planning) via retrieval-augmented reasoning, achieving performance comparable to supervised methods under unsupervised settings.

Background & Motivation¶

Understanding 3D scenes is fundamental for tasks such as autonomous navigation and augmented reality. However, existing methods face several key challenges:

Closed-vocabulary limitations: Traditional 3D scene graph methods (e.g., 3DSSG) rely on predefined label sets and supervised annotations, failing to generalize to unseen objects and relations in novel environments.

Dependence on static annotations: The requirement for annotated RGB-D data and known camera poses is impractical in real-world, open-world scenarios.

2D-to-3D projection errors: Methods relying on 2D VLMs to infer 3D semantics through projection suffer from occlusions and viewpoint variations.

Core Idea: Leverages the open-vocabulary capabilities of VLMs to achieve annotation-free 3D scene graph generation, and then encodes the scene graph into a vector database to support retrieval-based multimodal reasoning and interaction. This eliminates the need for manual annotation while enhancing the scene-aware reasoning capabilities of LLMs through retrieval.

Method¶

Overall Architecture¶

The framework consists of two main components: 1. 3D Scene Graph Generator: Incrementally constructs semantic and spatial representations from RGB-D sequences. 2. Retrieval-Augmented Reasoning Module: Translates the scene graph into a vectorized knowledge base to support text/image conditional queries.

Key Designs¶

Open-World 3D Scene Graph Generation

Multi-frame Object Detection: Objects are detected from an RGB-D frame sequence, where each frame contains a color image $I$, a depth map $D$, camera intrinsic parameters $K$, and a pose $T_w^c \in SE(3)$. The detected objects are represented by oriented 3D bounding boxes: $$b_i = (c_i, \ell_i, R_i), \quad c_i \in \mathbb{R}^3, \ell_i \in \mathbb{R}_{>0}^3, R_i \in SO(3)$$

The detection confidence is modeled with a Beta distribution: $\sigma_i \sim \text{Beta}(\alpha_i, \beta_i)$, and the adaptive scale factor $\tau$ is dynamically adjusted based on the entropy of the predicted probabilities.

Masks are used for back-projection to obtain 3D points: $X_j^c = K^{-1}[u_j\ v_j\ 1]D_j$, which are then transformed into the world coordinate system.

Duplicate objects are merged every $L$ frames based on cosine similarity: $$S(\tilde{f}_i, \tilde{f}_j) = \frac{\langle \tilde{f}_i, \tilde{f}_j \rangle}{\|\tilde{f}_i\| \|\tilde{f}_j\|}$$ where the features are preprocessed using Mahalanobis whitening.

Best Viewpoint Selection and Annotation: We select the best viewpoint that maximizes visibility and projection coverage for each object: $$T_{w,i}^{c*} = \arg\max_{T_w^c \in \mathcal{P}} \left[A(\mathcal{P}(X_i^w, T_w^c)) \cdot V(X_i^w, T_w^c)^\gamma - \lambda D(T_{w,i}^c, T_w^c)\right]$ Then, LLaVA is used to semantically annotate the object under the best viewpoint.

Design Motivation: The best viewpoint reduces occlusion and blur, enabling the VLM to produce more accurate open-vocabulary annotations.

Reliable Object Filtering: Valid object pairs are filtered based on Euclidean distance and 3D IoU ($d_{thresh} = 0.5m$) to control the computational overhead of subsequent relation reasoning.

Semantic Relation Extraction: Uses Qwen2-VL-72B to infer top-5 semantic predicates for each valid object pair: $$\mathcal{R}_{ij} = (o_i, r_{ij}, o_j), \quad r_{ij} \in \mathcal{C}_{edge}$$ Background elements (e.g., floors, ceilings) are filtered out to obtain the final 3D semantic scene graph.

Retrieval-Augmented Semantic Reasoning

Vector Database Construction: The scene graph is reorganized into object-label-centric "chunks", where each chunk aggregates all instance information of that object category. These are mapped to a high-dimensional vector space using semantic encoders (CLIP/BERT/Text2Vec): $$\boldsymbol{\zeta}_i = \phi(\boldsymbol{\eta}_i), \quad \mathcal{D} = \{(\boldsymbol{\zeta}_i, \boldsymbol{\eta}_i)\}_{i=1}^N$$

Grounded Prompt Reasoning: Given a user query $q$, the top-k similarity retrieval is performed after encoding: $$\mathcal{E}_q = \text{Top-}k(\mathcal{D}, \boldsymbol{\xi}_q)$$

The retrieved scene information and the user query are combined into a structured prompt, which is passed into the LLM (Qwen-2-72B-Instruct) for grounded reasoning.

Four Scene Interaction Tasks
- Task I: Textual Scene QA — Answering natural language questions based on scene graph facts.
- Task II: Text-to-Visual Grounding — Grounding textual queries to spatial locations and best-view images.
- Task III: Multimodal Instance Retrieval — Supporting instance-level search with text/image/hybrid queries.
- Task IV: Open-Scene Task Planning — Decomposing high-level instructions into executable step sequences.

Loss & Training¶

This method is a zero-shot reasoning framework and does not involve end-to-end training, primarily relying on the reasoning capabilities of pretrained VLMs and retrieval mechanisms. Key hyperparameters include: - Cosine similarity threshold for object merging $\tau_{merge}$ - Object pair distance threshold $d_{thresh} = 0.5m$ - Open-vocabulary label matching: BERT embedding cosine similarity thresholds of 0.95 (objects) and 0.9 (predicates).

Key Experimental Results¶

Main Results¶

3D Scene Graph Generation (3DSSG Dataset):

Method	Type	Object R@1	Predicate R@1	Predicate R@3	Relation R@1	Relation R@3
3DSSG	Closed	0.82	0.83	0.85	0.63	0.63
MonoSSG	Closed	0.86	0.89	0.90	0.89	0.90
VL-SAT	Closed	0.82	0.94	0.94	0.87	0.88
Open3DSG	Open	0.65	0.81	0.81	0.70	0.72
BBQ	Open	0.59	0.61	0.61	0.68	0.68
OSU-3DSG (Ours)	Open	0.83	0.95	0.97	0.78	0.80

As a zero-shot method, the predicate prediction surpasses all closed-vocabulary methods (R@1 0.95 vs. VL-SAT 0.94) and significantly outperforms open-vocabulary baselines.

Scene Interaction Tasks:

Task	Metric	OSU-3DSG	GPT-4o	Gemini	ChatGLM
Scene QA	Accuracy	0.84	0.82	0.80	0.72
Task Planning	Success Rate	87.5%	72.9%	65.4%	58.7%
Task Planning	Executability	81.25%	78.2%	69.8%	62.3%

Ablation Study¶

Semantic Relation Extractor (SRE) Filtering Strategies:

IoU	Distance	No. of Triplets	Predicate R@1	Relation R@1	Description
✗	✗	291	0.95	0.94	No filtering, high computational overhead
✔	✗	30	0.76	0.83	IoU filtering alone is insufficient
✗	✔	11	0.85	0.75	Distance filtering alone, too few triplets
✔	✔	34	0.87	0.78	Best balance

Combining IoU and distance constraints reduces the candidate triplets to 34, significantly decreasing VLM inference costs while maintaining high recall.

Key Findings¶

Zero-shot scene graph generation can reach or even exceed the performance of supervised methods in predicate prediction.
Retrieval-augmented reasoning outperforms GPT-4o in scene QA, demonstrating the value of structured scene knowledge.
Best viewpoint selection is crucial for object recognition accuracy.
A fixed distance threshold (0.5m) may not be suitable for all scene densities and object scales.

Highlights & Insights¶

Unified open-world 3D understanding framework: Integrates scene graph generation and multimodal reasoning into a consistent system, covering the entire pipeline from perception to planning.
Zero-shot outperforming supervised: Leverages the knowledge of large-scale VLMs to surpass fully supervised methods in predicate prediction, demonstrating the immense potential of zero-shot VLM capabilities.
Advantages of retrieval-augmented strategy: Compared to directly feeding all scene information into LLM prompts, retrieving relevant parts before prompting is more efficient and accurate.
Best viewpoint selection mechanism: Automatically selects the optimal viewing angle for each object, reducing occlusion and ambiguity to improve the quality of VLM annotations.

Limitations & Future Work¶

The distance threshold for object pair filtering (0.5m) is fixed, and its generalizability across scenes remains to be verified.
Relation reasoning heavily relies on the reasoning capability of Qwen2-VL-72B; the performance ceiling of the proposed method is bound by the LLM's capacity.
The absolute accuracy in the visual grounding task is low (~0.23); joint 3D spatial and textual reasoning remains an open challenge.
Validated only on indoor scenes; the scalability to large-scale outdoor scenes is unknown.
Task planning lacks closed-loop validation with real robot execution.

Open3DSG (2024): A pioneering work in open-vocabulary 3D scene graphs, but it still relies on annotated RGB-D data and fixed poses.
BBQ (2024, Linok et al.): An object-centric open-world scene graph model, serving as the open-vocabulary baseline in this paper.
ConceptGraphs: A similar VLM-driven 3D scene graph approach, but without integrated retrieval-augmented reasoning.
Insights: Formulating 3D scene understanding as a "knowledge-base construction + retrieval-augmented reasoning" paradigm can be extended to other structured scene understanding tasks.

Rating¶

Novelty: ⭐⭐⭐⭐ (The combination of retrieval-augmented reasoning and 3D scene graphs is a relatively new paradigm)
Experimental Thoroughness: ⭐⭐⭐⭐ (Covers four interactive tasks, but the evaluation scale for each task is small)
Writing Quality: ⭐⭐⭐⭐ (The framework is clearly described, but contains many formulas; some definitions could be simplified)
Value: ⭐⭐⭐⭐ (An important direction for open-world 3D understanding, with encouraging zero-shot performance)