RAG-3DSG: Enhancing 3D Scene Graphs with Re-Shot Guided Retrieval-Augmented Generation

Conference: ECCV 2026
Paper: ECCV Official Page
Code: https://github.com/CYandYue/RAG-3DSG
Area: 3D Vision
Keywords: 3D scene graphs, open-vocabulary perception, re-shot viewpoints, uncertainty estimation, retrieval-augmented generation

TL;DR

RAG-3DSG diagnoses conflicting multi-view captions using virtual re-shots of object point clouds, then retrieves reliable neighboring objects to rectify uncertain nodes, improving SceneFun3D object R@10 from OpenFunGraph's 87.8% to 90.2% with GPT-4o.

Background & Motivation

Open-vocabulary 3D scene graphs compress RGB-D observations into object nodes and their relationships, enabling language-based queries and reasoning for robots. Methods such as ConceptGraphs typically caption object crops in individual frames before aggregating them across views; another family aggregates visual embeddings before producing categories or text. Both depend on an easily overlooked assumption: different views describe the same object rather than its occluder or crop background.

For example, when a vase obscures a table, multiple crops may all be captioned as a vase. Averaging embeddings or asking a large language model (LLM) to merge descriptions can mistake repeated errors for consistent evidence. A stronger model cannot recover a viewpoint that was never captured. Existing keyframe selection only chooses relatively good photographs from the original trajectory. This paper instead uses reconstructed object geometry to choose a clearer virtual viewing direction, while recognizing that point-cloud rendering loses texture and cannot simply replace real photographs.

Re-shooting therefore checks the original descriptions before assigning a final category. Only relatively reliable objects enter the retrieval context that helps disambiguate the remaining nodes. Core Idea: assess node reliability through semantic agreement between re-shot views and original crops, then retrieve reliable neighbors and combine real and re-shot images to rectify uncertain nodes.

Method

Overall Architecture

The input is an RGB-D sequence with camera poses; the output is a 3D scene graph with natural-language node descriptions and relationship labels. Scale-Adaptive Fusion first constructs a global object list, followed by Re-shot Consistency Diagnosis and Object-level Retrieval Rectification, and finally relationship generation. Here, "active" refers to virtual viewpoint selection over reconstructed geometry, not commanding a robot to capture new photographs.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["RGB-D sequence<br/>and camera poses"] --> B["Scale-Adaptive Fusion"]
    B --> C["Global object clouds<br/>and original crop captions"]
    C --> D["Re-shot Consistency Diagnosis"]
    D -->|Reliable neighbors and uncertain nodes| E["Object-level Retrieval Rectification"]
    E --> F["Constrained Relationship Generation"]
    F --> G["3D scene graph"]

The diagram represents inference data flow, without an additional supervised training branch. Segmentation and vision-language representations come from pretrained models; the contribution primarily changes geometric fusion, caption selection, and the organization of prompting context.

Key Designs

1. Scale-Adaptive Fusion: preserve small-object detail while reducing redundant points on large objects

For each frame, class-agnostic SAM produces object masks, CLIP extracts semantic embeddings, and depth plus camera poses project masks into object point clouds. A fixed voxel size either leaves large background structures with too many points or samples small objects too sparsely. The latter also undermines subsequent re-shot recognition. The method therefore varies voxel size with the spatial extent of an object's 3D bounding box, sampling large objects more coarsely while retaining denser geometry for small objects. Equation (1) is corrupted in the text cache, so this note explains the scale-adaptive principle from the prose without guessing its exact expression.

Local instances are associated with global objects using both semantic similarity and spatial overlap. The former is CLIP embedding cosine similarity; the latter is a dynamic nearest-neighbor ratio whose matching-distance threshold adapts to object scale. The two scores are added, matches above a threshold merge point clouds and update semantic embeddings through a moving average, and unmatched instances become new global objects. This stage determines the geometry used for re-shooting, rather than merely accelerating preprocessing. If instances were incorrectly merged, semantic rectification may not recover the correct segmentation.

2. Re-shot Consistency Diagnosis: use another viewpoint to select captions supporting the same semantics

For each global object, the system retains the top-k original crops with the highest segmentation confidence and uses a vision-language model (VLM) to caption them individually. It then uniformly samples 64 virtual cameras on a hemisphere around the object, scoring each using visible-point proportion, a preference for horizontal viewing, and a prior based on the average original viewing direction. Visibility favors informative views, the horizontal term avoids ambiguity from top-down or bottom-up perspectives, and the prior discourages poorly reconstructed backsides. Only the highest-scoring view is rendered and passed to the VLM; semantic inference is not run separately on all 64 candidates.

The experimental weights are 0.6, 0.2, and 0.2, respectively. Because the component expressions in cached Equation (2) are incomplete, only the weighted structure explicitly described in the prose is retained, without inventing the normalization of the directional terms:

\[ S_{\mathrm{view}}=0.6S_{\mathrm{vis}}+0.2S_{\mathrm{horiz}}+0.2S_{\mathrm{prior}}. \]

After obtaining the re-shot caption, the system compares its CLIP text embedding with that of each original caption. The following equivalent notation summarizes the prose and Equation (3): \(C^*\) denotes the highest-similarity consensus cluster, and \(u\) is the uncertainty used for subsequent ranking.

\[ s_i=\cos\bigl(\mathrm{CLIP}(c^{\mathrm{reshot}}),\mathrm{CLIP}(c_i)\bigr),\qquad u=1-\frac{1}{|C^*|}\sum_{i\in C^*}s_i. \]

The similarity scores are clustered with KMeans using 3 clusters, retaining only the cluster with the highest mean similarity. An LLM merges the original captions in that cluster with the re-shot caption to form a consensus description. Selection is based on agreement with the re-shot, not on cluster population, so repeated occlusion-induced errors do not win merely by being numerous. Reliability is the mean similarity within this cluster, not a calibrated probability of correctness or the VLM's self-reported confidence.

3. Object-level Retrieval Rectification: allow only relatively reliable neighbors to supply context

The system ranks objects by \(u\) and uses a VLM to inspect crops and filter background objects. The top 50% with low uncertainty form the retrieval document, and their node captions directly adopt the consensus descriptions. The remaining objects are not classified in isolation: a 3D position-based retriever selects the nearest reliable object from the document and supplies its caption as environmental context. This is object-level retrieval within the current scene, not web search or text-similarity retrieval over generic knowledge passages.

For an uncertain node, the VLM receives a concatenation of its re-shot image and the original crop whose caption agrees most strongly with the re-shot. The accompanying text identifies the retrieved nearby object. The re-shot supplies overall object geometry, the real crop contributes color and texture, and the neighbor adds scene context. Together, these sources produce the revised node caption. The method section does not specify a stopping condition for repeatedly updating the retrieval document, so this note describes the explicit diagnosis-and-rectification pass rather than assuming an additional iterative convergence procedure.

4. Constrained Relationship Generation: reduce relational ambiguity after node rectification

Edge construction retains geometric proximity reasoning but replaces the fixed matching-distance threshold used in nearest-neighbor overlap calculations with a scale-dependent threshold consistent with dynamic sampling. An LLM receives the rectified object information for relationship generation, with few-shot in-context examples to stabilize spatial reasoning.

Although node semantics are open-vocabulary, final relationship labels are deliberately restricted to 8 categories: three bidirectional pairs for support, containment, and part-whole relations, plus proximity and none. This reduces synonym and spatial-expression variability, but means the final edges do not use an unrestricted open-vocabulary relation space. Reliable nodes improve the input to relationship reasoning, while the constrained taxonomy reduces output ambiguity; these are distinct contributions.

A Worked Example

Consider the vase occluding a table in Figure 1 as an illustration of the mechanism, without assigning invented similarity values. Some table crops from the original trajectory are described as a vase, while others correctly describe a table. After global fusion, the system selects a re-shot direction around the table's point cloud that reveals its tabletop structure.

If the re-shot caption identifies a table, semantically consistent original captions enter the highest-mean KMeans cluster and are aggregated with the re-shot description. Whether this node enters the reliable document depends on its relative uncertainty rank, not merely on a fixed absolute threshold.

For objects that remain in the uncertain half, the system retrieves the spatially nearest reliable neighbor and supplies its description alongside the object's paired images to the VLM. Final relationship generation then uses the revised object semantics rather than the initial labels confused by occlusion.

Loss & Training

The paper introduces no new learnable loss and reports no end-to-end fine-tuning. It uses SAM sam_vit_h_4b8939 and CLIP ViT-H-14, with either GPT-4o or locally deployed LLaVA-v1.5-7b for semantic reasoning. Results for these backends should be distinguished rather than attributing all framework gains to a stronger model.

Key settings are a matching threshold of 0.45, a base voxel parameter of 0.01 m, 64 candidate virtual views, 3 similarity clusters, and a 50% low-uncertainty document fraction. The Passive VLM control aggregates only the top-3 original crops; this control setting should not be mistaken for a specified top-k value in the main pipeline.

Key Experimental Results

Main Results

SceneFun3D contains 20 evaluated scenes and FunGraph3D contains 24, following the OpenFunGraph protocol with specialized functional interaction elements removed from the ground truth. Both objects and edges are evaluated with Recall@K, measuring whether the correct category or relationship is retrieved among the top K candidates. Open-vocabulary edges are mapped to ground-truth relationships using BERT embedding cosine similarity.

The following clearly recoverable columns are taken from Table 1, with values in %. The key Ours-GPT values are also confirmed in the prose of Section 4.2.

Method	SceneFun3D Object R@10	SceneFun3D Edge R@10	FunGraph3D Object R@3	FunGraph3D Object R@10
ConceptGraphs	77.1	95.0	56.6	65.6
OpenFunGraph	87.8	96.2	70.7	79.1
Ours-LLaVA	88.4	95.8	73.6	81.1
Ours-GPT	90.2	97.9	75.0	83.0

Relative to OpenFunGraph, Ours-GPT improves these columns by 2.4, 1.7, 4.3, and 3.9 percentage points. The LLaVA variant is not better on every metric: its SceneFun3D edge R@10 is 95.8%, below OpenFunGraph's 96.2%. Framework effectiveness does not imply that every backend leads on every metric.

Ablation Study

Figure 3(a) reports overall semantic point-cloud evaluation on Replica. Ground-truth masks are fused through gradSLAM, GPT-4o maps predicted descriptions to ground-truth semantic categories, and 1-NN point-cloud matching produces a confusion matrix. The metrics below are class-mean IoU, recall, precision, and F1, retaining the figure's numerical scale.

Config	mIoU	mRecall	mPrecision	mF1
Full method	23.60	37.37	34.87	30.78
Without image concatenation (w/o Concat)	21.41	33.61	30.04	28.09
Without RAG (w/o RAG)	21.55	32.36	31.40	27.96
Random retrieval (Random RAG)	18.90	31.59	26.19	25.39
Without re-shot diagnosis (w/o Reshot)	10.48	31.09	22.28	14.66

Removing re-shot diagnosis reduces mF1 by 16.12, the largest decrease in these overall results. Random retrieval is worse than no retrieval, showing that unreliable context can be harmful. Note that the overall values in Figure 3(a) cannot be reproduced by simply averaging the per-scene values in Figures 3(b)-(f), and the paper does not explain that aggregation difference here. This note preserves the authors' reported overall results without claiming that every individual scene benefits.

Table 2 is a separate human assessment of semantic precision and should not be conflated with the point-cloud metrics above. Three experts assess 8 Replica scenes, accepting correct synonymous descriptions but rejecting hallucinated attributes. Fleiss' kappa on 100 randomly selected evaluation units is 0.76.

Method	Mean Node Precision	Mean Edge Precision
ConceptGraphs	0.68	0.82
ConceptGraphs-Detector	0.58	0.85
RAG-3DSG	0.82	0.91

Key Findings

• Re-shooting alone is not sufficient: Table 1 reports SceneFun3D object R@3 of just 59.4% for Re-shot VLM, versus 83.0% for the full GPT variant. This control removes both RAG and image concatenation, so the gap cannot be attributed entirely to either component.
• Better nodes support better relationships, but edge design still contributes: Ours Node + CG Edge reaches 95.6% SceneFun3D edge R@10, compared with 97.9% for the full method.
• With Gaussian pose noise of 0.02 m translation standard deviation and 1 degree rotation standard deviation, GPT-variant SceneFun3D object R@3 decreases from 83.0% to 82.1%. This supports robustness under the tested noise, not arbitrary reconstruction failures.
• Section 4.6 reports a reduction in Replica fusion time from 6.65 s to 2.49 s per iteration, additional re-shot rendering of approximately 0.26 s/object, and GPT-4o API cost of approximately USD 0.50/scene on FunGraph3D. These are not complete end-to-end real-time latency measurements.

Highlights & Insights

• Re-shooting is more valuable as a verification signal than as the final answer. Real crops and point-cloud renderings have different failure modes, so agreement can filter semantic noise without assuming that a synthetic view is always correct.
• Retrieval-document quality determines whether RAG helps. Random retrieval underperforms no RAG, indicating that scene neighbors are useful evidence only when their semantics are sufficiently reliable; more context is not inherently better.
• Geometric sampling and semantic reasoning are interdependent. Preserving small-object structure affects both instance association and re-shot recognizability, suggesting that point-cloud compression should be evaluated against downstream semantic objectives.

Limitations & Future Work

• The system is currently offline. The robot experiment uses SCOUT MINI, RealSense D435, CH110 IMU, and dual Livox MID 360 sensors, with teleoperated collection followed by scene-graph generation. It is not an established real-time closed-loop navigation or manipulation system.
• Uncertainty is a proxy based on caption agreement, not a calibrated error probability. If original crops and re-shots fail together, or the entire scene is poorly observed, retaining a fixed top 50% can still admit incorrect anchors. This is a mechanism-level risk, and the paper does not separately quantify such correlated failures.
• Virtual re-shooting cannot recover geometry or texture that was never observed. View priors, real crops, and RAG mitigate missing information, but limited Gaussian pose-noise tests do not establish comparable robustness to major drift, incorrect instance merging, or dynamic scenes.
• Final relationships are restricted to 8 categories, and the real-scene evaluation removes functional interaction elements. The results therefore do not directly demonstrate improved arbitrary relation semantics or robotic task success.
• Some equations and table rows are corrupted in the text cache, and the connection between overall and per-scene aggregation in Figure 3 is unclear. Exact reproduction of sampling formulas and all results requires checking the typeset paper and supplementary material rather than guessing from this note.

Related Work & Insights

• vs ConceptGraphs: Both construct graphs from multi-view object observations. RAG-3DSG adds re-shot verification before caption aggregation and uses reliable within-scene neighbors to rectify nodes instead of merging all descriptions indiscriminately. The node replacement experiment shows that this also affects relationship prediction.
• vs OpenFunGraph: The paper adopts its evaluation protocol and real-scene data, but evaluates filtered object nodes and semantic edges. These comparisons cannot be extended to superiority on the full functional scene-graph task.
• vs BBQ / MomaGraph: Original-frame or keyframe selection is constrained by the acquisition trajectory; RAG-3DSG selects virtual new views over an existing reconstruction. It expands how the geometry is observed rather than adding new physical sensor measurements.
• Transferable insight: Object-level maps and language grounding can first test cross-modal evidence agreement before deciding which nodes may supply context. Absolute reliability thresholds and retrieval abstention merit investigation to avoid admitting incorrect anchors merely to satisfy a fixed fraction.

Rating

• Novelty: 4/5. Re-shot verification and reliable within-scene object retrieval form a concrete mechanism, although the components largely build on existing foundation models and geometry processing.
• Experimental Thoroughness: 4/5. Real scenes, human assessment, noise tests, and robotic deployment are covered, but task-level closed-loop gains and uncertainty calibration remain untested.
• Writing Quality: 3/5. The diagnose-then-rectify narrative is clear, but Figure 3 aggregation and some implementation details require clarification.
• Value: 4/5. The framework offers practical guidance for reducing semantic contamination in open-vocabulary 3D maps, subject to offline processing and foundation-model costs.