Beyond Isolated Objects: Relationship-aware Open Vocabulary Scene Understanding via 3D Scene Graph Analysis¶
Conference: ECCV 2026
Paper: ECCV 2026
Code: https://cxavireh.github.io/relgraphov-projectpage
Area: 3D Vision
Keywords: 3D open-vocabulary scene understanding, 3D scene graph, semantic segmentation, dual-stream GAT, hierarchical contrastive learning
TL;DR¶
Addressing the issue that existing 3D open-vocabulary methods treat objects as isolated entities and struggle with ambiguous appearances, RelGraphOV extracts relationship-aware 3D scene graphs via multi-view VLM reasoning and applies an Adaptive Gated Dual-Stream Contextual GAT with hierarchical contrastive learning to decouple geometric and semantic propagation, substantially outperforming prior arts in fine-grained and long-tail 3D segmentation.
Background & Motivation¶
Open-vocabulary 3D scene understanding aims to segment 3D environments with arbitrary natural language descriptions, breaking through the limits of fixed, closed category vocabularies to support spatial reasoning, robotics navigation, and augmented reality. Recent paradigms either lift dense language-aligned 2D features (such as LSeg or OpenSeg) into 3D points, providing geometric robustness at the cost of limited semantic generalization on tail concepts; or associate instance-level global CLIP embeddings with 3D object proposals, offering broader open-vocabulary vocabulary coverage but becoming vulnerable to partial observations and textureless geometries.
The fundamental limitation underlying both paradigms is that they process objects in the scene as isolated, context-independent entities, neglecting the structured spatial and semantic dependencies between interacting objects. In practical 3D environments, distinguishing objects with similar local appearances—such as disambiguating a generic "curtain" from a "shower curtain"—is nearly impossible when viewing the object in isolation. However, if contextual co-occurrence with neighboring entities (e.g., proximity to a bathtub or toilet) is explicitly integrated, such ambiguities vanish naturally. Leveraging 3D scene graphs to introduce relational context has long been hindered by two roadblocks: the lack of large-scale relationship ground-truth annotations in open-vocabulary settings, and the severe "feature interference" that arises when heterogeneous dense geometric features and high-level instance CLIP embeddings are naively mixed in graph neural message passing, which contaminates the semantic feature space with geometric noise.
This paper tackles these challenges by utilizing multi-view vision-language chain-of-thought reasoning to automate relationship inference without human annotations, while designing a decoupled dual-stream graph architecture to aggregate relational context safely. Core idea: construct relationship-aware 3D scene graphs via multi-view VLM reasoning and geometric pruning, and employ an Adaptive Gated Dual-Stream Contextual GAT with hierarchical contrastive learning to decouple dense geometric features from CLIP semantic embeddings, achieving noise-free contextual refinement for open-vocabulary 3D scene understanding.
Method¶
Overall Architecture¶
The RelGraphOV pipeline operates across four coordinated stages: class-agnostic 3D object proposals are first clustered from RGB-D frames, followed by a composite viewpoint scoring mechanism that selects optimal keyframes for multi-view VLM chain-of-thought reasoning to filter geometric candidate edges and build a relationship-aware 3D scene graph; an automated multi-view adjudication engine then generates consistent, fine-grained textual annotations and soft labels for each node; an Adaptive Gated Dual-Stream Contextual GAT propagates information along the graph, where a main stream refines dense LSeg features and an auxiliary stream preserves pure CLIP semantics, augmented by global scene-level context injected via a virtual super node; finally, the network is trained using a composite loss combining hierarchical contrastive objectives and dual-stream alignment regularization.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: Point Clouds & RGB-D Sequences<br/>Class-agnostic 3D Proposals"] --> B["Multi-View Reasoning Scene Graph Construction<br/>Viewpoint Scoring + VLM-CoT SVO Generation"]
B --> C["Multi-View Adjudication Node Annotation Engine<br/>SAM Masks + DAM Captions + Dual-View Arbiter"]
C --> D["Virtual Super Node Global Context Injection<br/>Cross-Attention Bridging Local Nodes & Scene Context"]
D --> E["Edge-Guided Propagation & Adaptive Gated Fusion<br/>Decoupled Streams with Unilateral Semantic Injection"]
E --> F["Hierarchical Contrastive & Dual Alignment Training<br/>Self-Consistency + Inter-Class Margin + Anti-Forgetting"]
F --> G["Output: Dense Open-Vocabulary 3D Segmentation"]
Key Designs¶
1. Multi-View Reasoning Scene Graph Construction: Labor-free Structural Prior Extraction To circumvent the scarcity of open-vocabulary 3D relationship annotations, an automated construction pipeline is designed. Given class-agnostic 3D proposals, representative keyframes are extracted via a composite viewpoint scoring function that evaluates node completeness, projected area, center alignment, and motion blur sharpness. Geometric candidate edges formed by nearest neighbors and bounding box overlaps are visually verified by feeding joint viewpoints into a vision-language model (Qwen-VL-Max) guided by Set-of-Mark visual prompts and chain-of-thought reasoning. Candidate connections returning "none" are pruned, while valid relations are parsed into Subject-Verb-Object (SVO) triplets. The subject, verb, and object are independently encoded by the CLIP text encoder and concatenated to form edge feature vectors \(f_{e_{ij}}\), yielding the refined graph \(\mathbf{SG} = (\mathbf{E}_{refine}, \mathbf{F}_v, \mathbf{F}_e)\).
2. Multi-View Adjudication Node Annotation Engine: Resolving Cross-View Inconsistencies Because observing the same 3D instance from varying camera poses can produce conflicting 2D descriptions, an automated multi-view adjudication engine is introduced. For each node, two distinct keyframes are selected—one optimizing visibility and the other emphasizing sharpness and spatial dissimilarity. Segment Anything Model (SAM) and the Describe Anything Model generate initial view-specific captions, which are subsequently examined by a lightweight language model (Qwen-Turbo) to verify base category consistency. In case of disagreement, an adjudication prompt forces the model to synthesize visual evidence from both views to determine the ground truth, resulting in a refined description encoded by the CLIP text encoder into soft target feature \(f_{\text{anno}}^i\).
3. Adaptive Gated Dual-Stream Contextual GAT: Eliminating Feature Interference To prevent geometric noise from corrupting high-level semantic spaces during graph propagation, the network decouples feature updates into a main stream (dense LSeg features \(f_l\) fused with Fourier-encoded coordinates) and an auxiliary stream (instance CLIP features \(f_c\)). A virtual super node connects to all real nodes to extract a global context vector \(G_{\text{global}}\), which is injected into the main stream via Cross-Attention. During neighborhood message passing, edge features \(f_{e_{ij}}\) are concatenated into the attention Keys to adaptively modulate interaction weights based on relationship semantics: $\(\text{Attention Key}_{ij} = [\tilde{h}_{m,j}^{(l)} \parallel f_{e_{ij}}]\)$ Following independent message aggregation, the auxiliary stream preserves its pure CLIP embeddings for future layers, while a learnable gating mechanism allows the main stream to unilaterally absorb generalizable semantics from the auxiliary stream: $\(\text{Gate} = \sigma(\text{MLP}([\hat{h}_{m,i}^{(l)} \parallel \hat{h}_{a,i}^{(l)}])), \quad h_{m,i}^{(l+1)} = \hat{h}_{m,i}^{(l)} + \alpha \cdot \text{Gate} \odot \hat{h}_{a,i}^{(l)}\)$ This unilateral gate prevents geometric noise from feeding back into and corrupting the semantic prior.
4. Hierarchical Contrastive and Dual-Alignment Learning Strategy: Mitigating Semantic Drift To train the network on open-vocabulary annotations without inappropriately penalizing semantically related categories, a hierarchical contrastive loss \(\mathcal{L}_{\text{hie}}\) is introduced. It builds an inferred positive mask \(M_{\text{pos}}^{ij}\) for pairs whose annotation cosine similarity exceeds threshold \(\tau\), while applying an additional margin \(\mathcal{M}\) to self-pairs. Furthermore, to guard against catastrophic forgetting and preserve modality-specific priors, symmetric dual-alignment losses \(\mathcal{L}_{\text{align}}\) are applied to both streams: $\(\mathcal{L}_{\text{align}}(F, f_{\text{init}}, \gamma) = \frac{1}{N}\sum_{i=1}^N \left( \gamma F^{v_i} \cdot f_{\text{anno}}^i + (1 - \gamma) F^{v_i} \cdot f_{\text{init}}^{v_i} \right)\)$ Here, \(\mathcal{L}_{\text{reg}} = \mathcal{L}_{\text{align}}(F_{\text{out}}, f_l, \gamma_{\text{reg}})\) anchors the main stream to the dense LSeg prior, while \(\mathcal{L}_{\text{aux}} = \mathcal{L}_{\text{align}}(F_{\text{aux}}, f_c, \gamma_{\text{aux}})\) anchors the auxiliary stream to the original CLIP space.
Loss & Training¶
The overall training objective is a composite weighted loss: $\(\mathcal{L}_{\text{total}} = \lambda_{\text{hie}} \cdot \mathcal{L}_{\text{hie}} + \lambda_{\text{reg}} \cdot \mathcal{L}_{\text{reg}} + \lambda_{\text{aux}} \cdot \mathcal{L}_{\text{aux}}\)$ Hyperparameters are set to \(\lambda_{\text{hie}}=0.5\), \(\lambda_{\text{reg}}=1.5\), \(\lambda_{\text{aux}}=1.5\), and \(\gamma_{\text{reg}}=\gamma_{\text{aux}}=0.7\). The model is trained on a single NVIDIA RTX 4090 GPU for 600 epochs using the Adam optimizer with an initial learning rate of \(1\times 10^{-3}\) and batch size of 256.
Key Experimental Results¶
Main Results¶
RelGraphOV is evaluated across four benchmarks: standard ScanNetV2, fine-grained ScanNet200, and cross-dataset zero-shot benchmarks ScanNet++ and Replica.
| Dataset | Metric | RelGraphOV (Ours) | Prev. SOTA (CUA-O3D / Mosaic3D) | Gain |
|---|---|---|---|---|
| ScanNetV2 (20-class) | mIoU (%) | 58.4 | 55.3 (CUA-O3D) | +3.1 |
| ScanNetV2 (20-class) | mAcc (%) | 73.4 | 65.6 (CUA-O3D) | +7.8 |
| ScanNet200 (200-class) | mIoU (%) | 14.5 | 12.4 (Mosaic3D) | +2.1 |
| ScanNet200 (200-class) | mAcc (%) | 26.1 | 25.1 (Mosaic3D) | +1.0 |
| ScanNet++ (Zero-shot 100-class) | mIoU (%) | 20.9 | 13.3 (OpenScene-2D) | +7.6 |
| ScanNet++ (Zero-shot 100-class) | mAcc (%) | 33.2 | 20.0 (OpenScene-2D) | +13.2 |
| Replica (Zero-shot 51-class) | mIoU (%) | 22.7 | 12.9 (OpenScene-2D) | +9.8 |
Per-category inspection on ScanNetV2 reveals that while baseline methods (OpenScene and CUA-O3D) achieve 0.0% IoU on "shower curtain" due to appearance confusion with general "curtains", RelGraphOV achieves 64.2% IoU by leveraging relational cues with bathtubs and toilets. On the challenging long-tail ScanNet200 benchmark, our method achieves 6.3% mIoU on Tail categories compared to 0.0% for CUA-O3D and 1.3% for Mosaic3D.
Ablation Study¶
| Config | ScanNetV2 mIoU | ScanNetV2 mAcc | ScanNet200 mIoU | ScanNet200 mAcc | Note |
|---|---|---|---|---|---|
| Ours (full model) | 58.4 | 73.4 | 14.5 | 26.1 | full model |
| w/o scene graph | 50.7 | 62.6 | 9.5 | 14.4 | isolated LSeg mask pooling without graph edges |
| w/o relation guidance | 55.5 | 70.7 | 11.0 | 20.1 | vanilla GAT without SVO edge features |
| w/o global feature | 56.0 | 71.3 | 11.1 | 20.2 | w/o virtual super node cross-attention |
| w/o dual graph fusion | 57.9 | 73.0 | 12.7 | 22.9 | naive concatenation of LSeg and CLIP in single GAT |
| w/o \(\mathcal{L}_{\text{hie}}\) | 52.2 | 65.5 | 7.8 | 12.4 | w/o hierarchical contrastive loss |
| w/o \(\mathcal{L}_{\text{reg}}\) | 54.9 | 71.7 | 10.9 | 20.1 | w/o main stream geometric regularization |
| w/o \(\mathcal{L}_{\text{aux}}\) | 55.9 | 70.4 | 13.4 | 25.6 | w/o auxiliary stream CLIP prior alignment |
Key Findings¶
- Contextual scene graph connectivity is the primary driver of performance gains: removing all graph edges (
w/o scene graph) causes a dramatic drop of 5.0% mIoU on ScanNet200, proving that isolated appearance modeling cannot resolve complex scene ambiguities. - Decoupled dual-stream propagation is critical to suppress feature interference: naive single-stream feature concatenation (
w/o dual graph fusion) degrades ScanNet200 mIoU by 1.8% and mAcc by 3.2%, validating that unilateral gated injection prevents cross-modal contamination. - Hierarchical contrastive learning is essential for long-tail discrimination: eliminating \(\mathcal{L}_{\text{hie}}\) causes ScanNet200 mIoU to plunge from 14.5% to 7.8% (-6.7%), demonstrating that positive-pair soft-masking based on text similarity prevents over-penalizing semantically adjacent concepts.
Highlights & Insights¶
- Annotation-free Relational Prior Extraction: Leveraging multi-view VLM chain-of-thought reasoning to extract SVO triplets transforms uncurated RGB-D scans into structured relational priors without manual labeling.
- Asymmetric Gated Dual-Stream Architecture: Decoupling dense geometry and sparse semantics while allowing unilateral semantic injection provides an effective blueprint for resolving multi-modal feature interference in graph learning.
- Superior Cross-Dataset Zero-Shot Transfer: Training on ScanNetV2 and testing on ScanNet++ and Replica achieves massive zero-shot gains (+7.6% and +9.8% mIoU over OpenScene), showing that relational object contexts are invariant across domain shifts.
Limitations & Future Work¶
- Frozen 2D Foundation Model Backbones: Relying on fixed LSeg and CLIP backbones constrains initial node representations to their pre-trained capacity; exploring joint end-to-end training with advanced dense VLMs is a promising next step.
- Assumption of Static Scenes: The graph generation pipeline assumes static reconstructed scenes; extending RelGraphOV to dynamic environments with moving agents and evolving relationships will require temporal graph consistency mechanisms.
Related Work & Insights¶
- vs OpenScene / CUA-O3D: Existing 3D open-vocabulary approaches rely on isolated multi-view back-projection or mask pooling, leading to ambiguities among co-occurring categories; RelGraphOV incorporates 3D scene graphs as contextual priors, raising the IoU of confusing categories like shower curtain from 0.0% to 64.2%.
- vs ConceptGraphs / Open3DSG: Previous relational 3D studies focus on scene graph prediction or open-set graph querying; RelGraphOV is the first to channel inferred scene graphs as intermediate structural priors directly into dense open-vocabulary 3D semantic segmentation.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ First to channel automated VLM-reasoned 3D scene graphs into dense open-vocabulary segmentation with an elegant dual-stream gating design.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorously tested across ScanNetV2, ScanNet200, ScanNet++, and Replica with comprehensive ablation studies.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear problem statement, cohesive method narrative, and disciplined mathematical formulations.
- Value: ⭐⭐⭐⭐☆ Offers a robust methodology for transitioning 3D open-vocabulary scene perception from isolated object recognition to relationship-aware cognitive reasoning.