OpenGround: Planning-based Online Perception for Open-World 3D Visual Grounding¶
Conference: ECCV 2026
Paper: ECCV Official
Project Page: https://why-102.github.io/openground.io/
Area: 3D Vision
Keywords: 3D visual grounding, zero-shot scene understanding, task-chain planning, context-guided perception, open-world
TL;DR¶
Addressing the failure of existing zero-shot 3D visual grounding methods on undefined targets outside the predefined Object Lookup Table (OLT), OpenGround decomposes complex queries into an ordered task chain of sub-goals and leverages grounded context anchors to perceive novel objects online, achieving superior grounding performance on the newly created OpenTarget benchmark.
Background & Motivation¶
3D Visual Grounding (3DVG) aims to locate specific target objects within complex 3D physical spaces according to natural language descriptions, serving as a core perceptual foundation for vision-language navigation, embodied robotics, and spatial computing. Supervised 3D vision-language models (3D-VLMs) suffer from severe scarcity of paired 3D point-cloud and text annotations, and their limited context windows heavily restrict the number of input views, impairing generalizability in extensive real-world environments. Consequently, recent methods have widely transitioned to zero-shot paradigms built upon 2D foundation vision-language models (VLMs). These pipelines construct an offline Object Lookup Table (OLT) via pre-trained 3D instance segmentors, project candidate bounding boxes onto selected multi-view images, and confirm the target via single-step VLM reasoning.
However, this offline OLT single-step paradigm breaks down fundamentally when deployed in unconstrained open-world environments. Because offline OLTs are pre-computed using 3D segmentation models trained on fixed semantic vocabularies, they inevitably omit long-tailed, novel, or fine-grained part-level entities (e.g., specific cabinet handles, drawer knobs, or small appliance fixtures). Once a query refers to an out-of-OLT object, single-step candidate matching immediately fails. Existing remedies fall into two flawed extremes: either employing heavier segmentors to exhaustively densify the OLT beforehand—which incurs prohibitive pre-computation costs yet still misses fine-grained parts—or abandoning the OLT entirely, which requires repeatedly scanning the entire scene without spatial priors and falters when confronted with multi-step compositional queries.
The paper tackles this challenge by mirroring human cognitive spatial search strategies: when tasked with "locating the silver handle of the top drawer of the kitchen cabinet", humans never conduct an exhaustive search across millions of minute components in the whole house; instead, they first ground prominent coarse-grained containers (the cabinet), narrow down the candidate region to intermediate entities (the drawer), and finally identify the fine-grained part (the handle) within that local context. Core idea: formulate open-world 3D visual grounding as an adaptive task-chain planning and online perception process, where complex queries are decomposed into a context-to-target sub-goal sequence, and already grounded objects serve as geometric anchors to guide local 2D-to-3D online perception, dynamically extending the static OLT into an online OLT on demand.
Method¶
Overall Architecture¶
OpenGround retains the efficiency of OLT-based single-step grounding while augmenting it with two interconnected modules: Task-Chain Planning and Context-Guided Perception (CGP). Given a natural language query and a 3D scene point cloud, the framework first parses the target and relevant context objects, ordering them into an optimal execution sequence based on candidate availability, grounding difficulty, and semantic dependencies. During execution, OpenGround iterates through the planned sub-goals: if the current sub-goal already exists in the OLT, standard single-step candidate retrieval and VLM confirmation proceed directly; if the sub-goal is missing from the OLT, CGP is activated using previously grounded objects as spatial anchors to greedily select high-coverage observation views, detect novel objects via 2D open-world segmentation, lift and merge them into 3D bounding boxes, and dynamically insert them into the OLT.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Natural Language Query Q + 3D Scene"] --> B["Task-Chain Planning<br/>Dependency parsing & sub-goal ordering"]
B --> C{"Current sub-goal exists in OLT?"}
C -->|Candidate available| D["Single-Step Grounding<br/>Multi-view candidate projection & VLM reasoning"]
C -->|Missing / Out-of-OLT| E["Context-Guided Perspective Selection<br/>Greedy coverage maximization over context anchors"]
E --> F["2D-to-3D Online Perception<br/>Open-world segmentation, lifting & OLT insertion"]
F --> D
D --> G{"Final target reached in task chain?"}
G -->|Intermediate anchor confirmed| B
G -->|Target grounded| H["Output Target 3D Bounding Box"]
Key Designs¶
1. Task-Chain Planning: dependency-aware sub-goal sequencing
Direct single-step reasoning over long, compositional queries frequently fails due to candidate space explosion and spatial relation ambiguity. The Task-Chain Planning module utilizes a VLM to parse query \(Q\) into target label \(L_{\text{tgt}}\) and context label set \(\{L_{\text{ctx}}^j\}_{j=1}^n\), retrieving initial candidate set sizes \(|C_j|\) from the offline OLT. Constructing an effective execution sequence is guided by three fundamental factors: candidate availability (\(|C_j|=0\) denotes out-of-OLT entities that must be scheduled after their dependent context anchors), grounding difficulty (larger candidate pools represent higher ambiguity and must be grounded later after prior anchors prune the search space), and semantic dependency (possessive and spatial relations such as "handle of the drawer" dictate parent-to-child topological progression). A structured prompt feeds these candidate counts and missing indicators into the VLM to generate the sequence: $\(\text{taskchain} = f_{\text{vlm}}\left(Q, L, \{|C_j| \mid j \in [1, n+1]\}\right)\)$ A flexible soft constraint encourages grounding the target object at the very end of the chain, while permitting early target grounding whenever the target exhibits salient, unique visual attributes or an exceptionally small candidate pool.
2. Context-Guided Perspective Selection: greedy coverage maximization over anchors
When discovering missing objects, scanning the full scene produces redundant viewpoints and detail loss, while bird's-eye views suffer from vertical occlusions. OpenGround leverages the insight that previously grounded context objects \([O_1, \dots, O_{t-1}]\) serve as spatial anchors for task-relevant regions. CGP formulates perspective selection as a greedy coverage optimization over the 3D point sets of grounded context objects. For each candidate view \(k\), the marginal coverage gain over all context objects is computed as: $\(\Delta^{(k)} = \sum_{O_i} \frac{|P_{O_i} \cap P_k \setminus \mathcal{R}_{O_i}^{v-1}|}{|P_{O_i}|}\)$ where \(P_{O_i}\) is the 3D point set of object \(O_i\), \(P_k\) denotes visible points from view \(k\), and \(\mathcal{R}_{O_i}^{v-1}\) represents the accumulated observed region after \(v-1\) selected views. The algorithm iteratively selects \(k^* = \arg\max_k \Delta^{(k)}\) until coverage reaches threshold \(\tau_{\text{cov}} = 0.95\) or the view budget \(V = 3\) is met, guaranteeing that rendered perspectives capture the precise local environment surrounding the anchor.
3. 2D-to-3D Online Perception: open-world segmentation and dynamic OLT extension
Given the selected context-guided views \(I^*\), the framework employs 2D open-world foundation perception models to overcome the closed vocabulary limits of 3D segmentors. For each view image \(I_k\), GroundedSAM generates 2D instance masks conditioned on sub-goal text label \(L_{T_t}\): \(M_k = \text{SEG}(I_k, L_{T_t})\). Using camera calibration matrices and depth point clouds, 2D masks are unprojected into 3D space. Because multi-view 2D predictions may produce fragmented 3D point clusters for the same physical entity, 3D proposals with spatial 3D IoU \(\ge \tau_{\text{iou}}\) (set to 0.5) are iteratively merged. Each consolidated cluster is converted into a 3D bounding box and inserted on-the-fly into the lookup table: $\(\text{OLT} \leftarrow \text{OLT} \cup \Big\{\big(\text{ID}(i),\, L_{T_t},\, \text{bbox}(P_i)\big)\Big\}_{i=1}^M\)$ This converts the static offline OLT into a dynamic online OLT, enabling downstream single-step VLM modules to seamlessly verify and retrieve novel 3D object proposals.
A Worked Example¶
Consider the query "Find the silver handle of the top drawer of the kitchen cabinet":
1. Planning: The VLM extracts labels: cabinet, drawer, handle. Candidate lookup in the initial OLT returns 2 cabinets, 6 drawers, and 0 handles (\(|C_{\text{handle}}|=0\)). The resulting task chain is [cabinet -> drawer -> handle].
2. Step 1 (Cabinet): Candidates exist in the OLT; standard single-step grounding identifies the target cabinet among the 2 candidates.
3. Step 2 (Drawer): The candidate set is pruned by spatial containment within the confirmed cabinet, narrowing 6 candidates down to 2, and single-step reasoning identifies the top drawer.
4. Step 3 (Handle): The handle is out-of-OLT. CGP selects 3 views that greedily maximize visual coverage of the top drawer point cloud. GroundedSAM detects the 2D handle mask, which is lifted into 3D, bound by a 3D bounding box, and assigned a novel ID in the OLT.
5. Confirmation: Downstream single-step grounding verifies the newly registered handle in the updated OLT and outputs its 3D bounding box.
Key Experimental Results¶
Main Results¶
Evaluation is conducted on the newly established open-world benchmark OpenTarget (7,724 query-object pairs spanning 50 parent and 70 part categories, derived from ScanNet++ and Articulate3D), as well as standard benchmarks ScanRefer and Nr3D.
Performance comparison on OpenTarget (Easy includes hierarchy length \(\le 2\), Hard includes hierarchy length \(>2\)):
| Method | OLT Source | Easy [email protected] | Easy [email protected] | Hard [email protected] | Hard [email protected] | Overall [email protected] | Overall [email protected] |
|---|---|---|---|---|---|---|---|
| SeeGround | Mask3D | 13.3 | 11.2 | 1.2 | 1.1 | 10.2 | 8.6 |
| VLM-Grounder* | None (OLT-Free) | 14.5 | 12.1 | 6.1 | 2.3 | 12.3 | 9.6 |
| VLM-Grounder* | Mask3D | 16.4 | 12.0 | 10.2 | 4.8 | 14.8 | 10.1 |
| SeqVLM | Mask3D | 13.6 | 11.2 | 1.4 | 1.1 | 10.5 | 8.6 |
| GPT4Scene | Mask3D | 10.1 | 7.7 | 0.9 | 0.7 | 7.7 | 5.9 |
| ZSVG3D | Mask3D | 8.9 | 6.9 | 0.8 | 0.7 | 6.8 | 5.3 |
| OpenGround (Ours) | Mask3D | 51.8 | 38.9 | 30.2 | 20.6 | 46.2 | 34.2 |
| SeeGround | Ground-Truth | 20.2 | 19.8 | 11.3 | 10.4 | 17.9 | 17.4 |
| VLM-Grounder* | Ground-Truth | 31.4 | 21.3 | 20.6 | 17.8 | 28.6 | 20.4 |
| SeqVLM | Ground-Truth | 21.5 | 21.3 | 13.4 | 13.2 | 19.4 | 19.2 |
| OpenGround (Ours) | Ground-Truth | 57.9 | 57.4 | 45.7 | 45.3 | 54.8 | 54.3 |
*Note: * indicates evaluation on 300 randomly sampled instances due to computational cost.
Performance comparison on ScanRefer (Unique denotes single-target scenes; Multiple denotes scenes with distractors of the same category):
| Method | Supervision | Backbone VLM | Unique [email protected] | Unique [email protected] | Multiple [email protected] | Multiple [email protected] | Overall [email protected] | Overall [email protected] |
|---|---|---|---|---|---|---|---|---|
| 3D-R1 | Supervised | - | - | - | - | - | 65.8 | 59.2 |
| GPT4Scene | Supervised | - | 90.3 | 83.7 | 56.4 | 50.9 | 62.6 | 57.0 |
| SeeGround | Zero-Shot | Qwen2-VL-72B | 75.7 | 68.9 | 34.0 | 30.0 | 44.1 | 39.4 |
| Ours | Zero-Shot | Qwen2-VL-72B | 76.6 | 70.3 | 48.1 | 40.4 | 53.7 | 46.3 |
| SeqVLM | Zero-Shot | Doubao-1.5-pro | 77.3 | 72.7 | 47.8 | 41.3 | 55.6 | 49.6 |
| Ours | Zero-Shot | Doubao-1.5-pro | 78.3 | 75.1 | 59.2 | 49.4 | 63.0 | 54.4 |
| SPAZER | Zero-Shot | GPT-4o | 80.9 | 72.3 | 51.7 | 43.4 | 57.2 | 48.8 |
| Ours | Zero-Shot | GPT-4o | 83.5 | 75.2 | 59.9 | 51.2 | 64.5 | 55.9 |
Ablation Study¶
Component ablations on OpenTarget measuring [email protected] across initial OLT status, planning strategies, and VLM backbones:
| # | Initial OLT | Grounding Strategy | VLM Backbone | [email protected] (%) | Analysis & Note |
|---|---|---|---|---|---|
| (1) | Yes (Mask3D) | Task Chain (Ours) | GLM-4.5V | 34.2 | Full framework configuration |
| (2) | No (Empty OLT) | Task Chain | GLM-4.5V | 27.1 | Outperforms baselines equipped with GT OLT (20.4%) |
| (3) | Yes | Jump (Skip to target on 1st anchor) | GLM-4.5V | 29.8 | Bypassing intermediate containers causes -4.4% drop |
| (4) | Yes | Relevance (Order by text relevance) | GLM-4.5V | 32.6 | Ignores physical containment topology, minor drop |
| (5) | Yes | Difficulty (Order by candidate count) | GLM-4.5V | 31.5 | May trigger missing targets prematurely |
| (6) | Yes | Random (Random permutation) | GLM-4.5V | 29.2 | Lacks topological ordering; error cascades |
| (7) | Yes | Task Chain | Qwen3-VL-32B | 30.4 | Smaller open VLM drops only 3.8% |
| (8) | Yes | Task Chain | Qwen3-VL-235B | 32.8 | Approaches frontier proprietary model accuracy |
| (9) | Yes | Task Chain | Step3 | 33.4 | Demonstrates strong cross-VLM generalizability |
Key Findings¶
- Overcoming the predefined OLT bottleneck: Baseline zero-shot methods collapse on OpenTarget with Mask3D OLTs (5.3%~10.5% overall [email protected]) because fine-grained parts are absent from the lookup table. Even with Ground-Truth OLTs, they only achieve 11.0%~20.4% due to extreme distractor ambiguity. OpenGround achieves 34.2% with Mask3D and 54.3% with GT OLT, demonstrating a qualitative breakthrough in open-world grounding.
- Pruning distractor ambiguity via progressive chains: In the ScanRefer Multiple split where distractors abound, OpenGround with GPT-4o improves [email protected] from SPAZER's 43.4% to 51.2% (an 18% relative gain), confirming that sequential context narrowing purifies candidate spaces.
- Observation budget saturation: View count ablation shows that increasing \(V\) from 1 to 3 yields steep accuracy gains. While \(V=4\) marginally peaks at 34.7% (+0.52%), it requires 33.3% more view inputs, and \(V > 4\) degrades performance due to visual redundancy and confusion; \(V=3\) represents the optimal Pareto trade-off.
Highlights & Insights¶
- From static tables to dynamic online memory: Rather than abandoning structured lookup tables, OpenGround equips OLTs with on-demand online expansion, preserving retrieval efficiency while expanding vocabulary coverage.
- Anchor-guided greedy viewpoint selection: Transforming viewpoint selection from global heuristic sampling into an anchor point cloud coverage optimization ensures high-resolution inspection of relevant local spatial volumes.
- Pluggable architecture: Task-Chain Planning and CGP integrate seamlessly into diverse single-step backbones (SeeGround, SeqVLM, SPAZER), boosting their accuracy across closed and open vocabularies.
Limitations & Future Work¶
- Spatial proximity assumption: CGP assumes spatial proximity between context anchors and target objects; long-range non-local relationships (e.g., "the chair that matches the one in the bedroom") remain challenging.
- Dependence on 2D segmentor fidelity: The precision of generated 3D bounding boxes is bounded by the open-world segmentation quality and boundary delineation of 2D foundation models.
- Dynamic scene and embodied exploration: Current formulation focuses on static reconstructed point clouds; extending online OLT updates to streaming RGB-D sequences in mobile robotic exploration is a natural next step.
Related Work & Insights¶
- vs SeeGround / SeqVLM: Prior zero-shot methods assume targets exist within offline OLTs; OpenGround introduces CGP to discover novel objects online, closing the gap in open-world part-level grounding.
- vs VLM-Grounder: VLM-Grounder discards OLTs entirely and relies on exhaustive, repeated full-scene scans; OpenGround preserves structured geometric caching and uses targeted local perception, achieving superior accuracy and efficiency.
- vs SPAZER: SPAZER enhances spatial-semantic reasoning for single-step matching; OpenGround serves as a higher-level planning and perception framework that can integrate SPAZER as a backbone, reaching 64.8% on Nr3D.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates a principled planning and online perception framework that lifts zero-shot 3DVG into open-world scenarios.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Establishes the 7,724-sample OpenTarget benchmark and validates across three datasets with extensive ablations.
- Writing Quality: ⭐⭐⭐⭐⭐ Crisp narrative structure, clear technical formulation, and well-aligned figures and experiments.
- Value: ⭐⭐⭐⭐⭐ Highly practical, modular design that directly benefits embodied AI, robotics, and 3D spatial computing.