Group3D: MLLM-Guided Semantic Grouping for Open-Vocabulary 3D Object Detection¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://ubin108.github.io/Group3D/
Area: Multimodal VLM / 3D Vision
Keywords: Open-Vocabulary 3D Object Detection, Multi-View RGB, Multimodal LLM, Semantic Compatibility Grouping, Gated Instance Construction
TL;DR¶
Group3D addresses the irreversible over-merging issue in multi-view RGB open-vocabulary 3D detection by utilizing an MLLM to construct a scene-adaptive vocabulary and semantic compatibility groups, introducing semantically gated merging alongside asymmetric voxel overlap/containment verification to achieve state-of-the-art results without ground-truth point clouds or 3D supervision.
Background & Motivation¶
Open-vocabulary 3D object detection aims to localize and recognize object instances in 3D space beyond a fixed training taxonomy. Traditional 3D detectors built upon LiDAR or dense point clouds depend heavily on expensive, labor-intensive 3D bounding box annotations. While recent efforts leverage 2D vision-language models (e.g., CLIP) to transfer open-vocabulary signals via pseudo-supervision, they typically assume access to explicit, high-quality 3D geometry during inference. In contrast, multi-view image-based 3D detection relies solely on readily accessible and inexpensive RGB observations across viewpoints, offering a significantly more scalable and generalizable paradigm for real-world physical perception.
However, existing multi-view open-vocabulary 3D detection frameworks (such as Zoo3D and OpenM3D) predominantly adopt a decoupled "geometry-first" pipeline: 2D predictions are back-projected and aggregated into class-agnostic 3D proposals based solely on spatial proximity, followed by post hoc open-vocabulary semantic labeling. In multi-view RGB setups, reconstructed geometry is inherently incomplete, noisy, and view-dependent. When merge decisions are governed purely by geometric consistency, spatial ambiguities easily cause adjacent but semantically distinct objects (e.g., a table and a chair, or a door and a wall) to be irreversibly fused into a single oversized instance. Once geometric boundaries collapse during instance construction, subsequent vision-language semantic alignment can no longer disentangle them.
Conversely, enforcing strict identical-category constraints during fragment association is severely crippled by cross-view taxonomy noise—where the same physical entity is predicted under slightly different yet semantically related names across varying viewpoints, scales, and lighting conditions (e.g., labeled as "sofa" in one view and "chair" in another, or alternating between "desk" and "table"). This paper tackles this dilemma by directly integrating language-driven semantic commonsense into the instance construction phase as an explicit merge-time gating mechanism. Core idea: leverage an MLLM to aggregate a scene-adaptive vocabulary and partition it into semantic compatibility groups capturing plausible cross-view category equivalence, gating 3D fragment association via semantic compatibility and asymmetric voxel overlap/containment to prevent cross-category over-merging while absorbing taxonomy noise.
Method¶
Overall Architecture¶
Given a collection of multi-view RGB images of a scene along with optional camera poses, Group3D predicts a set of 3D object instances with open-vocabulary category names, confidence scores, and 3D bounding boxes. The framework consists of four primary stages across two main phases: first, constructing a Scene Vocabulary Memory via MLLM queries across viewpoints and building a 3D Fragment Memory by lifting category-aware 2D masks (generated by SAM 3) via estimated or known multi-view geometry; second, querying the MLLM to partition the vocabulary into semantic compatibility groups; third, executing group-gated 3D fragment merging where fragments are associated only if they satisfy both group-level semantic compatibility and voxel-level geometric consistency (combining IoU and containment ratio); and finally, accumulating multi-view confidence evidence to determine the final category and 3D bounding box for each instance.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multi-View RGB Images Input"] --> B["Scene Memory Construction<br/>Scene Vocabulary & 3D Fragment Memories"]
B --> C["Semantic Compatibility Grouping<br/>MLLM-Induced Cross-View Equivalence Groups"]
C --> D["Group-Gated 3D Fragment Merging<br/>Semantic Gating with Voxel Overlap/Containment"]
D --> E["Multi-View Evidence Accumulation<br/>Confidence-Weighted Support Voting & 3D Bounding Boxes"]
E --> F["Final Open-Vocabulary 3D Object Instances"]
Key Designs¶
1. Scene Memory Construction: Dual Decoupled Vocabulary and Geometric Fragment Memories
To support flexible open-vocabulary instance synthesis, the framework first initializes two complementary scene-level memory structures. On the semantic level, the MLLM is queried across input views to extract a set of object category hypotheses (\(K=5\) per view), which are normalized through morphological standardization and casing canonicalization (e.g., Trash_can \(\to\) trash can) and de-duplicated into a compact scene-level vocabulary \(\mathcal{V} = \bigcup_n \mathcal{V}_n\). On the geometric level, a foundation segmentation model (SAM 3) takes each category \(\ell_i \in \mathcal{V}\) as text prompts to generate category-aware 2D binary masks \(m_{n,i}\) on image \(I_n\), assigned a composite confidence score:
$\(s_{n,i} = s^{\text{query}}_{n,i} \cdot s^{\text{pres}}_n\)$
balancing the image-level presence likelihood with region-level prompt alignment. In the pose-free regime, a reconstruction backbone (e.g., Depth Anything 3 or VGGT) estimates camera poses \(T_n\) and dense depth maps \(D_n\). Pixel coordinates within mask \(m_{n,i}\) are back-projected into world coordinates to form 3D point cloud fragments \(F_{n,i}\). After depth outlier filtering, each fragment is stored as an atomic tuple \((F_{n,i}, \ell_i, s_{n,i})\) in the 3D Fragment Memory \(\mathcal{F}\).
2. Semantic Compatibility Grouping: Encoding Language Priors for Cross-View Category Equivalence Due to open-vocabulary taxonomy noise across viewpoints, the same physical instance often receives different but semantically related labels. Enforcing a strict same-name constraint fragments instances, whereas geometry-only merging collapses distinct objects. Group3D prompts the MLLM to partition the scene vocabulary \(\mathcal{V}\) into semantic compatibility groups \(\mathcal{G} = \{G_g\}_{g=1}^G\). The MLLM groups categories that plausibly refer to the same physical object under taxonomy variability (e.g., chair–sofa, desk–table), while explicitly barring structural attachments (e.g., wall–window, wall–door), supporting structural relations (e.g., floor–wall), and part-whole pairs (e.g., table–cup). The resulting mapping \(g(\ell)\) acts as a semantic firewall capturing substitutability rather than spatial adjacency.
3. Group-Gated 3D Fragment Merging: Dual Semantic Gating and Asymmetric Voxel Containment Instance generation follows a greedy clustering approach initialized by sorting fragments by spatial extent in descending order to anchor clusters around dominant physical structures. When assessing whether a candidate fragment \((F_{n,i}, \ell_i, s_{n,i})\) can merge into an existing instance cluster \((C_F, C_\ell)\), two strict criteria must be satisfied simultaneously: first, the semantic gating requirement mandates that \(g(\ell_i) = g(C_\ell)\), ensuring that fragments from incompatible semantic groups can never fuse; second, geometric consistency is evaluated in a 5cm voxelized grid. Standard voxel IoU severely penalizes cases where a small fragment is heavily or entirely contained within a larger fragment due to the disproportionate union volume. Group3D integrates an asymmetric containment metric \(\text{Cont}_{\texttt{vox}}(B \to A) = \frac{|\texttt{vox}(A) \cap \texttt{vox}(B)|}{|\texttt{vox}(B)|}\) into a unified boolean predicate: $\(\text{Overlap}(A, B) = \big(\text{IoU}_{\texttt{vox}}(A, B) \ge \tau_{\text{iou}}\big) \;\lor\; \big(\text{Cont}_{\texttt{vox}}(B \to A) \ge \tau_{\text{cont}}\big)\)$ Only fragments satisfying both semantic group equivalence and geometric overlap/containment are merged into \(C_F\), updating the instance's associated label set \(C_\ell\).
4. Multi-View Evidence Accumulation: Saturation-Weighted Confidence Voting After group-gated clustering, each 3D instance retains a unified point cloud \(C_F\) and a set of candidate labels \(C_\ell\). To resolve the final open-vocabulary label from the compatibility group, the instance-level score for each associated category \(\ell \in C_\ell\) is computed by combining its mean fragment confidence \(\bar{s}(\ell)\) with a non-linear observation frequency saturation function: $\(s(\ell) = \bar{s}(\ell) \cdot \Big(1 - \exp\big(-\frac{N(\ell)}{\tau}\big)\Big)\)$ where \(N(\ell)\) is the count of merged fragments associated with label \(\ell\). This concave exponential weighting rewards stable cross-view multi-frame corroboration while preventing spurious background categories from dominating purely due to high fragment counts. The category achieving \(\arg\max_{\ell \in C_\ell} s(\ell)\) is assigned as the final label, and the 3D bounding box is extracted from the axis-aligned spatial extents of \(C_F\).
A Worked Example¶
Consider a conference room containing a large conference table and adjacent office chairs:
1. Memory Construction: In view 1, the MLLM detects the table corner and proposes desk; in view 2, a wide shot captures the same object and proposes table. SAM 3 creates masks for both, back-projecting them into 3D fragments \(F_{1,\text{desk}}\) and \(F_{2,\text{table}}\). An adjacent office chair in view 1 is segmented under chair, producing fragment \(F_{1,\text{chair}}\).
2. Compatibility Grouping: The MLLM groups desk and table into compatibility group \(G_1\) (work surfaces), while isolating chair in group \(G_2\) (seating furniture).
3. Gated Merging: Reconstructed points of \(F_{1,\text{chair}}\) lie directly against the table legs in 3D space. However, because \(g(\text{chair}) = G_2 \ne G_1\), semantic gating blocks the merge, preventing catastrophic over-merging. Next, when comparing \(F_{2,\text{table}}\) against the existing cluster containing \(F_{1,\text{desk}}\), \(g(\text{table}) = g(\text{desk}) = G_1\) is satisfied, and the asymmetric containment \(\text{Cont}_{\texttt{vox}}\) exceeds the threshold, successfully assembling the complete table geometry.
4. Decision Output: During evidence accumulation, table receives support from 3 viewpoints with high mean confidence, prevailing over desk (\(N=1\)). The final 3D box accurately encloses the table without leaking into adjacent chairs.
Key Experimental Results¶
Main Results¶
Quantitative evaluations are conducted on ScanNetV2 (covering ScanNet20 and the long-tail ScanNet60 taxonomy) and ARKitScenes under both pose-known and pose-free RGB settings, benchmarked against multi-view and point cloud-based baselines.
| Method | Modality / Input | Pose-free | Zero-shot | ScanNet20 [email protected] | ScanNet20 [email protected] | ScanNet60 [email protected] | ScanNet60 [email protected] |
|---|---|---|---|---|---|---|---|
| OV-3DET | Point Cloud + Images | No | No | 18.0 | - | - | - |
| CoDA | Ground-Truth Point Cloud | No | No | 19.3 | - | 9.0 | - |
| OV-Uni3DETR | Ground-Truth Point Cloud | No | No | 25.3 | - | 19.4 | - |
| Zoo3D (Zero-shot) | Ground-Truth Point Cloud + Images | No | Yes | 34.7 | 23.9 | 27.1 | 18.7 |
| Zoo3D (Self-sup) | Ground-Truth Point Cloud + Images | No | No | 37.2 | 26.3 | 32.0 | 20.8 |
| OpenM3D | Multi-View RGB | No | No | 19.8 | 7.3 | - | - |
| Zoo3D (Zero-shot) | Multi-View RGB | No | Yes | 30.5 | 17.3 | 22.0 | 10.4 |
| Zoo3D (Self-sup) | Multi-View RGB | No | No | 32.8 | 15.5 | 23.9 | 10.8 |
| Group3D (Ours) | Multi-View RGB | No | Yes | 51.1 | 27.4 | 29.1 | 13.9 |
| Zoo3D (Zero-shot) | Multi-View RGB | Yes | Yes | 24.2 | 8.8 | 13.3 | 4.1 |
| Zoo3D (Self-sup) | Multi-View RGB | Yes | No | 27.9 | 10.4 | 15.3 | 5.6 |
| Group3D (Ours) | Multi-View RGB | Yes | Yes | 41.2 | 18.5 | 22.3 | 8.5 |
Ablation Study¶
Ablations on ScanNet20 under the pose-free zero-shot setup dissect the contribution of each core component (Table 5) alongside different model backbones (Table 4).
| Configuration / Variant | Component Breakdown | ScanNet20 [email protected] | ScanNet20 [email protected] | Note |
|---|---|---|---|---|
| Geometry-only | Pure Voxel IoU & Containment Clustering | 28.2 | 9.9 | Severe cross-category over-merging without semantic constraints |
| + Scene Vocabulary Memory | Strict Same-Category Merge Constraint | 35.9 | 14.8 | Mitigates over-merging, but vulnerable to cross-view label noise |
| + Semantic Compatibility Grouping (Full) | Group3D Complete Pipeline | 41.2 | 18.5 | Optimal balance of semantic protection and synonym tolerance (+13.0 mAP25) |
| Alternative Reconstruction | VGGT instead of Depth Anything 3 | 40.0 | 18.7 | VGGT is pretrained on ScanNet; DA3 achieves superior zero-shot results |
| Alternative MLLM | Qwen3-VL-8B instead of GPT-5.1 | 38.5 | 16.9 | Smaller open-source MLLM retains strong competitive performance |
| Alternative Segmentation | Grounded SAM 2 instead of SAM 3 | 39.7 | 17.6 | Slight drop due to confidence formulation differences, overall trend intact |
Key Findings¶
- Semantic Gating Dictates the Upper Bound: Merging solely via geometric metrics yields only 28.2 mAP25. Enforcing exact label matching increases this to 35.9, while enabling MLLM semantic compatibility grouping leaps to 41.2 mAP25 (a massive +13.0 gain over pure geometry), proving that early semantic intervention is vital in noisy multi-view reconstruction.
- Outperforming Point Cloud Inputs via Pure RGB: With known poses on ScanNet20, Group3D achieves 51.1 mAP25 using multi-view RGB alone, surpassing the best point cloud-based baseline Zoo3D (37.2 mAP25) by 13.9 points, showing that structured semantic association compensates for imperfect 3D sensor measurements.
- Robustness to Category Scales and Long-Tail Scalability: Varying view hypothesis size \(K\) from 5 to 10 maintains 41.2 mAP25 (mAP50 slightly shifts from 18.5 to 18.8). On the fine-grained 200-class ScanNet200 benchmark, Group3D achieves 17.9 mAP25 (posed) and 12.6 mAP25 (pose-free), consistently dominating existing methods across long-tail categories.
Highlights & Insights¶
- Transforming Language Commonsense into Physical Boundary Firewalls: While prior works limit MLLMs to final classification scoring, Group3D harnesses MLLM relational reasoning to formulate merge-time constraints, explicitly distinguishing semantic equivalence from structural adjacency (e.g., forbidding wall-door or floor-wall associations).
- Asymmetric Geometric Containment Metric: Integrating the containment ratio \(\text{Cont}_{\texttt{vox}}\) overcomes the fundamental drawback of symmetric IoU on multi-scale fragments, enabling large instance cores to absorb small, fragmented peripheral structures without penalty.
- Training-Free Zero-Shot Pipeline: Group3D requires zero 3D bounding box training or parameter tuning, combining foundational 2D segmentation, monocular depth estimation, and language reasoning directly during inference to achieve superior zero-shot transfer across datasets (ScanNet to ARKitScenes).
Limitations & Future Work¶
- Dependency on Upstream 2D Open-Vocabulary Recall: If the MLLM fails to propose an object's category name during the initial multi-view scan, SAM 3 will never receive the corresponding prompt, permanently missing that object in 3D space.
- Axis-Aligned Bounding Box Constraints: 3D bounding boxes are currently formed via simple min-max coordinate extents of merged clusters, introducing significant volume bloat and background noise for rotated objects with arbitrary yaw angles.
- Future Directions: Integrating lightweight 3D oriented bounding box (OBB) estimation algorithms and designing iterative closed-loop prompting where 3D geometric uncertainty directs the MLLM to re-examine suspicious regions.
Related Work & Insights¶
- vs Zoo3D [17]: Zoo3D clusters lifted 2D masks using geometry alone before assigning CLIP text embeddings, leading to catastrophic over-merging in pose-free noisy regimes. Group3D enforces semantic compatibility grouping as a merge gate, outperforming Zoo3D by 13.3 mAP25 on pose-free ScanNet20 (41.2 vs 27.9).
- vs OpenM3D [11]: OpenM3D relies on complex pseudo-3D-box generation and expensive model training. Group3D is completely training-free, operating test-time foundation model synergy to outperform OpenM3D by over 31 mAP25 on ScanNet20 (51.1 vs 19.8).
- Insight: In 3D multi-modal perception, low-level geometry is noisy and ambiguous, whereas high-level language priors possess strong structural coherence. Projecting semantic constraints downward into low-level geometric aggregation is far superior to attempting semantic disambiguation after geometry has already collapsed.
Rating¶
- Novelty: ⭐⭐⭐⭐ [Ingenious integration of MLLM semantic equivalence grouping into 3D fragment clustering]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Exhaustive benchmarks across ScanNet 20/60/200 and ARKitScenes under both posed and pose-free settings]
- Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, rigorous mathematical formulation, and well-structured insights]
- Value: ⭐⭐⭐⭐ [Provides an effective, training-free blueprint for RGB-only open-vocabulary 3D scene perception]