Open-Vocabulary 3D Object Detection with Co-Distillation Discovery and Dual Guidance Robust Training

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/shangboyuan/Co-3DGT
Area: 3D Vision
Keywords: Open-vocabulary 3D object detection, Co-distillation, Uncertainty regularization, Hierarchical alignment, Pseudo-label discovery

TL;DR

To tackle inaccurate localization and mismatched semantics in novel object discovery alongside indiscriminate supervision during detector training, Co-3DGT unifies 2D-3D co-distillation bipartite matching with scene-aware uncertainty regularization and LLM-guided hierarchical alignment, substantially advancing unseen category detection on SUN RGB-D and ScanNetV2.

Background & Motivation

Open-vocabulary 3D object detection (3D-OVD) aims to localize and classify novel 3D object categories beyond training categories by harnessing the extensive vocabulary of vision-language foundation models like CLIP. However, due to the severe domain discrepancy between 2D images and 3D point clouds, along with the scarcity of large-scale annotated 3D datasets, training full 3D foundation models from scratch remains prohibitively expensive. Consequently, prevailing approaches adopt a two-stage paradigm: first leveraging 2D/3D foundation models to discover novel 3D object candidates as pseudo-labels in unlabeled scenes, and subsequently training a 3D detector supervised jointly by annotated base objects and discovered novel objects.

Despite its viability, this two-stage pipeline suffers from inherent structural dilemmas across both stages. In the novel object discovery stage, existing methods bifurcate into 2D-Detection-based and 3D-Proposal-based schemes, each exhibiting critical vulnerabilities. 2D-Detection-based methods leverage rich global contextual cues to predict accurate open-vocabulary semantic categories, but their back-projected 3D bounding boxes are heavily corrupted by depth noise and adjacent background clutter. Conversely, 3D-Proposal-based methods generate geometrically tight 3D bounding boxes via class-agnostic 3D detectors, yet cropping and projecting these proposals into 2D image planes for CLIP classification frequently triggers severe semantic mismatches due to perspective occlusions and partial views. Furthermore, during the subsequent training stage, existing pipelines uniformly enforce equal supervision loss weights on pristine ground-truth base objects and noisy discovered pseudo-labels, allowing inaccurate bounding box regressions and semantic classification noise to degrade learned feature representations.

To resolve these compounding issues, the key angle of attack is to decouple and unite the complementary strengths of 2D semantics and 3D geometry while explicitly regularizing supervision noise during training. Core idea: formulate novel object discovery as a 2D-3D bipartite Hungarian matching problem combining spatial overlap, 3D objectness, and 2D semantic certainty to distill high-purity pseudo-labels, while regularizing 3D regression via base-class scene-aware uncertainty and aligning classification features with LLM-generated hierarchical super-categories.

Method

Overall Architecture

The Co-3DGT framework comprises two fundamental phases: Co-Distillation Discovery and Dual Guidance Robust Training. In the discovery phase, the framework runs an open-vocabulary 2D detector and a class-agnostic 3D detector in parallel, binding accurate 2D category predictions with tight 3D geometric bounding boxes through bipartite matching. In the robust training phase, the 3D detector is trained on both base ground-truth boxes and distilled novel pseudo-labels, supervised by a regression head regularized with scene-aware uncertainty and a classification head aligned with LLM-derived multi-level semantic hierarchies.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Scene Data<br/>RGB-D Images + Point Clouds"] --> B["Dual Candidate Generation<br/>2D Semantic Detections & 3D Class-Agnostic Proposals"]
    B --> C["Cross-Modal Bipartite Matching<br/>3D IoU + 2D/3D Confidence Cost Matrix"]
    C --> D["Distilled Novel Pseudo-Labels<br/>Paired 3D Geometry BBoxes with 2D Semantic Labels"]
    D --> E["Dual Guidance Robust Training<br/>Joint Supervision of Base GT and Novel Pseudo-Labels"]
    E --> F["Scene-Aware Uncertainty Regularization<br/>Anchor Novel 3D Box Regression using Base Scene Uncertainty"]
    E --> G["LLM-Guided Hierarchical Alignment<br/>Multi-Level Semantic Alignments top / mid / bottom"]
    F --> H["Final Open-Vocabulary 3D Detections"]
    G --> H

Key Designs

1. Cross-Modal Co-Distillation Discovery: Unified Spatial-Semantic Bipartite Matching

Prior 3D-OVD discovery pipelines either suffer from oversized, noisy back-projected bounding boxes or false semantic labeling induced by occluded image crops. Co-3DGT resolves this dilemma by executing 2D open-vocabulary detection and 3D class-agnostic proposal generation concurrently, yielding back-projected 3D objects \(\mathcal{O}_{2\text{D}} = \{(\mathbf{b}_{2\text{D}}^i, c_{2\text{D}}^i, s_{2\text{D}}^i)\}_{i=1}^M\) and class-agnostic proposals \(\mathcal{O}_{3\text{D}} = \{(\mathbf{b}_{3\text{D}}^j, s_{3\text{D}}^j)\}_{j=1}^N\). To pair tight 3D spatial geometry with reliable 2D semantic labels, candidate assignment is modeled as a bipartite matching problem optimized via the Hungarian algorithm over cost matrix \(C \in \mathbb{R}^{M \times N}\):

\[C_{ij} = -\text{IoU}_{ij} - 0.1 \cdot (s_j^{\text{3D}} + s_i^{\text{2D}})\]

Here, \(\text{IoU}_{ij}\) measures the 3D Intersection over Union between the back-projected candidate and the 3D proposal. A minimum geometric threshold \(\tau = 0.1\) is enforced such that \(C_{ij} = \infty\) whenever \(\text{IoU}_{ij} < \tau\). The 3D foreground score \(s_j^{\text{3D}}\) and 2D classification score \(s_i^{\text{2D}}\) provide soft regularizing weights while preserving geometric alignment as the dominant criterion. By solving for optimal matching pairs \((i, j) \in \mathcal{H}\), the distilled novel set \(\mathcal{O}_{\text{Co-D}}^{\text{novel}}\) pairs the precise 3D spatial box \(\mathbf{b}_j^{\text{3D}}\) with the reliable open-vocabulary class label \(c_i^{\text{2D}}\), filtering out erroneous outliers before model training begins.

2. Scene-Awareness-Guided Uncertainty Regularization: Anchoring Novel Box Regression

During 3D detector training, ground-truth base objects possess accurate human annotations, whereas novel pseudo-labels retain unavoidable spatial noise. Direct unconstrained heteroscedastic regression on noisy novel boxes risks severe overfitting or gradient degradation. For base objects, standard heteroscedastic loss is applied:

\[\mathcal{L}_{\text{Reg}}^{\text{base}} = \sum_i \left[ \frac{\sqrt{2}}{\sigma_i} \|\hat{\mathbf{b}}_i - \mathbf{b}_i\|_1 + \log \sigma_i + \left\lfloor \frac{\sigma_i}{\sqrt{2}} \right\rfloor^{\frac{1}{2}} \right]\]

where \(\sigma_i\) is the predicted uncertainty and \(\lfloor \cdot \rfloor\) denotes a stop-gradient operator. For novel objects, Co-3DGT introduces scene-awareness-guided uncertainty regularization (SGUR), utilizing the scene-level expected log-uncertainty of base objects \(\mathbb{E}[\lfloor \log \sigma_{\text{base}} \rfloor]\) as an intrinsic baseline of scene difficulty (maintained via exponential moving average across scenes without base instances). The novel regression loss is formulated as:

\[\mathcal{L}_{\text{Reg}}^{\text{novel}} = \sum_j \left( \frac{\sqrt{2}}{\sigma_j} \|\hat{\mathbf{b}}_j - \mathbf{b}_j\|_1 + \left| \log \sigma_j - \mathbb{E}[\lfloor \log \sigma_i \rfloor] \right| \right)\]

This soft bidirectional penalty drives the predicted uncertainty of novel instances toward the calibrated base uncertainty distribution, preventing novel predictions from collapsing into arbitrary overconfidence or unchecked divergence.

3. LLM-Guided Hierarchical Alignment: Mitigating Semantic Ambiguity with Structured Super-Categories

Fine-grained category labels often suffer from visual ambiguity and label noise in open-vocabulary settings (e.g., distinguishing bedside tables from cabinets in cluttered scans). Directly aligning fine-grained CLIP embeddings with 3D features amplifies this noise. Co-3DGT leverages an LLM (such as ChatGPT-5) to construct a three-tier semantic taxonomy for all categories: fine-grained bottom labels \(\mathcal{C}^{\text{bot}} = \mathcal{C}_{\text{base}} \cup \mathcal{C}_{\text{novel}}\) (e.g., chair, table, fridge), functional/geometric mid-level super-categories \(\mathcal{C}^{\text{mid}}\) (e.g., seating, storage, domestic), and abstract top-level concepts \(\mathcal{C}^{\text{top}}\) (e.g., furniture, appliance).

Using a frozen CLIP text encoder, hierarchical textual embeddings \(\mathbf{h}^{(1)}, \mathbf{h}^{(2)}, \mathbf{h}^{(3)}\) are obtained and aligned with multi-level 3D visual features \(\mathbf{f}^{(1)}, \mathbf{f}^{(2)}, \mathbf{f}^{(3)}\) extracted by the detector. The model computes cosine similarity mapped via sigmoid to probabilities \(p_{ij}^{(l)}\) and optimizes multi-level binary cross-entropy (BCE) losses:

\[\mathcal{L}_{\text{Align}} = \mathcal{L}_{\text{bce}}^{(1)} + \mathcal{L}_{\text{bce}}^{(2)} + \mathcal{L}_{\text{bce}}^{(3)}\]

Because the mid and top levels group geometrically and functionally coherent classes together, they are significantly more robust against fine-grained label noise, providing stable manifold guidance even when bottom-level labels are noisy.

Loss & Training

The complete 3D-OVD training loss integrates both the regularized regression loss and the hierarchical alignment loss:

\[\mathcal{L}_{\text{total}} = (\mathcal{L}_{\text{Reg}}^{\text{base}} + \mathcal{L}_{\text{Reg}}^{\text{novel}}) + (\mathcal{L}_{\text{bce}}^{(1)} + \mathcal{L}_{\text{bce}}^{(2)} + \mathcal{L}_{\text{bce}}^{(3)})\]

Built on the Uni3DETR voxel encoder and transformer decoder architecture, Co-3DGT operates exclusively on 3D point cloud inputs during training and inference. By decoupling 2D detection into an offline discovery phase, the framework circumvents expensive online multi-modal cross-attention, significantly reducing training overhead.

Key Experimental Results

Main Results

Co-3DGT is extensively evaluated on SUN RGB-D (10 base classes, 36 novel classes) and ScanNetV2 (10 base classes, 50 novel classes) across unseen, seen, and mean category metrics.

Table 1: Main 3D-OVD comparison on SUN RGB-D and ScanNetV2 (AP25 / AR25 in %)

Method	Modality	SUN RGB-D APnovel25	SUN RGB-D APbase25	SUN RGB-D APmean25	ScanNetV2 APnovel25	ScanNetV2 APbase25	ScanNetV2 APmean25
Det-PointCLIP [43]	Point Cloud	0.09	5.04	1.17	0.13	2.38	0.50
Det-PointCLIPv2 [42]	Point Cloud	0.12	4.82	1.14	0.13	1.75	0.40
Det-CLIP2 [46]	Point Cloud	0.88	22.74	5.63	0.14	1.76	0.40
CoDA [3]	Point Cloud	6.71	38.72	13.66	6.54	21.57	9.04
INHA [17]	Point Cloud	8.91	42.17	16.18	7.79	25.10	10.68
CoDAv2 [4]	Point Cloud	9.17	42.04	16.31	9.12	23.35	11.49
OV-Uni3DETR [39]	Point Cloud + RGB	9.66	48.29	18.06	12.09	30.47	15.15
Co-3DGT (Ours)	Point Cloud	14.37	49.85	22.63	21.91	32.83	23.75

Under the annotation-free setting following OV-3DET (without any ground-truth 3D base annotations), a simplified version of Co-3DGT retaining Co-Distillation and LLM hierarchical alignment demonstrates superior generalization across 20 unseen categories:

Table 2: Per-category AP25 (%) on ScanNetV2 across 20 unseen classes in annotation-free setting

Method	Mean AP	toilet	bed	chair	sofa	table	bathtub	refrigerator	lamp
OV-3DET [24]	18.02	57.29	42.26	27.06	31.50	14.17	56.28	10.99	2.11
CoDA [3]	19.32	68.09	44.04	28.72	44.57	20.23	50.51	6.55	0.51
CoDAv2 [4]	22.72	77.24	43.96	15.05	53.27	13.96	55.60	24.41	4.37
OV-Uni3DETR [39]	25.33	86.05	50.49	28.11	31.51	24.03	63.73	14.41	5.58
Co-3DGT (Ours)	36.15	92.75	70.42	81.27	60.64	39.15	69.52	34.57	38.61

Ablation Study

Ablation studies analyze candidate discovery fidelity, training guidance contributions, and LLM backbone sensitivity.

Table 3: Ablation of Co-Distillation discovery and downstream detection performance (AP / AR in %)

Discovery Scheme	Training Discovery APnovel,disc25 (SUN)	Training Discovery ARnovel,disc25 (SUN)	Final Trained APnovel,det25 (SUN)	Final Trained APnovel,det25 (ScanNet)	Note
2D-Detection-based	6.21	16.10	9.59	11.92	Back-projection accumulates depth noise
3D-Proposal-based	8.49	12.02	9.88	12.71	Image crops prone to occlusion error
Co-distil (from scratch)	10.45 (+1.96)	19.72 (+7.70)	11.76 (+1.88)	16.68 (+3.97)	Hungarian matching removes mismatched pairs
Co-distil (CuTR detector)	10.97 (+2.48)	24.33 (+12.31)	12.69 (+2.81)	18.74 (+6.03)	Optimal geometry-semantic pairing

Table 4: Ablation on Dual Guidance Robust Training components (APmean25 in %)

Variant	SGUR (Uncertainty Reg.)	LGHA (Hierarchical Align.)	SUN RGB-D APmean25	ScanNetV2 APmean25	Note
Baseline w/o dual guidance	✗	✗	20.62	20.15	Direct pseudo-label fine-tuning
SGUR only	✓	✗	22.17 (+1.55)	23.12 (+2.97)	Calibrates regression uncertainty
LGHA only	✗	✓	21.35 (+0.73)	22.73 (+2.58)	Multi-granularity classification regularization
Full Dual Guidance	✓	✓	22.63 (+2.01)	23.75 (+3.60)	Complementary orthogonal improvements

Key Findings

• Co-distillation substantially improves pseudo-label recall and precision: incorporating CuTR into co-distillation boosts discovery recall on ScanNetV2 by +12.01%, directly translating into a +6.03% gain in final trained novel object detection AP.
• Uncertainty regularization and hierarchical alignment exhibit strong complementarity: SGUR stabilizes geometric bounding box regression while LGHA refines category discrimination, collaboratively driving a +3.60% APmean gain on ScanNetV2.
• Robustness to LLM model choice: evaluating alternative LLM generators in LGHA (ChatGPT-5 at 23.75%, Gemini-3-Flash-Thinking at 23.84%, Qwen3.5-397B at 23.34%, Llama-3.3-70B at 23.51% on ScanNetV2) shows minor variation (<0.5%), confirming that performance gains stem from the hierarchical taxonomic structure rather than specific LLM phrasing.

Highlights & Insights

• Complementary cross-modal pairing: Instead of allowing 2D back-projection errors and 3D crop occlusions to degrade performance independently, bipartite Hungarian matching extracts the best of both worlds—2D semantic breadth and 3D spatial tightness.
• Intrinsic scene-aware uncertainty anchoring: Grounding novel object regression uncertainty in the empirical base-class uncertainty of the same scene provides a principled, self-adjusting regularizer grounded in Maximum Mean Discrepancy (MMD) theory.
• Efficient point-cloud-only inference: Even though multimodal 2D-3D knowledge is exploited during offline discovery and training, the deployed model requires only point cloud inputs, cutting training time from 716 minutes (OV-Uni3DETR) to 490 minutes on SUN RGB-D while eliminating multimodal inference latency.

Limitations & Future Work

• Two-stage offline dependency: Co-distillation requires pre-generating 2D and 3D proposals across all scene scans prior to model optimization, preventing real-time or streaming open-vocabulary adaptation.
• Sensitivity to extreme sparsity and occlusion: In distant or severely occluded scenes where point cloud density is sparse, back-projected 2D boxes may fail to capture sufficient 3D points, preventing successful bipartite assignment.
• Future directions: Extending the co-distillation paradigm to streaming multi-frame temporal point cloud sequences for real-time mobile robotics and embodied navigation.

Related Work & Insights

• vs OV-3DET: OV-3DET relies entirely on back-projecting 2D bounding boxes, resulting in noisy, oversized 3D bounding boxes; Co-3DGT pairs 2D semantics with native 3D proposals, yielding much sharper spatial boundaries and substantially higher detection accuracy.
• vs CoDA / CoDAv2: CoDA relies on cropped 2D regions corresponding to 3D proposals for zero-shot CLIP classification, which suffers from severe occlusion ambiguity; Co-3DGT leverages full-frame 2D open-vocabulary detectors and hierarchical semantic structures to overcome visual ambiguity.
• vs OV-Uni3DETR: OV-Uni3DETR employs complex multimodal cross-attention fusion at inference; Co-3DGT shows that robust pseudo-label distillation and regularized training allow a pure point cloud detector to surpass multimodal baselines with greater efficiency.

Rating

• Novelty: ⭐⭐⭐⭐☆ Elegantly resolves long-standing pseudo-label noise in 3D-OVD via dual-source co-distillation and scene-aware uncertainty regularization.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive benchmarks across SUN RGB-D and ScanNetV2, detailed discovery-vs-training ablations, and extensive LLM sensitivity comparisons.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear exposition, intuitive visual comparisons, and rigorous mathematical formulations.
• Value: ⭐⭐⭐⭐☆ Highly impactful for open-world indoor perception, embodied robotics, and efficient 3D scene understanding.

Related Papers

• [ECCV 2026] From Perspective to Fisheye Depth Estimation and Open-Vocabulary Segmentation
• [ICLR 2026] GeoPurify: A Data-Efficient Geometric Distillation Framework for Open-Vocabulary 3D Segmentation
• [ECCV 2026] OP3DSG: Open-Vocabulary Part-Aware 3D Scene Graph Generation for Real-World Environments
• [ECCV 2024] Open Vocabulary 3D Scene Understanding via Geometry Guided Self-Distillation
• [ECCV 2026] TanGO: Training-Free 3D Editing via Tangent-Space Guidance and Optimization