GO-N3RDet: Geometry Optimized NeRF-enhanced 3D Object Detector

Conference: CVPR 2025
arXiv: 2503.15211
Code: https://github.com/ZechuanLi/GO-N3RDet
Area: 3D Vision
Keywords: Multi-view 3D Object Detection, NeRF, Voxel Optimization, Opacity Estimation, Indoor Scene Understanding

TL;DR

GO-N3RDet is proposed to address the lack of 3D spatial positioning and insufficient scene geometry awareness in NeRF-based multi-view 3D detection. By introducing three collaborative modules—the Position-Embedded Voxel Optimization Module (PEOM), Dual Importance Sampling (DIS), and Opacity Optimization Module (OOM)—it establishes a new SOTA on ScanNet and ARKitScenes.

Background & Motivation

1. Background: Multi-view 3D object detection, which utilizes low-cost cameras instead of expensive LiDAR or depth sensors, has attracted significant attention in applications such as indoor robot navigation, scene understanding, and AR. The core challenge lies in constructing high-quality 3D feature volumes from multi-view 2D images.
2. Limitations of Prior Work: (1) Methods like ImVoxelNet use average pooling to fuse multi-view features, which discards fine-grained geometric details; (2) NeRF-Det introduces a NeRF branch to predict voxel opacity but focuses on overall scene geometry while ignoring object-level details, resulting in imprecise opacity estimation; (3) 2D image features lack 3D spatial positioning information, making precise 3D object localization difficult.
3. Key Challenge: NeRF is inherently a scene-level rendering tool, whereas 3D object detection requires precise object-level perception. Existing NeRF-based methods fail to address two fundamental issues: the lack of 3D spatial encoding in projected 2D features and the multi-view inconsistency of voxel opacity.
4. Goal: (1) How to embed 3D spatial positioning information into voxel features to correct projection errors? (2) How to make the NeRF branch focus more on foreground object regions? (3) How to ensure that opacity predictions remain consistent across different views?
5. Key Insight: Comprehensively optimize the NeRF-based detector from three complementary perspectives—voxel construction (PEOM), sampling strategy (DIS), and opacity prediction (OOM)—which collaborate to form an end-to-end network.
6. Core Idea: Comprehensively enhance the geometric perception capabilities of the NeRF branch to improve 3D detection performance by optimizing voxels with embedded 3D spatial positioning, focusing on the foreground via dual importance sampling, and constraining opacity with multi-view consistency.

Method

Overall Architecture

Given \(N\) multi-view indoor images and their camera parameters as input, the method outputs 3D bounding boxes (center coordinates, size, orientation, and category). Pipeline: (1) ResNet-50/101 extracts 2D feature maps, which are projected onto an \(N_x \times N_y \times N_z\) voxel grid to obtain multi-view voxel features (similar to NeRF-Det); (2) The PEOM module dynamically adjusts the projection positions, fuses multi-view features, and embeds 3D position encodings; (3) Within the NeRF branch, DIS samples more foreground points, and OOM constrains the multi-view consistency of opacity; (4) The optimized opacity is used to adjust the voxel features, which are then fed into the 3D detection head to generate detection results.

Key Designs

1. Position-Embedded Voxel Optimization Module (PEOM):

   • Function: Dynamically selects projection positions and embeds 3D spatial information into voxel features, replacing traditional average-pooling fusion.
   • Mechanism: (1) After projecting voxel centers onto each view, an MLP predicts an offset \((Δu_i, Δv_i) = \text{MLP}(F_{I_i}, \mathbf{p})\) based on image features and voxel coordinates to adjust the projection position; (2) Max pooling (instead of average pooling) is used to fuse multi-view features, preserving the most salient information while suppressing noise; (3) The view corresponding to the maximum response in max pooling is identified, and the optimized 3D voxel position \(\mathbf{p}_s\) is determined through back-projection; (4) A positional encoder is employed to encode \(\mathbf{p}_s\), which is then added to the voxel features: \(V_{encoded}(\mathbf{p}) = V^{pooled}(\mathbf{p}) + \text{Encoder}(\mathbf{p}_s)\).
   • Design Motivation: The original voxel center may fall into the background region, where its projected features are unhelpful or even detrimental to detection. Dynamic offsets shift the projection toward more meaningful foreground locations. Embedding 3D positional information compensates for the spatial dimensions missing in 2D image features. Max pooling retains the most discriminative view features, which is more suitable for detection tasks than average pooling.

2. Dual Importance Sampling (DIS):

   • Function: Focuses the sampling strategy of the NeRF branch on foreground object regions, improving the accuracy of opacity predictions for foreground voxels.
   • Mechanism: First, \(N_{samples}\) points are uniformly sampled along the ray. For each sample point, two types of densities are calculated: the density \(\rho_i^m\) predicted by the NeRF MLP, and the density based on the distance to the nearest voxel center: \(\rho_i^v = ({\frac{1}{k}\sum_{j=1}^{k}\|\mathbf{p}_i - \mathbf{p}_{i_j}\|})^{-1}\). After normalizing both densities, they are combined as \(w_i = \alpha\hat{\rho}_i^m + \beta\hat{\rho}_i^v\), and an inverse transform sampling on the constructed CDF generates \(N_{fine}\) fine sample points. Regions with high voxel density are more likely to be object surfaces, while regions with high NeRF density point to where matter exists in the scene.
   • Design Motivation: Standard importance sampling in NeRF only considers rendering density, which is insufficient for detection tasks. Introducing the distance to voxel centers as an additional density signal naturally biases sampling toward object regions (where voxel centers are more densely populated on objects). The combination of dual densities balances scene coverage and foreground focus.

3. Opacity Optimization Module (OOM):

   • Function: Enforces consistent opacity predictions for the same 3D position across different views, improving the quality of the opacity grid.
   • Mechanism: Predicts the opacity of the same voxel position from multiple views and computes the variance of these predictions as a consistency constraint loss. Simultaneously, a ray-distance weighting is introduced—views that are further away have less reliable predictions (due to greater cumulative error) and are thus assigned smaller weights. Specifically, predictions from different views are weighted using \(w_d = 1/\|\mathbf{p} - \mathbf{o}\|\), and the final consistency loss is the weighted variance of the predictions from different views.
   • Design Motivation: NeRF-Det predicts opacity point-by-point independently, which can yield contradicting results across different views, resulting in an unsmooth opacity grid. Distance weighting accounts for cumulative errors during ray propagation, making predictions from closer views more trusted.

Loss & Training

The total loss consists of the detection loss (3D box regression + classification), the NeRF rendering loss (RGB rendering MSE), and the OOM consistency loss. The network is trained end-to-end, utilizing ResNet-50 or ResNet-101 as the 2D backbone. Some variants leverage depth rendering supervision (indicated by *).

Key Experimental Results

Main Results

ScanNet validation set [email protected]:

Method	[email protected]	Depth Supervision	Backbone
ImVoxelNet	48.4	No	-
NeRF-Det-R50	52.0	No	R50
NeRF-Det-R101*	53.3	Yes	R101
ImGeoNet	54.8	Yes	-
NeRF-DetS*	57.5	Yes	-
MVSDet*	56.2	Yes	-
GO-N3RDet-R50	56.3	No	R50
GO-N3RDet-R101*	58.6	Yes	R101

The version without depth supervision, GO-N3RDet-R50 (56.3), already outperforms most methods that utilize depth supervision. The R101* version achieves 58.6 mAP, establishing a new SOTA for NeRF-based methods.

Ablation Study

ScanNet [email protected], based on R50:

Configuration	[email protected]	Gain	Note
Baseline (NeRF-Det)	51.8	-	Baseline
+ PEOM (avg pool)	53.2	+1.4	PEOM + average pooling
+ PEOM (max pool)	54.9	+3.1	PEOM + max pooling
+ PEOM + DIS	55.4	+3.6	Adding dual importance sampling
+ PEOM + DIS + OOM	56.3	+4.5	Full model

Key Findings

• PEOM contributes the most (+3.1 mAP with max pool), demonstrating that embedding 3D positional information and dynamically adjusting projections are critical to detection performance.
• max pooling vs avg pooling: Max pooling yields a 1.7 mAP improvement over avg pooling (53.2 -> 54.9), validating that retaining the most salient features is more suitable for detection than averaging in multi-view fusion.
• Contribution of DIS: +0.5 mAP. Although the absolute gain is modest, it is stable and effective, as a foreground-focused sampling strategy leads to more accurate opacity predictions.
• Contribution of OOM: +0.9 mAP. The multi-view consistency constraint makes the opacity grid smoother and more reliable.
• Compared to CN-RMA (58.7 mAP), GO-N3RDet-R101* (58.6) achieves comparable performance but requires only ~10 hours of training compared to CN-RMA's ~30 hours, showing a significant efficiency advantage.
• It performs best on large objects like beds and chairs, while the improvement on small objects like pictures is limited.

Highlights & Insights

• Dynamic Optimization of Voxel Positions: Instead of passively accepting projection positions determined by camera parameters, the model actively learns offsets to shift the projections toward more meaningful regions. This trick can be migrated to any feature back-projection task from 2D to 3D.
• Complementarity of Dual Density Signals: The NeRF density reflects the overall scene geometry, while voxel distance density reflects object position priors. Fusing these two via their CDF balances both scene coverage and foreground focus. This complementary sampling strategy can be generalized to other NeRF applications requiring foreground attention.
• Balance Between Computational Efficiency and Performance: Compared to CN-RMA, which requires pretrained 3D reconstruction, GO-N3RDet is trained fully end-to-end and is 3 times faster, making it more suitable for practical deployment.

Limitations & Future Work

• Although the improvement on small objects like pictures is significant (0.8 -> 4.2 mAP), the absolute values remain very low; scenes with large object scale variations are still a challenge.
• The sampling efficiency of the NeRF branch can be further improved. Currently, DIS still relies on an initial uniform distribution; direct learned sampling could be considered.
• OOM uses variance as a consistency constraint, which may produce false-positive consistencies in highly occluded multi-view scenes.
• Validation has only been performed on indoor datasets (ScanNet, ARKitScenes), so its generalizability to outdoor scenes remains unknown.
• The possibility of replacing MLPs with Transformers for feature aggregation has not been explored.

Related Work & Insights

• vs NeRF-Det: While the original NeRF-Det shares MLPs for detection and NeRF, this work comprehensively enhances the geometric awareness of the NeRF branch from three perspectives (voxel, sampling, and opacity), representing a direct and comprehensive improvement.
• vs ImGeoNet: ImGeoNet improves geometric awareness by supervising voxel weights but does not embed 3D positional information. PEOM addresses this issue more precisely through dynamic projections and positional encodings.
• vs CN-RMA: CN-RMA utilizes Ray Marching Aggregation and pretrained 3D reconstruction, showing slightly better performance but requiring three times the training cost. GO-N3RDet achieves comparable results within an end-to-end training framework.
• The dynamic offset prediction idea in PEOM is conceptually similar to deformable convolution, but it is applied to 3D-2D projection rather than kernel offsets, representing an interesting analogy.

Rating

• Novelty: ⭐⭐⭐⭐ Each of the three modules has its own innovations; PEOM's dynamic projection + positional embedding design is the most novel.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated on two datasets, with detailed ablation studies, comparisons against multiple methods, and complete per-class analysis.
• Writing Quality: ⭐⭐⭐⭐ Clear problem analysis, intuitive module diagrams, and rigorous methodology descriptions.
• Value: ⭐⭐⭐⭐ Establishes a new SOTA in NeRF-based 3D detection while maintaining a training efficiency advantage.

Related Papers

• [CVPR 2025] PBR-NeRF: Inverse Rendering with Physics-Based Neural Fields
• [CVPR 2025] RDD: Robust Feature Detector and Descriptor Using Deformable Transformer
• [CVPR 2025] Flow-NeRF: Joint Learning of Geometry, Poses, and Dense Flow within Unified Neural Representations
• [CVPR 2025] Textured Gaussians for Enhanced 3D Scene Appearance Modeling
• [CVPR 2025] Guiding Human-Object Interactions with Rich Geometry and Relations