MoCA3D: Monocular 3D Bounding Box Prediction in the Image Plane¶
Conference: ECCV 2026
Paper: ECCV Official
Project Page: https://jeoncwcw.github.io/moca3d/
Area: 3D Vision
Keywords: Monocular 3D Understanding, Image-Plane Geometry, Dense Prediction, Pixel-Aligned Geometry, 3D Bounding Box Prediction
TL;DR¶
MoCA3D bypasses the reliance on camera intrinsics in conventional monocular 3D lifting by reformulating 3D bounding box prediction as an image-plane dense prediction task, achieving superior corner reprojection accuracy and competitive 3D performance with 57x fewer trainable parameters (19.0M).
Background & Motivation¶
Monocular 3D object understanding has predominantly been formulated as an instance lifting problem in 3D camera or world coordinates. Leading architectures like Cube R-CNN and DetAny3D predict oriented 3D cuboids or 6D poses directly from 2D regions of interest (RoIs). Under this paradigm, geometric information on the image plane is merely obtained post-hoc by projecting 3D parameters back into pixel space. Consequently, these frameworks strictly depend on known and reliable camera intrinsics at inference time—a brittle assumption in open-world settings where intrinsics are often absent, uncalibrated, or erratic.
Simultaneously, emerging downstream tasks such as controllable street-view synthesis and diffusion-based scene editing consume geometry primarily in the image plane (e.g., perspective-projected 3D box corners with depth assignments). When conventional lifting pipelines are forced to regress image-plane geometry directly from RoI features using compact coordinate vectors, they suffer from severe gradient sparsity, weak supervisory signals, and optimization instability. Furthermore, conventional 3D evaluation metrics such as 3D IoU and translation errors obscure reprojection discrepancies, failing to penalize critical geometric misalignments in the image plane.
To resolve the dual challenge of missing camera intrinsics and sparse regression instability, MoCA3D anchors the entire 3D geometry prediction process within the image plane. Core idea: reformulate camera-intrinsics-free monocular 3D object recovery as a dense prediction task of corner heatmaps and depth maps, employing relative-coordinate spatial priors, reversed cross-attention, and differentiable soft-argmax to achieve pixel-aligned geometric fidelity.
Method¶
Overall Architecture¶
Given a single RGB image and a tight 2D bounding box prompt, MoCA3D directly predicts the 8 projected 3D box corners \(\hat{\mathbf{p}}_i = (\hat{u}_i, \hat{v}_i)\) and their per-corner depths \(\hat{d}_i\) without requiring camera intrinsics at inference. The architecture comprises three sequential stages: (i) dense image feature extraction using a frozen DINOv3 ViT foundation model; (ii) a box-conditioned 3D Geometry Transformer that encodes 2D box relative-coordinate priors into image tokens and uses reversed cross-attention to condition dense tokens on box embeddings; and (iii) lightweight convolutional dense prediction heads that output 8 corner heatmaps and depth maps, followed by differentiable soft-argmax coordinate extraction and bilinear depth sampling.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: RGB Image + 2D Bounding Box"] --> B["Stage 1: Relative-Coordinate Box Prior Injection<br/>Inject offset and relative map into DINOv3 features"]
B --> C["Stage 2: Reversed Cross-Attention Decoding<br/>Image tokens act as Query to attend box embeddings"]
C --> D["Stage 3: Progressive Feature Upsampling<br/>Lightweight fusion and 4x spatial upsampling"]
D --> E["Stage 4: Dual-Branch Dense Prediction<br/>8-channel corner heatmaps and depth maps"]
E --> F["Stage 5: Differentiable Extraction & Sampling<br/>Soft-argmax corner localization and bilinear depth sampling"]
F --> G["Output: 8 Projected Corners & Per-Corner Depths"]
Key Designs¶
1. Relative-Coordinate Box Prior Injection: Anchoring dense tokens to target instances
Rather than relying on naive RoI cropping that discards global context, MoCA3D constructs an explicit 4-channel spatial box prior map \(\mathbf{M}(\mathbf{b}) \in \mathbb{R}^{4 \times h_d \times w_d}\). The map encodes center-normalized offsets \((dx, dy)\) (indicating whether a pixel is left/right or above/below the center) and box-normalized coordinates \((u, v)\) relative to the top-left corner. Projected by a learnable layer \(\psi\) and modulated by a learned scalar gate \(\alpha = \sigma(\gamma)\), the prior is additively fused with the frozen DINOv3 features \(\mathbf{F}\):
$\(\tilde{\mathbf{F}} = \mathbf{F} + \alpha \, \psi(\mathbf{M}(\mathbf{b}))\)$
This explicit injection allows the transformer encoder self-attention to immediately distinguish the target instance's spatial extent from background distractors.
2. Reversed Cross-Attention Decoding: Empowering dense tokens as queries
Standard vision transformers in object detection (e.g., DETR) utilize sparse object queries to aggregate information from image feature maps, which degrades dense spatial resolution. MoCA3D reverses this interaction direction: the encoded dense image tokens serve as Queries, while a small set of box embeddings \(\mathbf{B} \in \mathbb{R}^{C \times N_q}\) (\(N_q = 9\)) derived from the 2D box prompt serve as Keys and Values. Through multi-head cross-attention, every spatial token across the image grid directly incorporates instance-level geometric guidance while preserving fine-grained spatial representations.
3. Dual-Branch Dense Prediction with Soft-argmax: Eliminating sparse gradient bottlenecks
After decoding, the feature map is progressively upsampled by a factor of 4 to resolution \(h_o \times w_o\) with channels reduced to \(C/4\), yielding feature map \(\mathbf{U}\). MoCA3D deploys two lightweight convolutional heads: a corner head generating 8-channel heatmaps \(\mathbf{H} \in [0, 1]^{8 \times h_o \times w_o}\) with sigmoid activation, and a depth head generating non-negative depth maps \(\mathbf{Z} \in \mathbb{R}_+^{8 \times h_o \times w_o}\) with softplus activation. Continuous sub-pixel corner locations \(\hat{\mathbf{p}}_i\) are extracted via differentiable soft-argmax with inverse temperature \(\beta\):
$\(\pi_i(x,y) = \frac{\exp(\beta \, \mathbf{H}_i(x,y))}{\sum_{x',y'} \exp(\beta \, \mathbf{H}_i(x',y'))}, \qquad \hat{\mathbf{p}}_i = \left(\sum_{x,y} x \, \pi_i(x,y), \; \sum_{x,y} y \, \pi_i(x,y)\right)\)$
Per-corner depths \(\hat{d}_i\) are subsequently sampled from \(\mathbf{Z}_i\) at predicted positions \(\hat{\mathbf{p}}_i\) using bilinear interpolation, ensuring smooth, end-to-end differentiable gradient propagation.
4. Peak-Weighted and Confidence-Guided Loss Formulation: Fostering joint localization-depth synergy
Corner localization naturally faces severe foreground-background class imbalance. MoCA3D constructs Gaussian target heatmaps \(\mathbf{W}\) and introduces an indicator matrix \(\mathbf{A}_i(x,y)\) with an emphasis factor \(\lambda\) for pixels exceeding threshold \(\tau\), computing a peak-weighted coarse loss \(\mathcal{L}_{\text{coarse}}\). Sub-pixel precision is then enforced via smooth L1 fine coordinate loss \(\mathcal{L}_{\text{fine}}\). Crucially, depth supervision uses predicted corner heatmaps as pixel-wise confidence weights:
$\(\mathcal{L}_{\text{depth}} = \frac{\sum_{i,x,y} \mathbf{H}_i(x,y) \cdot \mathrm{SmoothL1}(\mathbf{Z}_i(x,y) - \bar{d}_i)}{\sum_{i,x,y} \mathbf{H}_i(x,y) + \epsilon}\)$
This formulation ties depth supervision strictly to spatial regions where corners are reliably predicted, preventing ambiguous background regions from corrupting depth feature learning.
Loss & Training¶
The complete multi-task loss is defined as: $\(\mathcal{L} = w_{\text{coarse}} \mathcal{L}_{\text{coarse}} + w_{\text{fine}} \mathcal{L}_{\text{fine}} + w_{\text{depth}} \mathcal{L}_{\text{depth}}\)$ A warm-up scheduling strategy is applied during training: initial optimization activates only the coarse heatmap loss (\(w_{\text{coarse}} = 50\), others 0); weights are subsequently rebalanced linearly (\(w_{\text{coarse}}: 50 \to 1\), \(w_{\text{fine}}: 0 \to 2\), \(w_{\text{depth}}: 0 \to 5\)). The model is trained on 3 NVIDIA RTX 3090 GPUs for 120 epochs (50,000 instances/epoch, ~60 GPU-hours total), benefiting from offline DINOv3 feature pre-extraction (~4 GPU-hours).
When evaluation against standard 3D bounding box formats is required, the authors develop MoCA3D-Cube, a two-layer MLP adapter that takes projected corners, depths, and RoI-pooled features to regress center, dimensions, and rotation under camera intrinsics \(K\).
Key Experimental Results¶
Main Results¶
To assess image-plane geometric precision, the paper establishes the Pixel-Aligned Geometry (PAG) benchmark: image-plane corner reprojection error \(PAG_{uv}\) (mean Euclidean distance in pixels at 512x512 resolution, lower is better) and relative corner depth error \(PAG_d\) (percentage error, lower is better). In 3D space, Normalized Hungarian Distance (NHD) and \(IoU_{3D}\) are evaluated under ground-truth camera intrinsics with oracle 2D bounding boxes across 6 diverse datasets:
| Model | KITTI (\(PAG_{uv}\downarrow\) / \(PAG_d\downarrow\)) | nuScenes (\(PAG_{uv}\downarrow\) / \(PAG_d\downarrow\)) | Objectron (\(PAG_{uv}\downarrow\) / \(PAG_d\downarrow\)) | ARKitScenes (\(PAG_{uv}\downarrow\) / \(PAG_d\downarrow\)) | SUN RGB-D (\(PAG_{uv}\downarrow\) / \(PAG_d\downarrow\)) | Hypersim (\(PAG_{uv}\downarrow\) / \(PAG_d\downarrow\)) |
|---|---|---|---|---|---|---|
| Cube R-CNN | 10.10 px / 12.63% | 11.73 px / 12.05% | 23.57 px / 15.03% | 22.47 px / 13.22% | 23.17 px / 13.92% | 13.92 px / 20.66% |
| Cube R-CNN* (Class-Agnostic) | 10.38 px / 13.85% | 11.86 px / 13.04% | 31.28 px / 33.18% | 23.75 px / 14.92% | 24.06 px / 16.61% | 14.17 px / 22.81% |
| OVMono3D-LIFT* | 10.29 px / 13.20% | 11.82 px / 12.45% | 23.55 px / 12.68% | 22.55 px / 12.60% | 22.36 px / 14.61% | 13.02 px / 19.94% |
| DetAny3D | 4.90 px / 4.82% | 6.75 px / 6.07% | 27.41 px / 12.55% | 17.15 px / 8.87% | 21.06 px / 8.71% | 10.71 px / 16.81% |
| MoCA3D (Ours) | 3.12 px / 5.04% | 5.63 px / 7.17% | 18.60 px / 14.59% | 14.90 px / 8.78% | 16.99 px / 10.08% | 9.56 px / 19.57% |
In 3D metric evaluation: - On KITTI, MoCA3D achieves the best NHD of 0.1902 (surpassing DetAny3D's 0.1911 and Cube R-CNN's 0.2811) and ranks second on \(IoU_{3D}\) with 0.4735 (behind DetAny3D's 0.4834); - On ARKitScenes, MoCA3D scores the best NHD of 0.2529; with the MoCA3D-Cube adapter, \(IoU_{3D}\) increases to 0.3435.
Ablation Study¶
Ablation experiments conducted on the Objectron dataset investigate architectural variations, parameter budgets, and training compute:
| Configuration | \(PAG_{uv}\downarrow\) (px) | \(PAG_d\downarrow\) (%) | \(IoU_{3D}\uparrow\) | Trainable Params | Training Compute |
|---|---|---|---|---|---|
| MoCA3D (Full Baseline) | 16.05 | 10.83 | 0.3768 | 19.0 M | 27.0 GPU-hrs |
| MoCA3D w/ Depth Anything v3 | 16.14 (+0.09) | 10.92 (+0.09) | 0.3682 (-0.0086) | 19.8 M | – |
| MoCA3D w/o box prior map | 16.35 (+0.29) | 10.88 (+0.05) | 0.3736 (-0.0032) | 19.0 M | 27.0 GPU-hrs |
| Direct RoI Vector Regressor | 17.38 (+1.32) | 14.89 (+4.06) | 0.3064 (-0.0704) | 22.1 M | 19.2 GPU-hrs |
Robustness to noisy 2D bounding boxes (tested with GroundingDINO detections instead of oracle 2D boxes): - MoCA3D: degrades gracefully under detected boxes, with \(IoU_{3D}\) decreasing by only 0.0147 (0.2405 \(\to\) 0.2258), \(PAG_d\) increasing by 0.41% (13.44% \(\to\) 13.85%), and \(PAG_{uv}\) increasing by 1.38 px (11.11 px \(\to\) 12.48 px); - DetAny3D shows a comparable degradation (+1.30 px in \(PAG_{uv}\)), confirming MoCA3D's robustness to realistic detector jitter.
Key Findings¶
- Dense supervision prevents optimization collapse: Replacing dense heatmaps and depth maps with an RoI-aligned direct coordinate regressor induces severe performance degradation (\(+1.32\) px in \(PAG_{uv}\), \(+4.06\%\) in \(PAG_d\), and \(-0.0704\) in \(IoU_{3D}\)), verifying that dense heatmap supervision and soft-argmax provide indispensable gradient flow.
- 57x parameter compression with 22.8% lower reprojection error: MoCA3D utilizes only 19.0M trainable parameters compared to DetAny3D's 1.08B (a 57x reduction), achieving fast 0.14 s/example inference and improving overall image-plane corner \(PAG_{uv}\) by 22.8% across Omni3D.
- External depth models introduce redundancy: Injecting pretrained depth features from Depth Anything v3 yields slight degradation across all metrics, showing that DINOv3 features conditioned on geometric box priors already provide sufficient depth cues without external noise.
Highlights & Insights¶
- Reversed cross-attention paradigm: Flipping the standard transformer query direction—allowing dense spatial image tokens to act as queries that attend to compact box condition keys/values—bridges global semantic scene representation with local instance geometry.
- Heatmap confidence as natural depth weighting: Leveraging predicted corner activation maps as spatial confidence masks during depth supervision filters out ambiguous background pixels and unifies coordinate and depth estimation.
- Direct enablement of intrinsics-free downstream generation: In real-world driving scenes where camera parameters are absent, MoCA3D extracts high-fidelity projected 3D cuboids that can directly condition diffusion pipelines (e.g., PerLDiff) for controllable weather and lighting synthesis.
Limitations & Future Work¶
- Visibility assumption on all eight corners: MoCA3D assumes that all 8 box corners lie within the image plane, exhibiting reduced efficacy on heavily truncated objects at image boundaries. Future work may explore truncation-aware augmentations or directional ray constraints.
- Orientation ambiguity in symmetric objects: Relying solely on class-agnostic corner coordinates can lead to pose and orientation ambiguity when dealing with geometrically symmetric objects.
- Two-stage dependency on 2D boxes: While resilient to bounding box jitter, the model remains a box-conditioned geometry estimator rather than an end-to-end, unprompted 3D object detector.
Related Work & Insights¶
- vs Cube R-CNN: Cube R-CNN regresses 3D parameters in a virtual camera coordinate system and strictly relies on known intrinsics to back-project into 2D; MoCA3D operates natively in the image plane without requiring camera calibration, yielding vastly superior \(PAG_{uv}\) reprojection accuracy.
- vs DetAny3D: DetAny3D utilizes massive foundation models (1.08B parameters) for open-vocabulary lifting; MoCA3D achieves a 22.8% improvement in image-plane corner accuracy with 57x fewer parameters (19.0M) and faster inference.
- vs RTM3D / CenterNet3D: While keypoint-based detectors infer 3D boxes via sparse center point regression and geometric constraints, MoCA3D predicts dense heatmaps and depth maps with continuous soft-argmax, ensuring more stable optimization.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ Reformulates monocular 3D reasoning directly into image-plane dense prediction, bypassing intrinsics requirements with elegant transformer conditioning.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across 6 major indoor/outdoor benchmarks with newly defined PAG metrics, extensive ablations, and real-world diffusion generation downstream tasks.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear methodology, concise mathematical exposition, and high-quality comparative visualizations.
- Value: ⭐⭐⭐⭐⭐ Offers a lightweight, highly accurate solution for intrinsics-free 3D perception and geometry-guided generative modeling.