Boxer: Robust Lifting of Open-World 2D Bounding Boxes to 3D¶
Conference: ECCV 2026
Paper: ECCV Official Page
Area: 3D Vision
Keywords: Open-world 3D detection, 2D box lifting, sparse depth, uncertainty regression, multi-view fusion
TL;DR¶
Boxer separates open-vocabulary 2D detection from 3D geometry estimation, lifts proposals with BoxerNet using optional sparse or dense depth, and fuses predictions into static scene-level boxes; with ground-truth 2D boxes and dense depth on CA-1M, it improves per-frame class-agnostic mAP from CuTR's 0.250 to 0.412.
Background & Motivation¶
Open-vocabulary 2D detectors can already locate many long-tail objects from text, but augmented reality and robotics also need their metric positions, dimensions, and orientations. Whereas ordinary images support straightforward 2D annotation, metric 3D boxes require depth sensors, calibration, or multi-view reconstruction. Their training data therefore lag behind in scale and category coverage. Asking one 3D network to learn both object recognition and spatial estimation spends scarce 3D supervision on two different problems.
Geometric inputs are also heterogeneous. RGB-D datasets provide dense depth, glasses-based simultaneous localization and mapping (SLAM) often provides only sparse points, and cameras may use pinhole or fisheye projections. CuTR's dedicated depth encoder expects dense inputs, while monocular approaches such as 3D-MOOD predict depth internally and cannot directly exploit available sensor geometry in the same way. A practical system needs an interface that retains strong 2D recognition and benefits from whatever geometric evidence is available.
The paper delegates semantic coverage to existing detectors, focuses BoxerNet on the geometry of supplied 2D boxes, and consolidates per-frame estimates into global static objects. Core Idea: learn the mapping from 2D boxes to metric 3D boxes through a unified image, ray, and patch-depth representation, then use separate geometric uncertainty and cross-view consistency to handle ambiguous observations and duplicate detections.
Method¶
Overall Architecture¶
Inputs are a calibrated, posed image sequence, text queries, and optional sparse point clouds or dense depth. An open-vocabulary detector first produces 2D boxes and confidence scores. For each box, BoxerNet predicts a 3D center, three dimensions, and one yaw angle in a gravity-aligned camera frame, giving 7 degrees of freedom, together with geometric uncertainty.
The complete Boxer system additionally transforms per-frame predictions into world coordinates using camera poses, clusters them by geometric overlap and text-semantic compatibility, and produces de-duplicated static 3D boxes. BoxerNet is therefore the single-frame lifting network, whereas Boxer includes 2D detection and non-learned multi-view fusion. Their outputs and runtime costs should not be conflated.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Calibrated images and text<br/>Optional depth"] --> B["Open-vocabulary 2D interface"]
A --> C["Unified geometric conditioning"]
B --> D["Box-conditioned lifting<br/>and uncertainty regression"]
C --> D
D --> E["Semantic-geometric<br/>multi-view fusion"]
E --> F["Global static 3D boxes"]
Key Designs¶
1. Open-vocabulary 2D interface: delegate semantic retrieval to existing detectors
Text queries are passed to a detector such as DETIC, OWLv2, or SAM3. One prompt can return multiple 2D boxes, each retaining its 2D confidence. BoxerNet does not reclassify these regions; it learns how the corresponding objects are situated in 3D. Grayscale images can enter the same interface through channel replication. Adding a category recognized by the 2D model therefore does not require adding a corresponding classification output to the 3D network.
Open-world coverage primarily comes from this frontend, not from the geometric network having observed every possible shape. Missed objects provide no proposals to lift, and incorrectly localized proposals also propagate downstream. The paper uses ground-truth 2D boxes, denoted GT2D, to separate this frontend bottleneck from lifting quality. These oracle results should not be presented as the performance of the complete detection system.
2. Unified geometric conditioning: share an interface across cameras and depth densities
A frozen DINOv3 backbone extracts image patch features. Sparse point clouds or depth points are projected into the image, and the mean depth is computed within each patch. Patches receiving no points are assigned -1. Depth may therefore cover most patches or only a few, without requiring an intermediate dense depth completion stage. The mean supplies local metric-scale evidence, while the missing-value marker identifies locations without measured geometry.
Depth alone does not specify viewing direction, so each patch also receives a 3D unit ray obtained by unprojecting its center with the camera model and expressing it in the gravity-aligned frame. Local camera translation is fixed at the origin and is not an additional channel. Appearance features, rays, and one depth scalar are concatenated per patch and processed by a shared self-attention encoder. This gives the model appearance, viewing direction, and any known distance together. Camera geometry is expressed through rays instead of requiring identical image formation across datasets; this supports pinhole and fisheye training, but does not remove the calibration requirement.
3. Box-conditioned lifting and uncertainty regression: distinguish 2D detection confidence from geometric reliability
The four coordinates of each 2D box are linearly embedded into a query token that cross-attends to all image patches. A query can access both object appearance and surrounding scene geometry. There is no attention between box tokens, so lifting a particular box does not depend on proposal ordering. Conversely, this stage does not determine whether several proposals describe the same physical object.
The decoded box representation feeds two prediction heads, each a two-layer MLP with ReLU and hidden dimension 128. One regresses the 3D center, positive dimensions, and yaw about gravity; the other predicts aleatoric uncertainty reflecting observation ambiguity from occlusion, missing depth, and single-view evidence. A confident 2D detection need not have a confident 3D size or distance estimate, motivating a separate geometric signal. The prose states that the final score averages 2D and 3D confidence for ranking, filtering, and precision-recall curves. Equation (2) is corrupted in the cached text, and the uncertainty-to-confidence mapping is not clearly specified there, so no exact formula is reconstructed here.
4. Semantic-geometric multi-view fusion: associate objects before resolving box symmetries
After transformation into world coordinates, predictions can be connected only if their 3D IoU passes a threshold and their prompt-text embeddings are sufficiently similar. Detections become graph nodes, and connected components define object clusters across frames. The geometric gate prevents distant instances of the same category from merging, while semantic compatibility reduces merging of overlapping predictions with different meanings. Thus, although evaluation ignores class labels, fusion still uses category text.
Within each cluster, positions, dimensions, and yaw are fused with confidence weights. The algorithm first resolves the 90-degree rotational representation ambiguity of gravity-aligned cuboids and then uses a circular mean for yaw instead of directly averaging angles. Otherwise, equivalent boxes represented with different axis conventions can distort the result. Finally, 3D non-maximum suppression removes residual duplicates. This module is non-learned and assumes a static world; exact thresholds and implementation choices omitted from the main text cannot be inferred from the conceptual description.
A Worked Example¶
Consider a wearer walking through a kitchen and querying the same stationary object with "spice jar." This is an illustrative walkthrough, not an additional experiment. OWLv2 supplies proposals in different frames. SLAM points may supply depth for only some patches, with -1 marking the rest. BoxerNet uses appearance, viewing rays, and available scale evidence to independently predict a 7-degree-of-freedom box and uncertainty in each frame.
As the viewpoint changes, the object's image position moves and occlusion may destabilize individual predictions. Camera poses first place these predictions in the same world frame, after which geometric overlap and query semantics determine compatibility. Compatible predictions form a cluster for orientation disambiguation and weighted fusion. Another object at a different location should not merge merely because it shares the same prompt. Picking up and moving the object breaks the static-fusion assumption, illustrating why this system is not a dynamic object tracker.
Loss & Training¶
Training supervises geometry with a Chamfer loss between predicted and ground-truth box corners and includes a predicted aleatoric uncertainty term to represent observational ambiguity. Equation (3) in the cache has visible character loss. The reliable information is the combination of Chamfer supervision and uncertainty modeling; the full weighting, regularization, and parameterization cannot be recovered confidently, so a guessed equation is not provided.
Training combines internal Aria and Quest data, public NymeriaPlus and CA-1M, and the SUN-RGBD and ScanNet portions of Omni3D. The authors report approximately 1.22M unique 3D boxes. Unique objects must be distinguished from video observations: recording one static object across many frames does not add many independent objects. Training focuses on indoor data and excludes ARKit because of overlap with CA-1M, Objectron because of its single-object focus, and synthetic Hypersim.
Projecting static 3D annotations into images can incorrectly make objects appear visible through walls, so training also checks visibility. A box must contain at least 2 visible depth points, and 80% of points sampled along its edges must lie in the valid visible image region. Augmentation covers photometric properties, cameras, depth, and 2D boxes; the main-text cache does not expand the exact augmentation distributions.
Training takes approximately two weeks on 16 H100 GPUs using AdamW and a cosine learning-rate schedule from 1e-4 to 1e-5. Excluding frozen DINO weights, the network has about 25M trainable parameters. With supplied 2D boxes, an RTX 4090 forward pass takes approximately 20 ms at 960 × 960 resolution with bfloat16. This excludes 2D detection and full-sequence fusion.
Key Experimental Results¶
Main Results¶
The metric is class-agnostic 3D mAP: all objects are treated as one "Anything" class, and AP is averaged across IoU thresholds \(0.05, 0.10, \ldots, 0.50\). Rather than limiting predictions to 100 per image, the evaluation retains proposals using a relatively low detection threshold. Per-frame and fused per-scene results are reported separately.
The following selection from Table 2 reports per-frame mAP. GT2D denotes ground-truth 2D boxes. Geometric input means sparse SLAM points on NymeriaPlus but dense depth on CA-1M, so these should not be treated as identical RGB-D settings.
| 2D input and lifting model | Geometric input | NymeriaPlus | CA-1M |
|---|---|---|---|
| OWLv2 + CuTR | None | 0.005 | 0.064 |
| OWLv2 + BoxerNet | None | 0.061 | 0.081 |
| GT2D + CuTR | None | 0.010 | 0.119 |
| GT2D + BoxerNet | None | 0.296 | 0.126 |
| OWLv2 + CuTR | Dense depth | Sparse input unsupported | 0.178 |
| OWLv2 + BoxerNet | Sparse points / dense depth | 0.297 | 0.204 |
| GT2D + CuTR | Dense depth | Sparse input unsupported | 0.250 |
| GT2D + BoxerNet | Sparse points / dense depth | 0.532 | 0.412 |
The abstract's NymeriaPlus comparison of "0.532 vs. 0.010" is not a matched RGB-only comparison: BoxerNet uses sparse depth, whereas CuTR uses RGB. The corresponding RGB-only GT2D comparison is 0.296 versus 0.010. On CA-1M, the GT2D-plus-dense-depth comparison of 0.412 versus 0.250 matches input modalities and gives an absolute difference of 0.162, although training data still differ.
Test coverage is restricted. ADT uses one sequence from one apartment and excludes dynamic objects. CA-1M uses only the first 10 validation sequences, excludes objects with any dimension exceeding 3 m, and expands thin dimensions to at least 5 cm for both predictions and ground truth. These are not evaluations over the complete unfiltered datasets.
Ablation Study¶
The following results come from Table 4 and use GT2D for per-frame mAP. The NymeriaPlus full-model value of 0.518 is retained rather than replaced by Table 2's 0.532; the main text does not explain the numerical difference between these settings.
| Config | NymeriaPlus | CA-1M |
|---|---|---|
| Full BoxerNet | 0.518 | 0.412 |
| Median rather than mean depth | 0.505 | 0.402 |
| Without data augmentation | 0.497 | 0.400 |
| Without aleatoric uncertainty head | 0.485 | 0.401 |
| Resolution reduced to 480 × 480 | 0.466 | 0.395 |
| Public training data only | 0.463 | 0.376 |
| Without depth | 0.279 | 0.126 |
| CA-1M training only | 0.002 | 0.357 |
Depth and training-domain coverage matter more than an individual regression component. On NymeriaPlus, removing depth costs 0.239, training only on CA-1M costs 0.516, and removing the uncertainty head costs 0.033. Public-only training remains competitive, but the full model includes internal data, so its entire advantage over baselines cannot be attributed to architecture.
Key Findings¶
- In Table 3, CA-1M per-scene mAP with GT2D and depth is 0.434 for BoxerNet and 0.305 for CuTR; with OWLv2 and depth, the values are 0.204 and 0.178. The oracle-versus-detector gap remains substantial.
- Section 5.5 compares fusion on the same OWLv2 + BoxerNet outputs on CA-1M. The proposed fusion reaches mAP 0.204 versus BoxFusion's 0.176; [email protected] is 0.547 versus 0.321, and [email protected] is 0.421 versus 0.309. The authors caution that BoxFusion was not tuned for this output distribution.
- Fusion does not necessarily increase a directly comparable scalar across all datasets. NymeriaPlus GT2D-plus-geometry mAP is 0.532 per frame in Table 2 and 0.400 per scene in Table 3. Evaluation units and de-duplicated proposal sets differ, so their difference cannot simply be interpreted as a fusion penalty.
- Throughput depends on the 2D frontend. On an RTX 4090, the paper reports approximately 35 ms for DETIC and 120 ms for OWLv2, while SAM3 takes approximately 45 s per image with 1000+ prompts. The 20 ms figure describes only BoxerNet lifting.
Highlights & Insights¶
- A flexible geometric interface can matter more than a more elaborate depth network. Patch means and explicit missing values turn existing sparse SLAM geometry into metric-scale conditioning, and the ablation shows much greater value from depth itself than from changing its aggregation statistic.
- Detection confidence and geometric uncertainty serve distinct roles. When recognition is reliable but 3D placement is ambiguous, a separate geometric head supplies information for ranking and fusion rather than equating semantic confidence with spatial reliability.
- Cuboid fusion must respect representation symmetries. Resolving 90-degree-equivalent representations before averaging dimensions and orientations is a concrete geometric technique transferable to other multi-view box-fusion systems.
Limitations & Future Work¶
- The authors explicitly target static objects, so moving handheld objects violate the fusion assumptions. Highly non-cuboidal objects such as wires and vines are also poorly represented by a single 3D box.
- Lifting requires camera calibration and gravity, while complete scene fusion additionally requires poses. Estimating missing calibration or gravity with existing models is a future direction; every tested dataset supplies full poses.
- Open-world semantics come from the 2D frontend, so the system inherits DETIC and OWLv2 errors in egocentric views. Class-agnostic mAP does not establish the correctness of final semantic labels.
- Approximately two weeks of multi-GPU training and internal datasets limit exact reproducibility. The public-data ablation helps quantify this dependence but does not fully separate backbone, training scale, and architecture contributions to baseline gaps.
- Dynamic instance association, robustness to realistic calibration errors, and uncertainty calibration deserve further evaluation. These are directions inferred in this note, not experiments completed by the paper.
Related Work & Insights¶
- vs CuTR: Boxer retains Transformer-based geometric estimation but externalizes 2D detection, replaces the dedicated dense-depth encoder with patch-depth conditioning, and predicts geometric uncertainty separately. Training-domain diversity is also an important source of the performance gap.
- vs 3D-MOOD / DetAny3D / OVMono3D: These methods likewise connect 2D semantics with 3D localization. Boxer emphasizes compatibility with available metric depth and static multi-view fusion rather than restricting the system to monocular geometric inference.
- vs ConceptGraphs / EgoLifter: These approaches lift 2D segmentation into object-centric scene representations. Boxer directly estimates compact 3D boxes and obtains cross-frame consistency through post-processing; box localization and complete instance geometry are different output tasks.
- vs BoxFusion: Both use non-learned fusion. Boxer uses compatibility-graph clustering and geometric averaging, while BoxFusion uses stochastic fitting. The comparison depends on the input-box distribution and does not establish universal superiority.
- The paper's project page provides an entry point for models and code. This records the link in the paper without independently checking its release status online.
Rating¶
- Novelty: 4/5. The contribution lies in flexible geometric conditioning, uncertainty, and system integration rather than a new Transformer paradigm.
- Experimental Thoroughness: 4/5. Multiple device domains, input modalities, and per-frame/per-scene evaluations are covered, but test subsets and internal data constrain generalization of the conclusions.
- Writing Quality: 4/5. The task decomposition and ablations are clear, while cross-table differences and modality boundaries in the abstract require careful reading.
- Value: 4/5. A practical route from existing 2D detectors to metric scene localization, with static-world and calibration assumptions remaining deployment prerequisites.