XYZ-IBD: Benchmarking Robust 6D Object Pose Estimation under Real-World Industrial Complexity

Conference: ECCV2026
Paper: ECCV Paper
Project: https://xyz-ibd.github.io
Authors: Junwen Huang, Jiaqi Hu, Peter KT Yu, Slobodan Ilic, Martin Sundermeyer, Benjamin Busam
Affiliations: Technical University of Munich; Munich Center for Machine Learning (MCML); XYZ Robotics; ROBOX; Google
Area: 3D Vision
Keywords: 6D pose estimation, industrial bin-picking, reflective objects, multi-view annotation, generalization evaluation

TL;DR

XYZ-IBD combines multi-sensor real-world bin data, coating-assisted geometric annotation, and noise-matched simulation to benchmark industrial 6D pose estimation across 15 parts, exposing a substantial performance gap: SAM6D achieves 0.578 6D AP on XYZ-IBD versus its BOP-Core 5 average of 0.704.

Background & Motivation

Industrial picking requires a part's three-dimensional position and rotation in camera coordinates, not merely a detection box. Texture, color, and semantic cues often distinguish household objects, whereas industrial metal parts tend to be textureless, reflective, and geometrically symmetric. Dense piles of identical parts further obscure their boundaries. Correct category recognition therefore does not guarantee that the visible surface determines the full pose.

Existing industrial datasets capture some of these challenges but do not fully combine part diversity, stochastic dense stacking, strong specular reflection, and precise ground truth. Reflection also corrupts the depth sensors used to produce annotations, not just the evaluated model's input. Fitting CAD models directly to distorted point clouds can make annotation errors obscure algorithmic differences. Repeatable robotic positioning, multiple sensors, and temporary anti-reflection coating enable a separation between difficult native observations and better annotation geometry.

Core Idea: acquire difficult native observations separately from reliable annotation geometry, then repeat the annotation process in noise-matched simulation to preserve industrial visual complexity while explicitly estimating label error.

Method

Overall Architecture

The contribution is a dataset, annotation process, and evaluation protocol, not a new pose network. Inputs are industrial CAD models and multi-view RGB, grayscale, and depth observations of bins. Outputs include instance masks, depth and 6D pose annotations, and detection/pose benchmark results for seen and unseen objects.

The workflow performs calibrated dual-pass capture, produces hybrid pose annotation from coated-surface depth, estimates annotation error through noise-matched validation, and conducts setting-controlled evaluation. The diagram describes data production and validation, not an inference network that supplies coating-derived ground truth to the evaluated model.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Industrial CAD and bins"] --> B["Calibrated Dual-Pass Capture"]
    B --> C["Hybrid Pose Annotation"]
    C --> D["Noise-Matched Validation"]
    D --> E["Setting-Controlled Evaluation"]
    E --> F["Label quality and baseline AP"]

Key Designs

1. Calibrated Dual-Pass Capture: retain reflections in observations, obtain reliable annotation geometry

The dataset contains 15 industrial parts with diameters of 54-300 mm, arranged into 5 randomly stacked scenes per part, totaling 75 real scenes. Scenes contain 11-60 instances, with a reported average of approximately 22. A FANUC M10iD/8L robot rigidly carries a RealSense D415, an XYZ Robotic AL-M DLP structured-light camera, and a Photoneo PhoXi M scanner at working distances of 600-1000 mm. These provide complementary color/grayscale and depth observations rather than interchangeable depth quality.

The system samples 50 viewpoints on a sphere centered at the bin, spanning elevations of 45-90 degrees. It first captures 4 precision calibration spheres, applies undistortion and hand-eye calibration, and refines their point-cloud alignment using ICP to relate the reference viewpoint to the remaining 49 viewpoints. The main text reports average calibration RMSE of approximately 0.248 mm. These transformations support both fusion and label propagation; nominal robot poses alone are not used as a substitute.

During scene capture, temporary Acksys SP-102 anti-reflection coating first enables more reliable depth measurements. After 15 minutes of evaporation at 25 degrees C +/-1 degree C, the robot revisits the same viewpoints to record the native appearance. Robot repeatability is +/-0.06 mm. Coating thus assists annotation while the dataset still records the difficult uncoated input; alignment depends on the scene remaining stationary between passes.

2. Hybrid Pose Annotation: annotate the fused scene once and propagate to all viewpoints

Coating-enhanced depth is fused into a common scene point cloud. Annotators coarsely align each CAD instance through a constrained GUI with translation and rotation increments of +/-1 mm and +/-1 degree. Multi-scale ICP then performs global adjustment on downsampled points and refines alignment with full-resolution data. Human intervention establishes correspondences among similar instances and symmetric geometry, while ICP reduces local geometric misalignment.

The calibrated transformations propagate final poses to 50 viewpoints without separate manual annotation of every image. The XYZ DLP is the primary camera, and calibrated inter-camera transformations project labels to the other two sensors. Multi-view and multi-sensor labels therefore share scene-level alignment rather than independent human estimates. This improves consistency but also propagates initial alignment or calibration biases across images. The approximately 273k annotated instances are not 273k distinct physical parts or independent scenes.

3. Noise-Matched Validation: assess the annotation chain rather than quote sensor specifications

Direct physical ground truth is difficult to obtain for real piles, so the authors reproduce calibration spheres, camera parameters, and 50 viewpoints in simulation. They inject Gaussian noise into rendered depth and repeat ICP calibration. The matching criterion is agreement with real calibration RMSE, not visual similarity between synthetic and real depth maps: a depth-noise standard deviation of 0.26 mm produces RMSE of 0.248 mm.

BlenderProc then generates cluttered bins by dropping CAD parts under physics simulation, discarding parts that fall outside the bin, and applying the matched noise. Annotators repeat multi-view fusion, manual adjustment, and multi-scale ICP on scenes with known ground-truth poses. Hungarian assignment on 3D centroid distances establishes one-to-one matches between annotations and simulated ground truth before positional and angular errors are computed. Evaluation covers 3 scenes per object and up to 60 samples per scene, yielding mean positional error of 0.999 mm with standard deviation 0.12 mm, and mean angular error of 0.432 degrees with standard deviation 0.08 degrees.

This validates cumulative error in a simulated process matched to real calibration error, not external metrology of every real annotation. Gaussian noise makes the process measurable but may not capture systematic reflection artifacts or coating-induced changes. Sub-millimeter accuracy is therefore a conditional validation result, not a per-instance error bound.

4. Setting-Controlled Evaluation: distinguish object adaptation, generalization, and detector effects

For each part, synthetic training data uses 120 free-fall scene simulations with 10-60 instances per scene and 25 rendered frames per scene. Lighting and materials vary, while camera intrinsics match the XYZ structured-light sensor. This yields approximately 3,000 frames per object and 45,000 RGB-D samples overall. Real data is split by scene into 15 validation scenes and 60 test scenes, corresponding to 2,250 and 9,000 frames. Neighboring views of one scene should not be treated as independent train/test examples.

Seen-object methods train on synthetic data for the target objects and evaluate on real scenes. Unseen-object methods use externally pretrained models without additional XYZ-IBD fine-tuning. Unseen does not mean unavailable CAD: this is a model-based task, and object models remain available. Unseen 6D baselines share SAM-6D segmentation results, whereas seen 6D baselines use YOLOX detections. This controls the frontend within groups, but cross-group differences still include frontend and pretraining-data effects.

The 2D protocol computes AP over instance-mask IoU thresholds from 0.50 to 0.95 in increments of 0.05 and averages across object categories. The 6D protocol uses Maximum Symmetry-Aware Surface Distance (MSSD) and Maximum Symmetry-Aware Projection Distance (MSPD), capturing spatial surface deviation and projected pixel deviation, respectively, and aggregates over their threshold sets. Accounting for symmetry prevents geometrically equivalent rotations from being penalized as incorrect. Final aggregation is:

\[ \mathrm{AP}=\frac{1}{2|\mathcal O|}\sum_{o\in\mathcal O}\left(\mathrm{AP}_{\mathrm{MSSD},o}+\mathrm{AP}_{\mathrm{MSPD},o}\right). \]

Here \(\mathcal O\) denotes the object set, and each component is the object's AP aggregated over evaluation thresholds for that error criterion. Reported AP is dimensionless on a 0-1 scale, not a millimeter error or the AR often used in BOP. Reproducing thresholds and symmetry transformations requires the BOP protocol specified by the paper.

Loss & Training

There is no new loss function; training details support baseline reproducibility. Appendix B trains YOLOX on one NVIDIA RTX 4090 with batch size 24 for 30 epochs, applying augmentation in the first 15 epochs and taking approximately 18 hours. GDRNet uses the same GPU type and batch size 24 for 10 epochs; the paper separately reports approximately 490,000 steps and 24 hours. These training-length descriptions require checking against the original configuration rather than silently converting between them.

SurfEmb trains per object with batch size 24 for 500,000 steps, taking approximately 20 hours on the same GPU type. Appendix A also defines a multi-view protocol: select 5 spatially spread viewpoints per scene, with 1 target and 4 references, and provide relative camera poses. Single-view methods use only the target. Table 4 is not a single-view versus multi-view ablation and does not establish the gain from multi-view input.

Key Experimental Results

Main Results

The following extracts Table 4. BOP-Core 5 averages AP across YCB-V, T-LESS, LM-O, IC-BIN, and TUD-L. Their objects, scenes, and training data differ, so cross-dataset gaps provide evidence of difficulty rather than a controlled experiment. Higher AP is better throughout.

Method	Task	Object setting	BOP-Core 5 AP	XYZ-IBD AP
YOLOX	2D	Seen	0.798	0.774
CNOS(SAM)	2D	Unseen	0.361	0.275
SAM-6D(SAM)	2D	Unseen	0.449	0.296
NIDS-Net	2D	Unseen	0.494	0.258
GDRN	6D	Seen	0.827	0.266
SurfEmb	6D	Seen	0.758	0.247
MatchU	6D	Unseen	0.705	0.529
FoundationPose	6D	Unseen	0.734	0.564
SAM6D	6D	Unseen	0.704	0.578

Ablation Study

The paper introduces no new network and reports no network-module ablation. Instead, Table 3 supplies a noise-sensitivity analysis. Noise is injected into simulated depth; RMSE measures viewpoint calibration using sphere point clouds, not final object-pose error.

Gaussian depth-noise standard deviation (mm)	Mean calibration RMSE (mm)	Note
0.00	0.199	Residual error without injected noise
0.10	0.209	Low-noise setting
0.20	0.233	Approaching real calibration error
0.26	0.248	Selected noise-matching point
0.30	0.259	Exceeds the matching target

Key Findings

• SAM-6D(SAM) 2D AP changes from 0.449 to 0.296, an absolute difference of 0.153 and a relative decrease of approximately 34.1% normalized by the former. This is not a decrease of 34.1 percentage points.
• SAM6D 6D AP changes from 0.704 to 0.578, a difference of 0.126. It remains the strongest XYZ-IBD 6D baseline in Table 4, but the task is far from solved.
• YOLOX reaches 0.774 2D AP, whereas GDRN reaches only 0.266 6D AP, illustrating that detection does not ensure precise pose. These measure different tasks and cannot be subtracted to attribute a loss to one module.

Highlights & Insights

• Separating native appearance for model input from reliable geometry for annotation avoids simplifying the test data while improving labels. The design is transferable to other reflective, difficult-to-measure perception datasets.
• Matching simulated noise to real calibration RMSE is more informative about the annotation chain than citing sensor resolution alone. The residual under zero injected noise also shows that noise is not the only source of error.
• Repeated instances of the same part require instance separation and geometric correspondence beyond category semantics. Sharing the frontend among some baselines helps reveal how those challenges affect pose estimation.

Limitations & Future Work

• There are only 15 parts and 75 real scenes; a large multi-frame annotation count does not imply as many independent scenes. The authors propose expanding object categories and operating conditions, which is important for evaluating broader generalization.
• Sub-millimeter accuracy comes from noise-matched simulation. The mean of 0.999 mm lies close to 1 mm, with standard deviation 0.12 mm; it does not establish that every real instance is below 1 mm. External metrology and real failure-case auditing would strengthen the evidence.
• Table 1 reports 22.5k frames, while validation and test counts in Section 4.1 sum to 11,250; the text does not explicitly reconcile counting conventions such as dual-pass capture. Table 2 gives total calibration RMSE of 0.245 mm versus 0.248 mm in the text and Table 3. Positional error is 0.99 mm in tables and 0.999 mm in the text; these source-specific values are retained rather than silently merged.
• There are no systematic controlled ablations of coating, sensor choice, occlusion, or detection frontend, and no closed-loop robotic picking success rate. Current results establish a difficult perception benchmark, not demonstrated downstream manipulation gains.

Related Work & Insights

• vs T-LESS / ITODD: These advance textureless and industrial object pose evaluation. XYZ-IBD emphasizes the combination of random dense stacking, repeated instances, and strong reflection, with dedicated annotation-error validation.
• vs ROBI / IPD: ROBI also targets reflective industrial parts, while IPD offers multiple sensors. XYZ-IBD emphasizes size/geometry coverage and bin instance density, without establishing superiority on every dataset dimension.
• vs SAM-6D / FoundationPose / MatchU: These are evaluated general-purpose pose methods, not new XYZ-IBD components. Their results motivate studying external pretraining and geometric matching but do not isolate which design causes better generalization.

Rating

• Novelty: 4/5. The contribution combines industrial data conditions and error validation rather than a new pose network.
• Experimental Thoroughness: 4/5. Seen/unseen and 2D/6D tasks are covered, but real metrology and controlled ablations remain missing.
• Writing Quality: 3/5. Acquisition is concrete, but data counts, error precision, and training-length conventions need clarification.
• Value: 4/5. A targeted benchmark and reusable annotation process for densely stacked reflective industrial parts.