Boosting 6D Object Pose Estimation via Monocular Depth Cues¶

Conference: ECCV 2026
Paper: Official paper page
PDF: Full paper
Area: 3D Vision
Keywords: 6D object pose estimation, monocular depth, scale correction, geometric consistency, recurrent pose refinement

TL;DR¶

MDC-Net couples monocular-depth calibration, geometric inlier selection, and recurrent pose refinement, achieving 66.9% mean AR across seven BOP datasets with one pose hypothesis and no real depth at inference, although several auxiliary results and training descriptions are inconsistent.

Background & Motivation¶

Knowing an object's CAD model does not make its precise position and orientation unambiguous in a single RGB image. Existing refiners commonly render the object at a coarse pose, match that rendering to the observation, and update the pose using optical flow or feature differences. This approach becomes fragile on textureless surfaces, reflections, and occlusions: an appearance match need not imply a correct 3D correspondence. The task here is refinement after initialization, not a complete localization system that eliminates the object model, camera intrinsics, or initializer.

Monocular depth offers dense geometry without a depth sensor, but its output is not automatically suitable for metric pose computation. Relative predictions have scale and shift ambiguity, while even metric-depth estimators retain local shape errors and unreliable regions. Using such depth as a fixed input can repeatedly inject the same errors into pose updates; applying one global scale cannot repair spatially varying mistakes. The challenge is therefore not simply adding a depth channel, but deciding which depth estimates and correspondences are trustworthy and how subsequent pose changes should revise them.

The authors use the current pose and known object geometry to check and update the depth representation. A scale-refinement module corrects global and local biases, an inlier-guided module suppresses geometrically inconsistent matches, and a recurrent network updates the pose from the corrected evidence. Core idea: let pose constrain the reliability of depth, then use corrected depth to constrain the next pose, improving both through a shared feedback loop.

Method¶

Overall Architecture¶

The inputs are one RGB image, the target CAD model, camera intrinsics, and an initial 6D pose; the output is a refined rotation and translation, with corrected monocular depth as an intermediate result. The system crops the target using its initial pose and obtains depth cues from a pretrained monocular estimator; the runtime configuration uses MoGe-2. At each iteration, the current pose supplies rendered RGB, depth, object masks, and visible surface points, while rendered and observed features form a 4D cost volume for dense matching. Scale Refinement calibrates the depth, Inlier-Guided Depth Refinement suppresses inconsistent 3D evidence, and Recurrent Pose Update predicts a residual pose from the resulting features. The new pose changes the rendering and pose-induced flow used in the next iteration, rather than merely applying depth cleanup after pose estimation is finished.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["RGB, CAD, intrinsics<br/>and initial pose"] --> B["Rendering, depth prediction<br/>and dense matching"]
    B --> C["Scale Refinement"]
    C --> D["Inlier-Guided<br/>Depth Refinement"]
    D --> E["Recurrent Pose Update"]
    E -->|Updated pose and flow| B
    E --> F["Refined 6D pose"]

Key Designs¶

1. Scale Refinement: make predicted depth usable for metric geometry

The Scale Refinement Module (SRM) does more than predict one image-wide multiplier. It regresses a global scale and shift from image-level features, predicts a spatial scale field through a lightweight convolutional branch, and adds a local depth residual. The global correction addresses systematic bias, while local terms allow different regions to receive different adjustments instead of forcing one scale to accommodate all depth errors. Equation (1) is sufficiently readable in the cached source to retain its central relationship:

\[ D_{\mathrm{abs}}(x,y)=S(x,y)\bigl(\alpha D_{\mathrm{rel}}(x,y)+\beta\bigr)+\Delta D(x,y). \]

Here, \(\alpha\) and \(\beta\) are the global scale and shift, \(S(x,y)\) is the spatially varying scale field, and \(\Delta D(x,y)\) is the local residual. The resulting metric depth is a predicted proxy for refinement, not a sensor measurement or an estimate with guaranteed metric accuracy. Training also constrains depth gradients, surface normals, and reprojection relationships, encouraging calibration to preserve boundaries and 3D structure rather than just change the numerical range. SRM is applied to every tested depth source, so with a metric-depth model it can still correct residual local biases.

2. Inlier-Guided Depth Refinement: reject matches that violate multi-point rigidity

Inlier-Guided Depth Refinement (IGDR) addresses matches that look plausible in the images but are inconsistent in 3D. Dense flow establishes correspondences from the rendering to the observation, and depth plus camera intrinsics lift their locations into 3D. The consistency test exploits the preservation of inter-point distances under rigid motion: two correspondences should not both be trusted when their point-pair geometry disagrees substantially across the two sides. A compatibility graph represents correspondences as nodes and connects pairs that pass a geometric threshold. Heuristic clique sampling and consistency scoring identify a mutually consistent subset instead of allowing every dense match to influence pose computation equally.

The module then combines compatible-neighborhood information, feature similarity, and flow confidence to weight matches and fuse depth estimates. This reduces the influence of unreliable geometry on the current update and passes improved depth into subsequent iterations, limiting repeated error propagation. The implementation discussion selects 2,048 keypoints and uses a greedy approximation for clique search; constructing the graph still has quadratic complexity and relies on GPU parallelism. Equations (4)–(9) are visibly corrupted in the cached extraction, so this note does not reconstruct exact compatibility, weighting, or fusion formulas, particularly the unspecified interpolation from sparse inliers to dense depth. There is also an implementation ambiguity: the overview supplies rendered depth, whereas Section 3.3 describes monocular prediction for both rendered and real images without clearly explaining how these depth branches are reconciled.

3. Recurrent Pose Update: make geometric correction change the next matching problem

A GRU-based regressor takes current matching features, corrected depth features, and the previous hidden state, predicts a residual pose, and applies it to the current estimate. The 4D cost volume supports dense rendered-to-observed matching, while pose-induced flow feeds the updated geometric state into the next iteration. SRM and IGDR are consequently not isolated preprocessing steps: they improve geometric evidence, and pose changes alter the rendering, correspondences, and subsequent inlier decisions. This distinguishes the framework from treating monocular depth as a fixed matching cue, even when the underlying depth estimator remains frozen.

The known CAD model provides shape and metric reference information, so RGB-only means no real depth input at inference, not an absence of geometric priors. The runtime breakdown uses three pose-regression iterations and three IGDR iterations, but the paper does not provide a sufficiently detailed iteration-versus-accuracy curve to establish that three is optimal. Feedback also does not guarantee convergence: sufficiently poor initialization and depth can still produce misleading, mutually consistent matches.

A Worked Example¶

Consider a partially occluded object with a known CAD model and a reflective surface; this is a mechanism illustration, not an additional quantitative experiment. The initial rendering is slightly misaligned with the observation, while the depth estimator places the reflective region too far away. SRM first calibrates the overall depth range and permits local corrections, after which flow connects rendered surface locations to observed pixels. IGDR checks whether the matches jointly support consistent rigid geometry and downweights or rejects those that violate multi-point relations rather than trusting appearance confidence alone. The recurrent regressor updates the pose from the filtered evidence; a better-aligned rendering then changes the matching conditions for the next depth and pose correction. This explains the feedback mechanism without implying that the true depth of an occluded surface is necessarily recoverable.

Loss & Training¶

Section 3.4 resizes target crops to \(256\times256\) and describes rendering with training ground-truth poses to obtain flow and depth supervision, with exponentially weighted per-iteration flow, pose, and depth-matching losses. The SRM description includes scale, pixelwise depth, depth-gradient, and geometric-consistency terms, with the latter involving normals and reprojection. Cached equations (2) and (3) contain missing terms and damaged operators, preventing reliable identification of all reference quantities and weights; no reconstructed total loss is supplied here. Section 4.1 reports approximately 90,000 meshes and 3 million synthetic images drawn from ShapeNet-Objects, Google Scanned Objects, and Objaverse, using existing synthetic renderings. However, that section also says training does not use ground-truth pose supervision, conflicting with Section 3.4; no real depth at inference is supported, but fully pose-unsupervised training is not established.

Key Experimental Results¶

Main Results¶

The table selects the seven-dataset mean and three representative datasets from source Table 1, evaluated on the BOP unseen track using Average Recall (AR, %, higher is better). AR aggregates recalls based on VSD, MSSD, and MSPD error criteria; these are recall results, not raw geometric distances for which lower would be better. The seven datasets are LM-O, T-LESS, TUD-L, IC-BIN, ITODD, HB, and YCB-V.

Method	Seven-dataset mean AR ↑	LM-O AR ↑	T-LESS AR ↑	YCB-V AR ↑
MegaPose	62.3	62.0	48.5	76.4
SCFlow	60.5	62.9	57.9	70.7
Co-op, 5 hypotheses	65.7	65.5	64.8	68.9
MDC-Net, 1 hypothesis	66.9	66.8	65.2	74.1

MDC-Net exceeds Co-op, the strongest listed mean, by 1.2 percentage points, but their hypothesis budgets differ; this is not a controlled equal-budget ablation. The highest mean does not imply winning every dataset: MegaPose reaches 76.4 on YCB-V versus MDC-Net's 74.1. These comparisons are results reported in this paper, not a ranking against all subsequent methods.

Ablation Study¶

The following entries are AR (%, higher is better) from source Table 2 on LM-O and YCB-V, excluding component labels that conflict between the table and accompanying prose.

SRM	IGDR	LM-O AR ↑	YCB-V AR ↑
Off	Off	59.9	66.3
On	Off	63.0	70.8
Off	On	64.5	72.3
On	On	66.8	74.1

Against both modules disabled, SRM alone adds 3.1 points on LM-O, IGDR alone adds 4.6 points, and their joint gain is 6.9 points. The joint YCB-V gain is 7.8 points, supporting complementarity on both datasets without implying that the isolated gains add independently. Not every component metric improves monotonically: the full configuration does not have the highest YCB-V VSD recall in Table 2, so an across-all-metrics improvement claim would be inaccurate.

Key Findings¶

The clearest evidence supports using correctable geometric cues over disabling these modules, not equivalence between predicted and real depth.
Table 6 reports 164.3 ms of refinement on an RTX 3090, including 62.0 ms for MoGe-2 depth prediction and 63.6 ms for three IGDR iterations.
The runtime discussion reports 1,753.6 ms for FoundPose initialization and about 1.92 s for the complete pipeline; refinement-only latency must not be presented as end-to-end latency.
Depth-source substitutions, alternative initializers, and RGB-D comparisons are reported, but the inconsistencies below prevent using them to support additional precise gain claims.

Highlights & Insights¶

Depth becomes an updatable state. Improved pose constrains depth and corrected depth supports pose, unlike merely appending a fixed depth channel.
Scale error and correspondence error are treated separately. SRM adjusts continuous depth biases, while IGDR evaluates match reliability; the ablation shows remaining gains when only one issue is addressed.
Learned refinement retains explicit geometric checks. Multi-point compatibility adds evidence beyond appearance confidence, although it still requires sufficiently reliable depth scale and correspondences.

Limitations & Future Work¶

Author-proposed directions: integration with generative reconstruction and real-time deployment on resource-limited platforms are future work, not demonstrated capabilities.
Input requirements: known CAD geometry, camera intrinsics, and a coarse pose remain necessary; the results do not establish model-free localization or unconstrained absolute-scale recovery.
Compute: graph construction remains quadratic and initialization can dominate latency; describing refinement as real-time does not make the complete demonstrated system real-time.
Reproducibility: damaged extracted equations and incomplete explanations leave dense depth updates, precise training weights, and some depth-branch connections unresolved.
Data consistency: Table 3's dataset columns conflict with the MoGe-2 dataset assignments in Tables 1 and 2, and its accompanying “1–2%” statement does not explain the full tabulated spread.
Auxiliary comparisons: Section 4.3 swaps the LM-O VSD/MSSD labels relative to Table 2; Table 4 conflicts with the subsequent SCFlow example; Table 5's Avg. cannot be interpreted as the arithmetic mean of its two displayed datasets.
Reader assessment: the central AR comparison is verifiable, but these issues constrain stronger conclusions about depth-source robustness, initializer generalization, and supervision; revised tables and implementation details are needed.

vs SCFlow / GenFlow: all use render–match–refine loops; MDC-Net adds explicit scale calibration and multi-point geometric screening rather than relying only on implicit robustness in feature or flow networks.
vs OLD-Net / SingRef6D: these use object-level depth or depth-aware matching for different pose settings, whereas this paper centers on bidirectional pose–depth updates after initialization; their task assumptions are not interchangeable.
vs TurboReg / TurboClique: compatibility graphs and consistent subsets inspire the geometric filter, but its geometry comes from monocular predictions and image matches, making depth quality an important prerequisite.
vs RGB-D methods: removing measured depth reduces inference input requirements but changes the available information; these results do not establish that RGB-only refinement generally replaces RGB-D systems.

Rating¶

Novelty: 3/5. The closed-loop integration is useful, while recurrent matching, scale correction, and compatibility graphs have established precedents.
Experimental Thoroughness: 3/5. Seven BOP datasets and a two-module ablation provide coverage, but conflicting auxiliary results weaken completeness.
Writing Quality: 2/5. The main pipeline is understandable; supervision, metric labels, and implementation details need clarification.
Value: 3/5. Relevant to RGB-only pose refinement and robust geometric fusion, subject to further implementation verification before reproduction or deployment.