Skip to content

Consistent Monocular Depth Estimation with Contact Region Boundary-Aware Refinement

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/abai969/contact-depth-refinement.git
Area: 3D Vision
Keywords: Monocular Depth Estimation / Depth Consistency / Contact Boundaries / Boundary-Aware Feature Fusion / Structural Geometric Priors

TL;DR

To eliminate pervasive and erroneous depth discontinuities at object contact regions in monocular depth estimation, this paper introduces a contact boundary-aware estimation and refinement framework that distills physical contact boundaries using vision-language semantics, fuses them into the depth decoder via multi-scale structural priors, and enforces multi-order geometric constraints, significantly resolving contact surface gaps with only 4,000 synthetic images and supporting controllable interactive refinement.

Background & Motivation

Monocular depth estimation (MDE), which infers per-pixel depth from a single RGB image, forms the fundamental perception cornerstone for robotic manipulation, autonomous driving, and 3D scene reconstruction. Recently, depth foundation models such as Depth Anything have achieved remarkable prediction accuracy and zero-shot cross-domain generalization via large-scale weakly supervised pretraining; concurrently, affine-invariant 3D point cloud regression methods like MoGe have further introduced local and global multi-scale alignment geometric constraints to mitigate the neglect of inter-pixel spatial continuity inherent in traditional pixel-wise regression. Nevertheless, both direct depth regression models and 3D point-cloud architectures suffer from a systematic and persistent defect at object contact regions—particularly between an object and its supporting surface, or between connected components of an articulated object—frequently producing sharp depth drop-offs, structural detachments, and floating artifacts despite physical adjacency.

The core tension underlying this widespread failure is that existing depth decoding architectures lack explicit awareness of physical contact structures. Traditional smoothness regularization or edge-preserving constraints inherently treat all image gradients and visual edges as indicators of depth discontinuities to avoid over-smoothing across occlusion boundaries. However, physical contact areas exhibit the exact opposite geometric behavior: even in the presence of dramatic photometric contrast and intense edge responses (e.g., a dark laptop resting on a light wooden floor), physical reality dictates that 3D depth must remain strictly continuous across the interface. Standard decoders, unable to distinguish whether an edge denotes an occlusion boundary or a physical contact, uniformly decode high-contrast edges into depth cliffs.

To resolve this dilemma, this paper proposes a boundary-aware monocular depth estimation framework that explicitly leverages physical contact boundaries as structural priors. The framework first isolates true physical contact boundaries from dense edge candidates using cross-boundary semantic features, projects the contact probability map into a high-dimensional structural representation to guide depth decoding, and incorporates multi-order boundary geometry constraints alongside an auxiliary consistency regularizer. Core idea: introduce physical contact boundaries as explicit structural priors into the monocular depth decoding pipeline, decoupling contact regions via cross-normal vision-language semantic difference filtering and regularizing them with zero-order depth alignment and first-order normal gradient projections to eliminate spurious contact discontinuities.

Method

Overall Architecture

Building upon the affine-invariant 3D geometry estimator MoGe, the framework consists of two primary operational stages: (1) a contact boundary detection and semantic filtering module that decouples genuine physical contact contours from the raw RGB image, and (2) a boundary-aware feature fusion and depth decoding module that injects the contact structural priors across multiple scales of the 3D point decoder under multi-order geometric supervision. Furthermore, the pipeline naturally accommodates an interactive mode where users can manually annotate missing contact lines to trigger local depth re-optimization.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input RGB Image"] --> B["Contact Boundary Detection & Cross-Normal Semantic Filtering"]
    A --> C["Structural Dimension Lifting & Multi-Scale Feature Fusion"]
    B -->|Contact Boundary Map B| C
    B -->|Supervision Mask Mb| D["Multi-Order Geometric Boundary Constraints & Auxiliary Consistency"]
    C -->|Predicted Point Map P'| D
    C --> E["Consistent 3D Point Map & Continuous Depth"]
    E -.->|User-Specified Contact Boundaries| F["Interactive Controllable Boundary Refinement"]
    F -.->|Updated Boundary Prior| C

The feed-forward pipeline operates seamlessly: given an input image, an edge detector and a SigLIP visual encoder extract cross-interface semantic difference descriptors along candidate edge normals, which are processed by a lightweight MLP classifier to yield a pixel-wise contact probability map; a dedicated convolutional boundary encoder then lifts this map into the deep multi-scale feature space to fuse with primary visual features from DINOv2, enabling the multi-scale convolutional decoder to output a physically consistent, affine-invariant 3D point cloud.

Key Designs

1. Contact Boundary Detection & Cross-Normal Semantic Filtering: Decoupling Physical Contact from General Edges

Generic edge detectors (such as MuGE) produce dense boundary maps capturing multi-scale image transitions, but their outputs inevitably conflate physical contact boundaries with irrelevant texture transitions, illumination shadows, and depth occlusion boundaries. Indiscriminately feeding all raw edges into a depth decoder introduces heavy structural noise. Determining whether an edge constitutes a contact interface fundamentally hinges on verifying whether the regions on its two sides satisfy physical relationships like support, placement, or joint attachment.

To capture this relational context, the framework leverages a pretrained SigLIP visual-language encoder to extract dense feature maps \(S \in \mathbb{R}^{H \times W \times C}\). For each candidate boundary point \(q_i = (x_i, r_i)\) provided by the edge detector (where \(x_i\) is the pixel coordinate and \(r_i\) is the local unit normal vector), features are sampled symmetrically along the normal on both sides: $\(s_i^+ = S(x_i + \delta r_i), \quad s_i^- = S(x_i - \delta r_i)\)$ The cross-boundary difference \(\Delta s_i = s_i^+ - s_i^-\) captures local semantic transitions and boundary relational properties. The comprehensive point descriptor is formed by concatenation: \(t_i = [s_i, s_i^+, s_i^-, \Delta s_i]\). A lightweight MLP classifier \(f_\theta(\cdot)\) then predicts the contact probability \(\hat{b}_i \in [0, 1]\) to construct the contact boundary map \(\hat{B}\). During training, in addition to a position-weighted binary cross-entropy loss \(\mathcal{L}_{\mathrm{bce}}\), an edge variance loss \(\mathcal{L}_{\mathrm{var}}\) is introduced over connected curves \(L = \{l_k\}\) to penalize probability fluctuations along contiguous contact lines, ensuring topological smoothness and spatial continuity: $\(\mathcal{L}_{\mathrm{var}} = \frac{1}{|L|} \sum_{k=1}^{|L|} \mathrm{Var}(\hat{l}_k)\)$

2. Structural Dimension Lifting & Multi-Scale Feature Fusion: Injecting Boundary Priors into the 3D Decoder

The single-channel contact probability map \(\hat{B}\) provides only static, pixel-level confidence scores without hierarchical contextual representations, creating a severe dimensional and semantic discrepancy with the multi-scale visual features \(F\) extracted by the primary DINOv2 backbone. Directly concatenating raw probabilities at early layers would allow shallow boundary noise to disrupt global 3D point cloud reconstruction.

To bridge this representational gap, a dedicated ResNet-18 boundary encoder \(R_\phi(\cdot)\) is introduced to lift \(\hat{B}\) into high-dimensional, multi-scale feature space: \(C = R_\phi(\hat{B})\). The multi-scale boundary representations \(C\) are progressively aligned and fused with image features \(F\), yielding \(\tilde{F} = \psi(F, C)\). Operating on this fused representation, the multi-scale convolutional decoder \(D_\theta(\cdot)\) reconstructs the boundary-aware affine-invariant point map \(\hat{P}' = D_\theta(\tilde{F})\). This lifting-and-fusion strategy enables the decoder to suppress depth divergence along the line of sight specifically at contact locations, restoring smooth transitions across adjacent surfaces.

3. Multi-Order Geometric Boundary Constraints & Auxiliary Consistency: Balancing Local Smoothing with Global Integrity

Point-wise coordinate regression alone cannot guarantee gradient continuity across boundaries. Hence, the framework introduces complementary zero-order alignment and first-order normal gradient constraints. After optimizing the optimal global scale \(s^*\) and translation \(t^*\) via affine alignment, aligned depths \(\hat{d}_i\) and ground-truth depths \(d_i\) are extracted. On the predicted contact boundary pixel set \(M_b\), the zero-order depth alignment loss is defined as: $\(\mathcal{L}_{\mathrm{bd}} = \frac{1}{|M_b|} \sum_{i \in M_b} |\hat{d}_i - d_i|\)$ To constrain boundary geometric orientations and prevent spatial oscillations, a first-order gradient projection residual loss and a normal smoothness regularizer are enforced along the ground-truth depth gradient normal \(n_i = \nabla d_i / \|\nabla d_i\|_2\): $\(\mathcal{L}_{\mathrm{grad}} = \frac{1}{|M_b|} \sum_{i \in M_b} |n_i^\top (\nabla \hat{d}_i - \nabla d_i)|, \quad \mathcal{L}_{\mathrm{sm}} = \frac{1}{|M_b|} \sum_{i \in M_b} |n_i^\top \nabla \hat{d}_i|\)$ Furthermore, to mitigate the risk of decoder deviation when boundary cues are absent or corrupted, an auxiliary consistency constraint (Auxiliary Consistency) is designed. When feeding an all-zero map \(\hat{B} = \mathbf{0}\) into the boundary encoder, the decoder's auxiliary prediction \(\hat{p}_i^{\mathrm{aux}}\) is regularized against the frozen baseline MoGe output \(\hat{p}_i\) across all pixels \(M\): $\(\mathcal{L}_{\mathrm{cons}} = \frac{1}{|M|} \sum_{i \in M} \|\hat{p}_i^{\mathrm{aux}} - \hat{p}_i\|_2\)$ This regularizer ensures that the model stably falls back to the robust baseline geometry in boundary-free regions, guarding against global surface distortion induced by false-positive boundary predictions.

4. Interactive Controllable Boundary Refinement: Human-in-the-Loop Geometric Rectification

Under challenging visual conditions such as intense specular reflection, extreme texture sparsity, or heavy occlusion, automated boundary detection may miss subtle contact lines. Because the boundary guidance is decoupled from the depth decoding pathway, the framework inherently supports human-in-the-loop interactive refinement. Users can draw missing contact edges directly on a graphical interface; the manually supplied lines update \(\hat{B}\), allowing the network to rectify local depth discontinuities in a single feed-forward pass without re-training.

Loss & Training

The framework is trained using a progressive two-stage optimization strategy under a unified total objective function: $\(\mathcal{L}_{\mathrm{total}} = \mathcal{L}_{MoGe} + \mathcal{L}_F + \mathcal{L}_B\)$ Here, \(\mathcal{L}_{MoGe}\) comprises the global affine loss \(\mathcal{L}_G\), three-scale (\(\alpha \in \{1/4, 1/16, 1/64\}\)) local spherical alignment losses \(\mathcal{L}_{S(\alpha)}\), surface normal loss \(\mathcal{L}_N\), and valid mask loss \(\mathcal{L}_V\). The contact boundary filtering loss is defined as \(\mathcal{L}_F = \mathcal{L}_{\mathrm{bce}} + \lambda_e \mathcal{L}_{\mathrm{var}}\) with curve-variance weight \(\lambda_e = 100\). The boundary-aware depth loss is formulated as: $\(\mathcal{L}_B = \lambda_b \mathcal{L}_{\mathrm{bd}} + \lambda_g \mathcal{L}_{\mathrm{grad}} + \lambda_m \mathcal{L}_{\mathrm{sm}} + \lambda_{\mathrm{aux}} \mathcal{L}_{\mathrm{cons}}\)$ with hyper-parameters set to \(\lambda_b = 0.4\), \(\lambda_g = 0.3\), \(\lambda_m = 0.3\), and \(\lambda_{\mathrm{aux}} = 500\).

Implementation details: MuGE and the SigLIP vision backbone are frozen in stage one, while the boundary filtering MLP is trained for 5 epochs (batch size 2, learning rate \(1 \times 10^{-4}\)). In stage two, the entire system is jointly fine-tuned for 12 epochs (batch size 8 on a single NVIDIA A100-80G GPU). Learning rates are set to \(1 \times 10^{-6}\) for the SigLIP backbone and depth ViT encoder, and \(1 \times 10^{-5}\) for newly added modules (MLP classifier and ResNet-18 boundary encoder), halving every 200 steps. Training utilizes only 4,000 synthetic indoor images sampled from the Hypersim dataset, demonstrating outstanding data efficiency.

Key Experimental Results

Main Results

Evaluation is conducted across three zero-shot unseen indoor benchmarks: NYUv2, the high-resolution geometric benchmark iBims-1, and the challenging HAMMER benchmark containing untextured, reflective, and transparent surfaces. Metrics include absolute relative error \(\mathrm{Rel}_d \downarrow\), inlier percentage \(\delta_{d1} \uparrow\), relative point error \(\mathrm{Rel}_p \downarrow\), and local point error \(\mathrm{Local} \ \mathrm{Rel}_p \downarrow\).

The table below summarizes relative depth performance across benchmarks (strictly faithful to Table 1 and Table 3 of the original paper):

Dataset Alignment Setting Metric Ours Baseline MoGe Depth Anything V2 Gain (vs MoGe)
iBims-1 Scale-invariant depth \(\mathrm{Rel}_d \downarrow / \delta_{d1} \uparrow\) 3.35 / 97.7% 3.46 / 97.0% 4.32 / 97.9% \(\mathrm{Rel}_d\) reduced by 0.11 / \(\delta_{d1}\) +0.7%
iBims-1 Affine-invariant depth \(\mathrm{Rel}_d \downarrow / \delta_{d1} \uparrow\) 2.67 / 98.1% 2.74 / 97.9% 3.44 / 98.3% \(\mathrm{Rel}_d\) reduced by 0.07
iBims-1 Affine-inv disparity \(\mathrm{Rel}_d \downarrow / \delta_{d1} \uparrow\) 3.12 / 98.2% 3.23 / 98.0% 3.47 / 98.5% \(\mathrm{Rel}_d\) reduced by 0.11
NYUv2 Scale-invariant depth \(\mathrm{Rel}_d \downarrow / \delta_{d1} \uparrow\) 3.43 / 98.4% 3.44 / 98.4% 5.03 / 97.3% Maintains top performance
HAMMER Scale-invariant depth \(\mathrm{Rel}_d \downarrow / \delta_{d1} \uparrow\) 3.74 / 98.1% 3.77 / 98.1% 5.92 / 97.7% \(\mathrm{Rel}_d\) reduced by 0.03
iBims-1 Contact Region \(\mathrm{Rel}_d \downarrow / \delta_{d1} \uparrow\) 1.25 / 98.8% 1.71 / 98.7% — \(\mathrm{Rel}_d\) drops by 0.46 (-26.9%)
NYUv2 Contact Region \(\mathrm{Rel}_d \downarrow / \delta_{d1} \uparrow\) 1.31 / 99.2% 1.59 / 99.0% — \(\mathrm{Rel}_d\) drops by 0.28 (-17.6%)

In 3D point estimation (Table 2 of the original paper), our method lowers the affine-invariant point error \(\mathrm{Rel}_p\) on iBims-1 from 3.61 (MoGe) to 3.52 and elevates \(\delta_{p1}\) from 97.3% to 98.0%. The local point error \(\mathrm{Local} \ \mathrm{Rel}_p\) drops to 4.03 (surpassing DUSt3R at 5.44 and UniDepth at 5.92), verifying substantial gains in fine-grained structural fidelity.

Ablation Study

A systematic component-wise ablation on the iBims-1 benchmark validates the respective contributions of edge detection, contact filtering, and the auxiliary consistency loss \(\mathcal{L}_{\mathrm{cons}}\) (strictly faithful to Table 4 of the original paper):

Detection Filtering Regularizer (\(\mathcal{L}_{\mathrm{cons}}\)) Scale-inv Depth \(\mathrm{Rel}_d \downarrow\) Scale-inv Point \(\mathrm{Rel}_p \downarrow\) Affine-inv Point \(\mathrm{Rel}_p \downarrow\) Local Point \(\mathrm{Rel}_p \downarrow\) Observation
✗ ✗ ✗ 3.46 4.63 3.61 4.16 Baseline MoGe
✓ ✗ ✗ 3.42 9.02 5.45 4.05 Unfiltered raw edges induce severe global point distortion (\(\mathrm{Rel}_p\) doubles)
✓ ✗ ✓ 3.36 4.70 3.58 4.03 Consistency loss effectively recovers global geometric stability
✓ ✓ ✓ 3.35 4.49 3.52 4.03 Full model: semantic filtering eliminates spurious edges, yielding best overall metrics

Key Findings

  • Naively fusing unfiltered edges wrecks global 3D geometry: As shown in row 2 of the ablation table, directly feeding all detected MuGE edges into the depth decoder reduces scale-invariant depth \(\mathrm{Rel}_d\) slightly to 3.42, but causes scale-invariant point error \(\mathrm{Rel}_p\) to surge from 4.63 to 9.02. This confirms that smoothing across non-contact edges or texture gradients severely distorts 3D spatial layout.
  • Auxiliary consistency provides crucial structural anchoring: Incorporating \(\mathcal{L}_{\mathrm{cons}}\) drives \(\mathrm{Rel}_p\) back down from 9.02 to 4.70, validating its role in preventing decoder drift when edge cues are absent or noisy.
  • Gains are heavily concentrated at contact regions: While full-image metrics can obscure local improvements, the contact-centric evaluation reveals that relative depth error drops by 26.9% on iBims-1 (from 1.71 to 1.25) and by 17.6% on NYUv2 (from 1.59 to 1.31), proving the precise elimination of contact-surface depth cliffs.

Highlights & Insights

  • Cross-Normal Semantic Difference Representation: Leveraging pretrained VLM (SigLIP) representations along local normal vectors converts subtle physical contact and support relationships into concrete feature divergence descriptors \(\Delta s_i\), offering an elegant and effective formulation.
  • Complementary Multi-Order Boundary Geometry: Combining zero-order depth alignment \(\mathcal{L}_{\mathrm{bd}}\) with first-order gradient projection \(\mathcal{L}_{\mathrm{grad}}\) and normal smoothness \(\mathcal{L}_{\mathrm{sm}}\) ensures depth continuity while preventing high-frequency boundary oscillations.
  • High Data Efficiency via Structural Priors: While foundation models require millions of training images, this method achieves state-of-the-art contact consistency by fine-tuning on only 4,000 synthetic indoor images, highlighting how targeted structural priors alleviate data dependency.
  • Generalizable Interactive Rectification Paradigm: Decoupling boundary extraction from depth decoding enables real-time user-guided correction of missing contact boundaries, providing a valuable template for interactive perception in robotics and augmented reality.

Limitations & Future Work

  • Admitted Limitations: The pipeline depends on the quality of automated edge detection and semantic filtering; challenging visual factors such as extreme specular reflections or low-contrast transparent objects can still cause missed contact detections, requiring manual interactive correction.
  • Scene Domain Scope: Experiments focus exclusively on indoor static benchmarks (NYUv2, iBims-1, HAMMER). The framework has not been evaluated on high-speed outdoor driving scenarios (e.g., KITTI, nuScenes) where dynamic wheel-ground contact stability is crucial.
  • Future Directions: Consolidating the two-stage detection-and-depth pipeline into a fully unified, single-stage end-to-end framework, and exploring differentiable boundary tracing to replace discrete curve variance constraints.
  • vs Depth Anything V1/V2: Depth Anything models rely on large-scale pseudo-label distillation, but frequently predict floating or detached artifacts at supporting interfaces due to the lack of explicit contact constraints; this method eliminates these artifacts while requiring significantly less training data.
  • vs MoGe: While MoGe enforces global and local spherical 3D alignment, it still exhibits gaps at fine-grained contact boundaries; this work enriches MoGe with semantic boundary priors and multi-order gradient supervision, substantially reducing local point map errors.
  • vs Traditional Edge-Aware Depth (e.g., GeoNet++, SharpNet): Prior edge-aware methods utilize edges primarily as barriers to preserve depth discontinuities and prevent over-smoothing; this paper identifies edges that represent physical contact and enforces continuity across them, flipping the conventional role of edge priors in depth estimation.

Rating

  • Novelty: ⭐⭐⭐⭐ [Insightful distinction between depth-discontinuous edges and contact-continuous edges using VLM cross-normal semantic difference]
  • Experimental Thoroughness: ⭐⭐⭐⭐ [Rigorous evaluations across three unseen benchmarks with comprehensive depth, 3D point, and contact-specific metrics]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Clear exposition, seamless logical progression, well-defined mathematical formulations, and high-quality visualizations]
  • Value: ⭐⭐⭐⭐ [Directly tackles the long-standing physical detachment artifact in monocular depth estimation with both autonomous and interactive utility]