Masked Depth Modeling for Spatial Perception¶

Conference: ECCV2026
Paper: ECCV Paper
Project: LingBot-Depth
Code: https://github.com/robbyant/lingbot-depth
Area: Autonomous Driving (current archive); the research concerns general 3D Vision
Keywords: depth completion, masked depth modeling, RGB-D, geometric representations, robotic grasping

TL;DR¶

MDM turns missing sensor depth into physically motivated masks and learns geometric representations through joint attention over complete RGB and remaining depth, reducing NYUv2 Extreme RMSE from PromptDA's 0.324 to 0.167 while improving real-world robotic grasping.

Background & Motivation¶

Robots need to know not only what an image contains, but how far each surface lies in the physical world. Multi-view reconstruction depends on viewpoints and computation, monocular depth has scale ambiguity, and RGB-D cameras provide directly aligned metric depth but develop holes and errors on reflective, transparent, or textureless surfaces and under difficult lighting. For localization and grasping, those holes often occur precisely where reliable geometry matters most.

PromptDA and PriorDA already show that visual priors can repair incomplete sensor depth. This paper asks a further pretraining question: how can appearance representations absorb geometry instead of merely attaching sparse depth to an existing predictor? Natural missingness is not uniformly random; it correlates with materials and imaging ambiguities, making it closer to deployment difficulties than arbitrary depth removal. However, holes do not provide correct targets by themselves, so synthetic ground truth or real stereo pseudo-depth remains necessary for supervision.

MDM therefore designs both data collection and masked reconstruction: it keeps RGB complete as context, exposes only part of the depth, and asks the model to recover dense depth. Core Idea: turn sensor failure locations into targeted reconstruction tasks, using shared attention to incorporate visible depth scale and geometric relationships into RGB representations instead of treating every missing region as identically distributed random noise.

Method¶

Overall Architecture¶

The inputs are a spatially aligned RGB image and incomplete single-channel sensor depth; the output is an image-aligned dense depth map. Training first prepares RGB-D data with realistic failure patterns, then applies natural-missingness-first masking, joint multimodal encoding, and depth decoding solely from RGB contextual tokens.

The critical information path is that depth tokens participate in encoding but never enter the final decoder directly. To exploit depth, the encoder must transfer it through attention into the fully retained RGB tokens, which consequently become more than appearance-only features.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Aligned RGB-D data"] --> Data["Sensor-like Data Curation"]
    Data --> Mask["Natural-Missingness-First Masking"]
    Mask --> Encode["Joint Multimodal Encoding"]
    Encode --> Decode["Contextual Depth Decoding"]
    Decode --> Output["Dense depth and downstream perception"]

Key Designs¶

1. Sensor-like Data Curation: separate imperfect inputs from more reliable targets

The synthetic branch, MDM-S, does not simply feed perfect rendered depth into the model. Blender generates RGB, perfect depth, and stereo images with speckle patterns; SGM then produces sensor-like input depth with artifacts. RGB comes from the left camera to preserve pixel alignment. Sampling stereo baselines of 0.05-0.2 m and focal lengths of 16-28 mm produces 1M samples from 442 indoor scenes. Missingness and errors thus arise from imaging and matching, while geometric ground truth remains available for supervision.

The real branch, MDM-R, uses a capture rig with interchangeable commercial RGB-D cameras to record synchronized RGB, raw depth, and stereo observations. Since real scenes lack perfect ground truth, the authors generate pseudo-depth with a FoundationStereo-based network trained on synthetic data and filter it using left-right consistency. The text commonly summarizes the curated data as 2M real plus 1M synthetic samples, supplemented by public datasets to form 10M training samples; conflicting curated totals are discussed under limitations. This pipeline supplies supervision, so the training should not be understood as self-supervision from holes alone.

2. Natural-Missingness-First Masking: prioritize physical failures, then add random masks

Depth is divided into 14-by-14 patches. A patch with no valid depth is always masked; a patch mixing valid and invalid pixels is masked with probability 0.75. If the target ratio has not been reached, additional fully valid depth patches are randomly selected. The overall training target is 60%-90% masking, while RGB remains complete. Mixed patches are not universally removed because some imperfect observations can still provide useful scale or planar cues.

The rule exploits structured sensor missingness without discarding samples with nearly complete depth, which rely mainly on additional random masks. Public synthetic data with complete depth also use random masking. MDM therefore does not eliminate random masking, and its gains cannot all be attributed to natural masks. Depth completion uses the remaining observations; masking all depth tokens reduces the same encoder structure to an RGB-only monocular path suitable for transferring visual geometric priors, without establishing that physical scale ambiguity disappears when depth anchors are absent.

3. Joint Multimodal Encoding: absorb sparse geometric constraints into the full RGB grid

RGB and depth use separate patch embeddings, avoiding the assumption that three-channel color and single-channel depth share input statistics. Both receive the same 2D spatial embedding plus different modality embeddings. This identifies corresponding locations while distinguishing color from distance. All RGB tokens are concatenated with unmasked depth tokens and passed through a DINOv2-initialized, 24-layer ViT-L/14, retaining a [cls] token for global context.

Shared self-attention allows locally reliable depth to interact with visual cues elsewhere, supporting completion through coplanarity, near-far relationships, and surface continuity. Depth-query attention visualizations highlight corresponding RGB regions, supporting the cross-modal representation interpretation without causally proving a specific geometric function for every attention head. Dense prediction uses only the final encoder layer, without aggregating intermediate features from multiple layers.

4. Contextual Depth Decoding: discard depth tokens and retain geometry-enriched RGB tokens

After encoding, all depth tokens are discarded. The [cls] token is broadcast and added to each RGB contextual token before the features enter a ConvStack decoder adopted from MoGe. A shared neck and task-specific heads use residual blocks and transposed convolutions for progressive upsampling. Both kernel and stride are 2; spatial features grow to 16 times the original patch-grid resolution, and predictions are subsequently resized to the input resolution.

Unlike the shallow Transformer used by conventional MAE to reconstruct RGB pixels, this decoder starts with a complete RGB grid whose tokens already carry contextual information and can recover continuous geometry through convolution. The paper describes learned full-map prediction rather than directly pasting valid sensor values back into the output, and specifies no hard constraint preserving each original reading. This permits correcting bad measurements but can also alter initially correct depth.

A Worked Example¶

Consider a transparent cup on a table: the camera measures the table and nearby objects but leaves holes on the cup walls. During training, completely empty cup-wall patches are removed, mixed patches are removed with probability 0.75, and additional masks fill the 60%-90% target. The cup outline and table texture remain visible in the complete RGB image.

The encoder jointly processes reliable table depth and the full image, allowing RGB tokens to acquire scale and geometric context from surrounding objects. The decoder then receives only these fused RGB tokens and reconstructs continuous depth around the cup. Synthetic ground truth or filtered pseudo-depth constrains the output, rather than treating sensor holes as target values.

This example explains information flow; it is not a patch-by-patch measurement reported by the authors. Transparent surfaces can still induce incorrect completion. The real grasping table reports only 10/20 successes for the transparent storage box, showing that a plausible reconstructed shape does not guarantee completely accurate geometry.

Loss & Training¶

The authors explicitly use an L1 depth loss on valid supervision pixels. The following conventional equivalent notation expresses that description; it is not a numbered equation from the paper:

\[ \mathcal{L}_{\mathrm{depth}} = \frac{1}{|\Omega_{\mathrm{valid}}|}\sum_{p\in\Omega_{\mathrm{valid}}}\left|\hat D_p-D_p^*\right|. \]

The valid set depends on supervision reliability, not on which input positions remain unmasked. Hidden input regions still contribute whenever reliable targets exist. The paper does not restrict the loss to masked patches alone or specify additional normal or temporal losses in the main text, so those terms are not added here.

The encoder starts from DINOv2 weights and the decoder is randomly initialized. Training uses 250K iterations, batch size 1,024, 128 GPUs, BF16, and gradient clipping at norm 1.0, taking about 7.5 days. The main text refers other training details to supplementary material absent from the current cache; undisclosed settings such as learning rate are not inferred.

Key Experimental Results¶

Main Results¶

Protocol 1 in Table 2(a) randomly masks spatial blocks of ground-truth depth and adds Gaussian and shot noise, with Easy, Medium, Hard, and Extreme levels. The table below selects only Extreme. All metrics are lower-is-better: RMSE is root mean squared depth error, and REL is mean absolute relative depth error, not accuracy.

Dataset / Extreme	MDM RMSE	PromptDA RMSE	PriorDA RMSE	MDM REL	PromptDA REL
iBims-1	0.303	0.607	0.845	0.063	0.129
NYUv2	0.167	0.324	0.309	0.029	0.074
DIODE-Indoor	0.219	0.465	0.665	0.022	0.083
DIODE-Outdoor	3.913	4.313	5.114	0.085	0.156

The strongest NYUv2 Extreme RMSE baseline is PriorDA at 0.309, not PromptDA at 0.324. The main text's blanket identification of PromptDA as the best indoor competitor should therefore not be repeated literally. MDM leads on all four Extreme settings, although outdoor absolute error remains substantially larger.

Protocol 2 tests generalization with sparse observations. In Table 2(b), MDM achieves an average rank of 1.60 versus PriorDA's 2.60. However, its KITTI DC value is 1.412 versus OMNI-DC-DA's 1.310, so it does not win under every sparsity pattern. The cached table header does not explicitly identify the numerical metric; only the reported values and ranks are retained here, without inventing units.

Ablation Study¶

The available main text does not provide component-removal ablations for natural masking ratios, the decoder, or data sources. The following downstream analysis uses Table 4's refined-depth versus raw-depth comparison and cannot isolate an individual MDM component. Percentages are calculated by dividing successful trials by 20.

Grasped Object	Trials per Evaluable Condition	MDM Successes / Rate	Raw-Depth Successes / Rate
Stainless steel cup	20	17 / 85%	13 / 65%
Transparent cup	20	16 / 80%	12 / 60%
Toy car	20	16 / 80%	9 / 45%
Transparent storage box	20	10 / 50%	N/A

N/A denotes the authors' report that severely corrupted raw depth prevented grasping; it does not mean 20 completed trials with a 0% success rate. The system uses a Rokae XMate-SR5, X Hand-1, and Orbbec Gemini 335, conditioning a diffusion policy on RGB features and depth-derived point cloud features and training with retargeted HOI4D hand-object interactions.

Key Findings¶

Improvements extend beyond the appearance of completed maps: stainless steel and transparent cups each gain 4 successful grasps, and the toy car gains 7. With only 20 trials per condition, these results alone do not establish statistical significance.
In the TUM-RGBD camera-trajectory comparison, ATE decreases from 0.131 to 0.118 and rotational RPE from 0.844 to 0.791, but translational RPE increases from 0.038 to 0.039. Not every metric improves.
FoundationStereo's RGB-only prior replacement supports representation transfer, but the stated training duration conflicts with Figure 7 and the cached chart text is crowded, so exact per-dataset chart values are not transcribed.

Highlights & Insights¶

Missingness patterns carry task information. Targeting actual sensor failure regions better matches deployment than random holes in clean data, although the independent contribution still requires controlled ablation.
Discarding depth tokens is central to the information flow. Depth participates in attention and then exits the decoding path, making the complete RGB grid a carrier of transferable geometric representations.
Data curation preserves the distinction between imperfect inputs and more reliable targets. This principle can transfer to other sensor-repair tasks, but pseudo-label filtering reliability also limits what can be learned.

Limitations & Future Work¶

Curated-data counts conflict: the abstract and conclusion state 3M, Figure 4 lists 2.1M real plus 1.0M synthetic, and Section 4.1 states 3.2M. The reported total training set is 10M, which does not resolve the curated-data discrepancy.
Section 5.2 states 15 training epochs for stereo models, whereas Figure 7 and the subsequent results report 20 epochs. Without supplementary material and clean chart exports, the final comparison budget cannot be determined.
Natural masks, data scale, DINOv2 initialization, and the decoder vary together, with insufficient component ablations in the main text. Gains cannot all be assigned to the masking strategy.
Transparent objects remain difficult, and pseudo-depth inherits stereo matching errors. Calibrated uncertainty could help downstream planning avoid treating completed predictions as sensor ground truth.
Current evidence emphasizes static depth, indoor tracking, and a small set of grasped objects. It lacks extensive closed-loop driving or safety validation and an end-to-end latency table substantiating real-time operation.

vs MAE / CroCo: MAE reconstructs randomly masked RGB, while CroCo learns geometry through cross-view completion. MDM keeps RGB complete and reconstructs depth affected by sensor failures using depth supervision.
vs PromptDA / PriorDA: PromptDA introduces depth conditions on the decoder side, while PriorDA fuses depth before encoding. MDM uses separate modality tokens and shared attention, emphasizing the pretraining task itself.
vs MoGe / FoundationStereo: MDM adopts MoGe's ConvStack, and its RGB-only representation can serve as FoundationStereo's depth prior, connecting completion pretraining with stereo matching.
Testable next direction: compare natural, random, and mixed masks with fixed data and training budgets, then break errors down by material and sensor to distinguish difficult-example selection from increased training scale.

Rating¶

Novelty: 4/5. Combines natural missingness with cross-modal masked pretraining through a clear information path, although most architectural components already exist.
Experimental Thoroughness: 3/5. Covers completion, prior transfer, tracking, and grasping, but lacks sufficient controlled ablations, statistical intervals, and budget clarity.
Writing Quality: 3/5. The method is understandable, but conflicting data counts and training durations complicate reproducibility assessment.
Value: 4/5. Offers a practical direction for RGB-D perception and robotics, without establishing a validated safety-critical driving system.