MetricAnything: Scaling Metric Depth Pretraining with Noisy Heterogeneous Sources

Conference: ECCV2026
Paper: ECCV Paper
Project: MetricAnything
Area: 3D Vision
Keywords: metric depth estimation, heterogeneous-data pretraining, sparse depth prompts, knowledge distillation, monocular geometry

TL;DR

MetricAnything organizes pretraining on roughly 20 million heterogeneous image-depth pairs through random sparse depth prompts, then distills dense labels from a conditioned teacher into RGB-only students, achieving strong average rankings in depth completion and monocular metric depth while showing scaling gains that depend jointly on data, prompting, and distillation design.

Background & Motivation

Relative depth needs to recover ordering and geometry, whereas metric depth must also determine how many meters separate an object from the camera. The same image projection can result from different focal lengths, object sizes, or distances, so adding images does not automatically resolve scale ambiguity. Depth Anything has demonstrated the value of large mixed datasets for relative depth; metric methods such as Metric3D, UniDepth, and Depth Pro must additionally manage the coupling between camera parameters, absolute scale, and fine detail.

The difficulty is not simply a shortage of labels. LiDAR, ToF, RGB-D, stereo reconstruction, and rendered data differ in valid regions, noise patterns, and distance distributions. Reconstruction fails on weakly textured or reflective surfaces; sensors introduce occlusion, temporal misalignment, and sparse sampling; synthetic data offers clean geometry but limited realism. Naively mixing these sources can teach device-specific biases instead of general geometry. Existing prompt-based methods exploit a few known distances, but often simulate sensor-specific prompt patterns, coupling task engineering to data expansion.

This paper treats sparse metric observations as a common pretraining condition rather than a specialized input for one completion task. Known distances anchor scale, images provide information about unobserved regions, and the teacher converts noisy multi-source labels into more consistent supervision so that students can use learned geometric priors without prompts. Core Idea: first recover dense metric geometry from heterogeneous data using simple random sparse prompts, then transfer that capability to RGB-only prediction through distillation adapted to the properties of the pseudo labels.

Method

Overall Architecture

The system has a prompted teacher stage and a prompt-free student stage. The teacher receives an RGB image and measured distances at a small set of pixels, and predicts dense metric depth; students learn from teacher-generated pseudo depth but require only RGB at inference time. The paper separately trains Student-DepthMap, which predicts depth maps, and Student-PointMap, which is fine-tuned from MoGe-2 and predicts point maps. These are not two evaluation names for the same model.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Sources["Reconstructed, captured, and rendered data"] --> Data["Multi-Source Metric Data Unification"]
    Data --> Prompt["Sparse Metric Prompt Preparation"]
    RGB["RGB image"] --> Prior["Pretrained depth prior"]
    Prior --> Prompt
    Prompt --> Teacher["Conditioned Teacher Pretraining"]
    RGB --> Teacher
    Data -.->|Training supervision| Teacher
    Teacher -->|Generate dense pseudo labels| Student["Prompt-Free Student Distillation"]
    RGB --> Student
    Student --> Output["Inference: RGB-only depth or point maps"]
    Teacher --> PromptOutput["Prompted inference: dense metric depth"]

Prompt preparation does not simply feed a few points into a Transformer. A pretrained depth prior first fills out the condition, which is then regularized and injected into the DPT decoder. The dashed edge denotes supervision from original valid depths, not information available to the student at test time. The claim of requiring no manually designed prompts primarily refers to avoiding task-specific sampling patterns; it does not mean that the pipeline lacks prompt preprocessing or an external depth prior.

Key Designs

1. Multi-Source Metric Data Unification: converting device outputs into a common supervision interface

The authors aggregate three categories of open-source 3D data: reconstructed, sensor-captured, and rendered. The collection contains roughly 20 million image-depth pairs spanning more than 10000 camera models. Each depth map is accompanied by a validity mask indicating measured pixels, and depth is defined along the camera axis rather than treating every sensor's raw range as equivalent. Raw point clouds are projected into the image using known sensor-to-camera poses and camera intrinsics, producing depths and valid regions. Data preparation therefore still requires calibration; the absence of a specialized camera-conditioned network does not imply that raw data needs no calibration.

The sources have complementary roles. Reconstructed data expands scene coverage but can contain false geometry from matching failures. Captured data provides physical distances but is limited by sparsity, material reflectance, and sensing range. Rendered data contributes sharp, clean geometric boundaries. Validity masks remove known invalid measurements, but do not guarantee that every retained point is noise-free. Random prompting and robust supervision address these remaining errors rather than assuming that all inputs have already been cleaned into exact ground truth.

2. Sparse Metric Prompt Preparation: retaining scale anchors while regularizing irregular conditions

During training, 2000 to 40000 valid pixels are randomly sampled, with each prompt point containing image coordinates and its metric depth. The authors do not explicitly simulate laser scan lines, regular low-resolution grids, or a particular radar distribution. Instead, the sources themselves supply diverse missing-data patterns. The model must propagate sparse distance information into unobserved regions using image content instead of memorizing a fixed sensor layout. Evaluation can use as few as 100 points, but this is a generalization setting sparser than the training range, not the default training configuration.

Irregular point sets still need to become efficient two-dimensional conditions. Depth Pro first supplies a prior depth map. Pixel-wise Depth Scale Alignment (PDSA) and Global Metric Depth Recovery (GMDR) then fill missing prompt regions under that prior, after which the two processed maps and the prompt mask are concatenated into a three-channel condition. Local alignment and global recovery provide corrections at different spatial scales, while the mask retains the locations of actual observations. The main text does not give complete numerical implementations of these operations, so they should not be described as a particular unverified fitting algorithm. What is established is that preparation is non-learnable, can propagate noise, and therefore requires further correction by the teacher.

3. Conditioned Teacher Pretraining: assigning correction to the backbone rather than a heavy prompt branch

The teacher follows the Depth Pro-style ViT and multi-scale DPT architecture. Interpolation and shallow convolutions form a conditioned DPT head that injects the prepared prompt into decoding, adding approximately 5% more parameters. The original patch encoder and image encoder are also merged into a shared ViT. Most capacity consequently remains devoted to learning geometry and correcting errors from RGB, instead of making a complex conditioning branch perform all prediction. Cross-attention, AdaLN, and ControlNet-style conditioning are mentioned as alternatives, but the implemented choice is a conditioned decoder head; those alternatives should not be drawn as active modules.

Sparse observations are not hard constraints that must always be satisfied. The teacher is intended to identify anomalous prompt values and use image structure to produce more consistent dense geometry. Clean synthetic data supervises both depth values and gradient structure. For real data, the highest-loss 20% of regions in each image are discarded, reducing the influence of erroneous measurements. Robustness therefore comes from data and loss design together, not from random sampling automatically removing every sensor bias. Once pretrained, the same teacher can use low-resolution depth or sparse observations for super-resolution and completion without retraining for each test prompt type.

4. Prompt-Free Student Distillation: adapting the student's inductive bias to cleaner supervision

The teacher generates dense pseudo labels for collected real images using their sparse prompts, converting annotations with different sources and valid regions into unified predictions. It particularly fills distant backgrounds beyond the coverage of the original sensors. Simply removing the teacher's prompt layers and training a student is suboptimal: direct depth losses can damage fine detail, while inverse-depth supervision suppresses distant gradients too strongly. The authors introduce distance-balanced supervision in log-depth space to preserve learning signals for both nearby structure and distant geometry, ultimately selecting a balance coefficient of 400.

Feature connections also change with supervision quality. Conventional U-Net-style skips route shallow color and texture features toward decoder stages near the output, which can stabilize learning under noisy labels but encourage excessive reliance on low-level cues. The paper reverses the ViT-to-DPT skip arrangement: deep semantic features feed decoder layers closer to the output, while shallow features enter earlier decoding layers. The authors argue that more uniform, cleaner pseudo labels allow final predictions to depend more strongly on semantic geometry. This rationale is conditional: reversed connections are not better with original real-world labels in the experiments, so the architecture is not an unconditional replacement for all depth networks.

A Worked Example

Consider a street image with sparse LiDAR depth. During training, the pipeline identifies valid projected measurements and randomly retains 2000 depth points. A Depth Pro prior supplies full-image shape; PDSA, GMDR, and the mask form the three-channel condition. The RGB backbone and conditioned DPT jointly predict vehicles, road surfaces, and distant regions without direct measurements. Valid depths still supervise the teacher, with high-loss regions receiving reduced influence.

After teacher training, the image receives a dense pseudo label. The student learns from the image and that pseudo label, but does not receive the 2000 prompt points. At deployment, a new RGB image alone can produce metric depth. When sparse sensor measurements are available, the teacher branch can instead be used directly. This street scene is a walkthrough of the pipeline, not an additional quantitative test reported by the paper.

Loss & Training

Pretraining uses mean absolute error (MAE) and scale-and-shift-invariant mean absolute gradient error (SSI-MAGE) for synthetic data, and robust MAE with high-loss-region removal for real data. The cached total-loss and distance-balancing equations contain missing symbols, so this note does not reconstruct exact operators, weights, or implementation details. The prose supports log-domain supervision and the choice of coefficient 400, but the damaged formula alone cannot establish a reproducible definition of the claimed distance-weighting mechanism.

The teacher trains on 144 H200 GPUs for 100000 steps with 10000 warm-up steps. Peak learning rates are 0.000001 for the ViT and 0.00001 for the DPT and prompt layers; added prompt convolution kernels and biases are zero-initialized. Although the authors call Student-DepthMap training "from scratch," its ViT is initialized from DINOv3 ViT-H+/16, and only its DPT head is randomly initialized. This model trains on 144 H200 GPUs for 200000 steps with 5000 warm-up steps, using peak learning rates of 0.000002 and 0.00002 for the ViT and DPT, respectively.

Student-PointMap initializes all parameters from MoGe-2 and is fine-tuned on 80 H200 GPUs for 10000 steps with 1000 warm-up steps. Peak learning rates are 0.0000025 for the ViT and 0.000025 for the DPT. Radar fusion, VLA, and VLM transfer have additional downstream training settings; zero-shot monocular depth evaluation does not imply that these applications require no adaptation.

Key Experimental Results

Main Results

Table 1 excerpts the paper's Table 1 on zero-shot depth super-resolution and completion: the pretrained teacher directly processes unseen test datasets without task-specific fine-tuning. AbsRel is the mean absolute depth error divided by ground-truth depth over valid pixels; values below are percentages, with lower being better. The 8x and 16x settings denote input depth downsampling factors; Extreme denotes 100 sparse points.

Dataset / Prompt	PriorDA AbsRel (%)	MetricAnything AbsRel (%)	Note
KITTI / 8x	4.54	2.34	Super-resolution
KITTI / 16x	8.20	3.53	Lower-resolution input
ETH3D / LiDAR	1.90	0.87	LiDAR-like scan prompts
ETH3D / Extreme	1.61	0.84	Only 100 points
NYUv2 / Extreme	2.01	2.08	Worse than PriorDA here
Average rank across all 12 settings	3.25	1.50	Rank, not AbsRel; lower is better

Table 2 excerpts the paper's Table 3 for RGB-only zero-shot monocular depth estimation using Student-DepthMap, without sparse prompts. \(\delta_1\) is the fraction of pixels where the larger of the predicted-to-true depth ratio and its reciprocal is below 1.25. Values are percentages, with higher being better.

Dataset	Depth Pro \(\delta_1\) (%)	Metric3D-v2 \(\delta_1\) (%)	Student-DepthMap \(\delta_1\) (%)
ETH3D	41.5	87.7	79.9
Booster	46.6	39.4	59.5
nuScenes	49.1	82.6	88.1
SunRGBD	89.0	75.6	97.7
Sintel	40.0	38.3	27.7
Middlebury	60.5	29.9	65.8

Student-DepthMap has an average rank of 1.50 among all methods in the original table, but stronger competitors clearly exist on ETH3D and Sintel. Both tables support broad competitiveness rather than leadership on every dataset.

Ablation Study

Table 3 corresponds to the paper's Table 12, with student training data, schedule, and evaluation settings held constant. AbsRel is reported as a percentage, with lower being better. This is a dedicated component-ablation experiment, so its absolute error values should not be directly combined with the different metric in Table 2.

Config	Booster AbsRel (%)	ETH3D AbsRel (%)	Average AbsRel (%)
Full MetricAnything	28.2	14.7	21.5
PriorDA-style task-specific prompts	34.1	33.2	33.7
Single-source synthetic / rendered data only	45.4	17.4	31.4
U-Net-style student connections	39.7	32.0	35.9
Without distance-balanced loss	40.3	16.1	28.2

The paper's Table 13 further shows that with ETH3D pseudo-label training, reversed connections reduce Abs from 0.269 to 0.182 but increase RMSE from 1.440 to 1.898. With real labels, Abs changes from 0.327 to 0.334. Architectural benefits do depend on supervision source, and not every error metric improves simultaneously.

Key Findings

• Data scaling requires the method to scale with it: Section 4.4 reports that expanding teacher training from 8 million to 20 million samples reduces full-model AbsRel by 22.5% relatively, while the controlled PriorDA-style baseline worsens by 4.4%. The comparison uses the same 100000 training steps and evaluation protocol. This is an empirical trend over the studied scale range, not an established universal power law.
• Denser prompts are not always cost-effective: in the paper's Table 15, increasing Hypersim prompt points from 500 to 64000 reduces AbsRel from 0.043 to 0.031 while latency rises from 224 to 308 milliseconds, with diminishing accuracy gains. This supports a training-density trade-off, but does not imply identical latency across deployment hardware.
• Transfer results require their training conditions: in the paper's Table 2, nuScenes radar-fusion MAE over 0 to 50 meters drops from 1335.4 millimeters for training from scratch to 651.4 after pretraining and fine-tuning. The VSI-Bench average of 58.3 in Table 8 comes from a model additionally fine-tuned on 200000 QA pairs and 4225 planning instances, not a zero-shot VLM evaluation.

Highlights & Insights

• Prompts do more than supply depth at inference time: they also organize heterogeneous supervision during training. The transferable idea is to anchor scale with limited reliable observations, then use a teacher to construct more consistent dense targets.
• Distillation changes the supervision distribution rather than merely compressing a model. Jointly adjusting connections and losses highlights why an architecture suited to noisy labels may not suit cleaner pseudo labels covering longer distances.
• Controlled scaling comparisons are more informative than simply presenting a large model. They partly separate having more data from using data effectively, although fuller compute-matched experiments are still needed to explain cost-effectiveness.

Limitations & Future Work

• The authors explicitly discuss prompt preparation remaining sensitive to sampling patterns and noise, and conventional student losses and connections underusing long-range pseudo labels. The main text has no standalone limitations section, so these motivations should not be treated as evidence that all such issues are solved.
• This note's assessment: the method depends on a pretrained depth prior, calibrated source data, and large-scale H200 training resources. Simplicity mainly concerns the prompting task and network modifications, not necessarily low data-processing or training costs.
• This note's assessment: dense pseudo labels beyond the original sensor range are still model predictions, not newly measured ground truth. Distant geometry can inherit teacher biases, motivating distance-stratified confidence evaluation and independent long-range measurements.
• Reproducibility boundary: only the main-paper cache was supplied, without supplementary material; some equations and side-by-side tables have damaged text extraction. This note does not reconstruct corrupted formulas. More precise loss definitions, data splits, and implementation details require the original materials.

Related Work & Insights

• vs Depth Pro: the paper reuses its depth architecture and parts of its supervision design, and also uses it to produce the prior for prompt preparation. The contribution centers on organizing heterogeneous-data pretraining, lightweight conditioning, and student distillation rather than inventing a completely new depth backbone.
• vs PriorDA / PromptDA: all use sparse metric priors to guide prediction. MetricAnything shifts emphasis from task-specific prompt engineering to random prompts and data scaling, but still borrows prior alignment and conditioned decoding rather than discarding the existing processing pipeline entirely.
• vs MoGe-2: it supplies both Student-PointMap initialization and a controlled pseudo-label-source baseline. This shows that teacher supervision transfers to point-map prediction, but student improvements cannot be attributed entirely to a new architecture.
• Research direction: jointly control data source, prompt noise, and distance range under fixed training compute to determine when adding low-quality data remains beneficial. This is a question proposed by this note, not a conclusion already established by the paper.

Rating

• Novelty: 4/5. The central contribution combines scalable pretraining and distillation; individual components largely build on prior methods.
• Experimental Thoroughness: 4/5. The study covers prompted, prompt-free, and cross-task transfer settings with controlled ablations, but task-specific training conditions require careful distinction.
• Writing Quality: 3/5. The main argument is clear, although some blanket superiority claims exceed individual results, and cached equations limit exact verification.
• Value: 4/5. The framework offers a scalable route for heterogeneous metric-depth data, with substantial reproduction and deployment costs remaining.