FoundationGeo: Learning Spatial Pixel-Wise Fields for Monocular Metric Geometry

Conference: ECCV2026
Paper: ECCV Paper
Project: FoundationGeo
Area: 3D Vision
Keywords: monocular depth estimation, metric geometry, pixel-wise scale fields, ray-direction correction, focal-length coverage

TL;DR

FoundationGeo learns a strong relative point map, recovers physical scale through pixel-wise scale fields and ray-direction correction, and repairs camera-distribution gaps with focal-diverse synthetic data, reaching average AbsRel of 14.8% and \(\delta_1\) of 80.8% across seven zero-shot metric-depth benchmarks.

Background & Motivation

A single RGB image provides rich shape and occlusion cues but cannot uniquely determine physical object size and distance. Relative depth models can consequently recover sharp contours and plausible depth ordering while retaining scale and shift ambiguity, making their predictions unsuitable for direct measurement. Metric depth estimation must exploit additional camera and scene priors to convert this relative structure into geometry expressed in physical units.

MoGe-2 provides an effective starting point: retain a strong relative geometry predictor and learn an image-level scale. However, Figure 2 in FoundationGeo shows that scale errors need not be uniform across the image. Fitting optimal scales over progressively smaller patches keeps reducing error. Ray-direction bias is a separate problem: even a point at the correct distance from the camera may lie in the wrong direction. A single scalar can neither rescale different regions independently nor rotate a ray.

Correcting these within-image errors still leaves a mismatch between training and test cameras. More training images do not necessarily cover more focal lengths, so a model can continue relying on a biased implicit camera prior. Core Idea: learn relative shape first, calibrate local scale and ray direction separately, and supplement under-covered focal regimes to distinguish within-image geometry errors from cross-camera distribution errors.

Method

Overall Architecture

The input is a single RGB image, with no ground-truth camera intrinsics required at inference time. In the first stage, a DINOv3-initialized ViT-Large and a lightweight upsampling convolutional decoder predict an affine-invariant point map and a validity mask. The second stage inherits these weights and adds two lightweight spatial-field outputs: direction correction followed by pixel-wise scaling produces a metric point map, from which depth and surface normals are derived.

Focal-coverage augmentation is a second-stage training-data intervention, not an additional focal-length input at test time. Nor does the second stage freeze the relative branch and fit a few scalars: the backbone and prediction heads remain jointly optimized, while relative geometry supervision remains active to preserve local structure during metric fine-tuning.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Single RGB image"] --> B["Relative Geometry Base"]
    B --> C["Ray-Direction Correction"]
    C --> D["Pixel-Wise Scale Calibration"]
    D --> E["Metric point map<br/>Depth and surface normals"]
    F["Focal-Coverage Augmentation"] -.->|Second-stage training| C
    F -.->|Second-stage training| D

Key Designs

1. Relative Geometry Base: establish reliable global structure and detail

The relative branch does not immediately regress physical scale. Instead, it learns point maps under scale-and-shift alignment. A global alignment loss constrains scene layout, multi-scale local patch losses preserve boundaries and high-frequency structure, surface-normal consistency constrains surface shape, and a validity-mask loss handles unreliable regions. Edge supervision is enabled only on detail-rich datasets rather than imposing potentially poor boundary labels equally across all sources.

Features from different encoder levels are fused through element-wise summation, supplying both semantic and local cues to the convolutional decoder. The purpose is not to build a much larger decoder, but to provide reliable shape for subsequent calibration. Multiplication by a scale field cannot by itself recover an object boundary that the relative point map has already blurred away.

The corpus spans indoor, outdoor, driving, synthetic, and in-the-wild imagery, with 19 datasets and approximately 10.2M frames. Filtering removes severe motion blur, abnormal exposure, and overhead views without useful near-far ordering; depth is clipped or invalidated according to dataset-specific physical ranges. Not every source in Table 1 supplies metric supervision, so relative losses also help exploit heterogeneous data.

2. Ray-Direction Correction: change direction while preserving range

Each predicted 3D point is decomposed into its Euclidean distance from the camera and a unit ray direction. This range is not the camera-coordinate depth component. The model constructs two orthonormal basis vectors in the plane perpendicular to the ray, predicts two perturbation coefficients, and bounds them with tanh. A reference axis that is not nearly parallel to the ray avoids a degenerate cross product during basis construction.

After adding the tangent perturbation, the direction is normalized and multiplied by the original range, changing direction without changing distance. This establishes a clear division of labor: the scale field moves points along rays, while the direction field changes where the rays point. The operation described in Section 3.2 is:

\[ \delta_{k,i}=\delta_{\max}\tanh(\hat\Delta_{k,i}),\qquad \hat{\mathbf r}'_i= \frac{\hat{\mathbf r}_i+\delta_{1,i}\mathbf b_{1,i}+\delta_{2,i}\mathbf b_{2,i}} {\|\hat{\mathbf r}_i+\delta_{1,i}\mathbf b_{1,i}+\delta_{2,i}\mathbf b_{2,i}\|_2},\qquad \hat{\mathbf p}'_i=\|\hat{\mathbf p}_i\|_2\hat{\mathbf r}'_i. \]

The tangent basis vectors are orthonormal unit vectors. Training additionally penalizes angular disagreement with ground-truth rays and unnecessary correction magnitude; the paper uses a quadratic penalty, with \(q=2\). This constrains the direction branch more tightly than unrestricted coordinate updates, although it does not guarantee improved depth scores on every dataset.

3. Pixel-Wise Scale Calibration: constrain local rescaling with direct supervision

A scale output block reads shared point-map branch features and predicts one scale per pixel instead of one per image. The metric point is the direction-corrected relative point multiplied by its predicted scale. Local scales can accommodate spatially varying drift, but also introduce an identifiability problem: if only the final metric point is supervised, the point-map and scale branches can compensate for each other, obscuring which branch learns shape and which learns calibration.

The authors therefore construct a scale target using the least-squares projection of the ground-truth point onto the direction-corrected prediction. The target is clamped to a physical range and supervised in the log domain with a Huber penalty. This closed-form solution supplies a training target; it is not an inference procedure that uses ground-truth points:

\[ \tilde{\mathbf p}_i=\hat s_i\hat{\mathbf p}'_i,\qquad s_i=\frac{(\hat{\mathbf p}'_i)^\top\mathbf p_i}{\|\hat{\mathbf p}'_i\|_2^2},\qquad \log s_i^*=\log\operatorname{clamp}(s_i,s_{\min},s_{\max}). \]

Here, \(\mathbf p_i\) is the ground-truth 3D point. Coupled metric regression still applies an L1 loss to the final predicted point, weighted by inverse ground-truth depth to emphasize near-range geometry. Field-specific supervision separately anchors scale and direction, reducing the role ambiguity left by supervision on final point error alone.

4. Focal-Coverage Augmentation: repair camera-distribution gaps rather than merely add volume

The authors examine the 50 most frequent training focal values and find that several test datasets occupy an under-covered band of approximately 1000-1373 px. Appearance, depth, and focal length are coupled in monocular images. Even without receiving explicit intrinsics, a network learns an implicit focal prior from its data and can systematically over-scale or under-scale predictions outside that prior.

A Blender engine generates 23,700 images across 7 indoor and outdoor scenes, varying focal lengths uniformly over the missing regime for metric training. To distinguish the value of synthetic images from that of focal diversity, the authors keep scenes and camera trajectories fixed and compare the diverse-focal version against a fixed 1320 px version. The latter improves some datasets but worsens the overall average, supporting camera coverage rather than simple data expansion as the useful intervention.

Loss & Training

The second stage retains relative geometry supervision and adds final metric-point regression, log-scale-field Huber supervision, ray-angle Huber supervision, and correction-magnitude regularization. Field-level averages over valid pixels use inverse ground-truth depth weights. The scale, ray, and magnitude-regularization weights are 0.2, 0.1, and 0.05, respectively.

Stage one runs for 55K iterations, with initial encoder and decoder learning rates of \(1\times10^{-5}\) and \(1\times10^{-4}\), halved every 20K iterations. Stage two runs for 20K iterations, with backbone and head learning rates of \(1\times10^{-6}\) and \(1\times10^{-5}\). Full training uses 32 NVIDIA H20 GPUs. Both stages apply color jittering, Gaussian blur, JPEG compression-decompression, and random cropping.

Some loss equations in the cached PDF text are damaged by extraction. This note therefore describes those losses using verifiable prose and does not invent Huber thresholds, scale-clamping bounds, or a numerical maximum direction perturbation; the paper refers these details to its supplementary material.

Key Experimental Results

Main Results

Metric evaluation in Table 2 covers seven held-out datasets: NYUv2, KITTI, ETH3D, iBims-1, DDAD, DIODE, and HAMMER. Relative evaluation additionally includes Sintel, making eight datasets. Relative predictions undergo global scale-and-shift alignment, so their scores must not be interpreted as physical-scale errors.

AbsRel averages \(|\tilde d-d|/d\) over valid pixels; \(\delta_1\) is the fraction satisfying \(\max(d/\tilde d,\tilde d/d)<1.25\). Both are reported as percentages below, and lower average rank is better.

Method (paper Table 2, metric depth)	Seven-dataset mean AbsRel, lower is better	Seven-dataset mean \(\delta_1\), higher is better	Average rank
UniDepth V2	21.3	75.3	3.25
DepthPro	27.6	54.4	5.36
Metric3D V2, requires ground-truth intrinsics	18.3	73.9	Excluded
MoGe-2	15.7	76.8	2.82
FoundationGeo	14.8	80.8	2.32

Against MoGe-2, mean AbsRel falls by 0.9 percentage points, approximately a 5.7% relative reduction; \(\delta_1\) rises by 4.0 percentage points, approximately a 5.2% relative increase. Metric3D V2 receives ground-truth intrinsics and is included only for reference; the paper also excludes training-overlap benchmark results from ranking. DA V1/V2 comparisons select the better indoor/outdoor variant rather than evaluating a single unified release.

Ablation Study

Table 4 uses approximately 5% stratified training subsets, the same ViT-Large, and 8 H20 GPUs. The single-stage baseline trains for 40K iterations; two-stage variants use 20K relative plus 20K metric iterations. These results must not be mixed directly with the full-training score of 14.8/80.8.

Config (paper Table 4)	Seven-dataset mean AbsRel, %	Seven-dataset mean \(\delta_1\), %
Single-stage direct prediction	22.4	57.7
Two-stage direct prediction	19.6	65.9
Two-stage + global scale	20.1	66.8
Two-stage + spatial fields	19.3	68.0
Two-stage + spatial fields + field losses	18.8	69.7

The following selection from Table 5 uses full data. FGD denotes the authors' rendered FoundationGeo dataset; fixed-focal and diverse-focal versions use identical scenes and camera trajectories.

Config (paper Table 5)	Seven-dataset mean AbsRel / \(\delta_1\), %	DIODE AbsRel / \(\delta_1\), %
Without FGD	15.0 / 79.5	20.5 / 55.2
With fixed-focal FGD	15.3 / 78.9	18.5 / 63.9
With diverse-focal FGD, full model	14.8 / 80.8	17.5 / 69.7
MoGe-2	15.7 / 76.8	17.5 / 66.4
MoGe-2 + FGD	15.1 / 79.0	16.2 / 72.1

Key Findings

• In Table 4, switching from single-stage to two-stage direct prediction adds 8.2 percentage points of \(\delta_1\). This exceeds the subsequent gains from spatial fields or field losses along this cumulative comparison, but is not an independent factorial attribution.
• Table 6 averages only NYUv2, DIODE, DDAD, and HAMMER. Ray correction lowers mean angular error MaeDeg from 2.889 degrees to 2.355 degrees and raises Pct3, the percentage within 3 degrees, from 60.5% to 71.0%. Corresponding mean \(\delta_1\) rises from 72.8% to 76.8%.
• In Table 2, FoundationGeo-Base achieves eight-dataset relative-depth mean AbsRel / \(\delta_1\) of 4.87% / 96.0%. In Table 3, raising the 313M-parameter Base model's budget from 3600 to 6000 tokens increases HAMMER boundary F1 from 4.93% to 5.78%; this is not metric-depth accuracy.

Highlights & Insights

• Splitting point-map error into direction and distance has explicit geometric meaning. Bounded tangent perturbations followed by range restoration prevent direction correction from simultaneously performing arbitrary scaling.
• The closed-form scale is a source of training supervision, not a replacement inference algorithm. It gives the auxiliary head a direct target instead of relying entirely on indirect gradients from final 3D error.
• The focal experiment offers an actionable data-collection principle. Varying camera coverage with scenes and trajectories held fixed tests cross-camera generalization more directly than simply increasing dataset size.

Limitations & Future Work

• The authors acknowledge possible interference between jointly optimized heads. In Table 6, HAMMER ray-angle metrics improve while \(\delta_1\) falls from 72.9% to 69.6%; better directions do not imply better depth on every dataset.
• Leading averages do not mean leading each benchmark. Table 2 reports ETH3D metric AbsRel of 17.8%, worse than MoGe-2's 10.4%; NYUv2 is likewise 10.2% versus 7.33%. The advantage is primarily more balanced aggregate performance.
• As an experimental-scope limitation, correlation between focal coverage and error cannot rule out scene content, sensor noise, and other covariates. Controlled rendering strengthens the focal explanation, but 7 synthetic scenes do not establish reliability for arbitrary optics or distortion.
• This is not a compression or lightweight-deployment method: full training requires 32 H20 GPUs, and the main paper provides no comprehensive latency comparison. The authors propose temporal or multi-view observations to reduce monocular ambiguity and enforce cross-frame geometry consistency.

Related Work & Insights

• vs MoGe / MoGe-2: FoundationGeo inherits relative point maps and local geometric supervision, replacing image-level scaling with pixel-wise scaling and direction correction. FGD also benefits MoGe-2, indicating that the data contribution is not exclusive to this architecture.
• vs Metric3D V2: In this comparison, Metric3D V2 receives ground-truth intrinsics; FoundationGeo instead learns implicit camera priors through training coverage without requiring that input. Their information conditions differ, so averages alone are insufficient for comparison.
• vs UniDepth / DepthPro: These approaches explicitly involve camera-parameter estimation or focal modeling. FoundationGeo emphasizes relative-to-metric transfer and data-driven camera coverage, illustrating complementary model-design and data-design routes to better camera priors.

Rating

• Novelty: 4/5. Clearly separates direction and scale, and turns camera-distribution analysis into controlled augmentation.
• Experimental Thoroughness: 4/5. Includes cross-domain comparisons, subset component ablations, full-data studies, and ray ablations, but lacks comprehensive deployment evaluation.
• Writing Quality: 4/5. The problem decomposition is clear; per-dataset tables support more qualified conclusions than some broad claims of superiority.
• Value: 4/5. Useful for relative-to-metric geometry transfer and camera-aware data-collection strategies.