Honey, I Shrunk the Arc de Triomphe!¶
Conference: ECCV 2026
Paper: ECCV Official
Project Page: https://metricscenes.github.io/
Area: 3D Vision / Autonomous Driving
Keywords: In-the-wild metric-scale dataset, Monocular metric geometry, Monocular metric depth, Scale-collapse, Poisson completion
TL;DR¶
To eliminate the severe "scale-collapse" flaw where monocular geometry foundation models metrically underestimate distant structures and landmarks in the wild, this work builds MetricScenes—the first large-scale real-world metric dataset anchored by geo-tagged street-view imagery and physical stereo baselines—and develops a two-stage edge-aware Poisson depth completion strategy, yielding WildMoGe which restores faithful absolute physical dimensions without sacrificing standard benchmark accuracy.
Background & Motivation¶
Determining absolute physical scale and 3D scene geometry from an uncalibrated single image is an intrinsically ill-posed inverse problem. When humans interpret real-world scenes, they naturally deduce absolute dimensions from visual contextual priors, such as reference objects of known physical scale like human heights, door frames, or vehicle wheelbases. Modern monocular geometry and depth estimation foundation models (e.g., MoGe-2, Metric3D v2, UniDepth v2) are theoretically expected to internalize these semantic deductions. However, when deployed in unconstrained, in-the-wild environments, existing state-of-the-art models systematically suffer from a persistent and severe "scale-collapse" phenomenon: while the predicted metric scale for near-field objects remains plausible (e.g., tourists standing in the foreground have realistic ~1.4 m heights), background architectural structures and distant landmarks are dramatically shrunken and pulled unnaturally close to the camera. For instance, when presented with a tourist photo of the Arc de Triomphe whose physical width is 44.8 m, MoGe-2 predicts a metric point cloud measuring merely 18.8 m wide—underestimating reality by more than a factor of two despite explicit visual cues.
This critical performance gap does not stem from flaws in deep neural architectures, but rather from a fundamental data bottleneck imposed by sensor constraints. Existing metric-scale datasets are restricted to two narrow categories: vehicle-mounted terrestrial LiDAR scans (such as KITTI and Argoverse 2), which are strictly confined to urban roadway corridors and constitute 59% of outdoor metric training frames, and short-range active 3D scanner captures (such as ScanNet++ and ARKitScenes), which are limited by indoor room boundaries and a few meters of effective sensor range. Conversely, synthetic datasets lack the semantic and physical complexity of real environments, while internet-scale multi-view photo collections (such as MegaDepth and BlendedMVS) offer rich semantic diversity but are reconstructed via conventional Structure-from-Motion (SfM) only up to an arbitrary, unknown scale factor. The community has thus lacked a large-scale real-world dataset that simultaneously offers unconstrained semantic diversity and verifiable absolute metric grounding.
This paper breaks free from dedicated range-sensing hardware by unlocking the vast, freely accessible repositories of internet photo collections and consumer stereo video recordings. Core idea: Anchor arbitrary-scale SfM reconstructions to absolute metric dimensions using geo-referenced street-view photography in an ECEF frame, scale stereo video sequences via physical camera baselines optimized with multi-view spatiotemporal geometry, and resolve gradient-anchor scale conflicts through a two-stage edge-aware Poisson completion pipeline to construct MetricScenes and train WildMoGe.
Method¶
Overall Architecture¶
The pipeline for curating the MetricScenes dataset and training the WildMoGe geometry model consists of three interconnected stages. First, for unstructured internet photo collections sourced from AerialMegaDepth and MegaScenes, the pipeline uses Doppelganger classifiers to eliminate perceptual ambiguities from repetitive architectural facades, computes initial multi-view stereo (MVS) depth, and registers geo-tagged street-level panoramic views to align the 3D reconstructions into a Manhattan frame and compute absolute metric scale via RANSAC against Earth-Centered, Earth-Fixed (ECEF) coordinates. Second, for stereoscopic sequences from Stereo4D, clips with documented camera specifications are processed with the permutation-equivariant multi-view geometry model \(\pi^3\) across temporal frames, aligning the reconstructed camera trajectories with known physical baselines to recover dense metric depth. Third, recognizing that MVS depth maps lack dynamic foregrounds while stereo depth lacks distant backgrounds, a two-stage edge-aware Poisson solver fuses sparse metric anchors with monocular gradient fields, producing complete, artifact-free, and metrically consistent supervision maps to fine-tune the MoGe-2 foundation model into WildMoGe.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Uncalibrated Multi-Source Imagery<br/>Internet Photo Collections / Stereo4D Videos"] --> B["Geo-Registration & Baseline Scaling<br/>Street-view ECEF alignment + Stereo baseline scaling"]
B --> C["Geometric Consistency & Drift Filtering<br/>Doppelganger pruning + Stereo reprojection verification"]
C --> D["Two-Stage Edge-Aware Poisson Completion<br/>Background metric solve + Foreground edge-weighted stitching"]
D --> E["MetricScenes Dataset Delivery<br/>99.7k dense metric depth maps"]
E --> F["WildMoGe Fine-Tuning<br/>MoGe-2 ViT-Large multi-scale geometry optimization"]
Key Designs¶
1. Geo-Metadata Registration and Stereo Baseline Scaling: Sensorless Absolute Scale Grounding
To infuse absolute physical dimensions into unscaled SfM point clouds without ground LiDAR hardware, the authors sample approximately 100 uniformly distributed viewpoints along roads surrounding each target landmark from online mapping services. Using MASt3R feature descriptors matched against the reconstructed landmark model and filtered with a Doppelganger classifier to prune spurious correspondences from repetitive patterns, COLMAP's model aligner rotates the scene into an axis-aligned Manhattan world. A robust RANSAC estimator then computes a similarity transformation mapping the recovered 3D camera centers directly into ECEF coordinates, fixing the global metric scale. For Stereo4D stereoscopic videos, instead of running optical flow (SEA-RAFT) or single-pair matching (FoundationStereo) which frequently distort geometry in uncalibrated wide-angle rigs, the pipeline feeds \(N=16\) surrounding frames into \(\pi^3\) to jointly estimate camera poses and scene structure. The absolute metric scale factor \(s\) is computed by aligning the predicted inter-camera translation against the known ground-truth physical baseline \(b_{\text{gt}}\): $\(s = \frac{b_{\text{gt}}}{\frac{1}{N+1}\sum_{i=0}^{N}\|\mathbf{t}_{\text{L}}^{(i)} - \mathbf{t}_{\text{R}}^{(i)}\|_2}\)$ Scaling raw predictions by \(d_{\text{metric}} = s \cdot d_{\text{pred}}\) reliably yields absolute metric depth across all valid pixels.
2. Geometric Consistency and Multi-Tier Drift Filtering: Eliminating Spurious Visual Artifacts
Unconstrained web images and videos contain dynamic pedestrians, vehicles, and fluctuating illumination that would corrupt depth supervision if directly utilized. For internet photo collections, after MVS reconstruction, a stability filtering step prunes unstable pixels exhibiting high depth variance across neighboring camera viewpoints, while MoGe-2's initial predictions are leveraged to detect and excise depth-bleeding regions where distant background depth erroneously bleeds into foreground silhouettes. For stereoscopic sequences, frames are screened based on median intensity and Tenengrad gradient energy to reject underexposed or motion-blurred inputs. Crucially, a stereo depth reprojection check discards any frame where more than 10% of valid pixels exhibit disparity inconsistency between left and right views, or where the reconstructed baseline deviates by \(>10\text{ mm}\) or relative rotation exceeds \(1^\circ\). Furthermore, because monocular models are highly reliable on near-field objects, sequences are discarded if the derived metric scale deviates from MoGe-2's near-field predictions by more than a factor of \(2\times\) or falls below \(0.5\times\).
3. Two-Stage Edge-Aware Poisson Completion: Resolving the Anchor-Gradient Scale Tension
Standard logarithmic-space Poisson depth completion minimizes gradient differences between a dense guide map and sparse boundary anchors: $\(\min_{d^*} \sum_{i \in \Omega} \|\nabla(\log d^*_i) - \nabla(\log \hat{d}_i)\|^2 \quad \text{s.t.} \quad d^*_i = d^{\text{gt}}_i, \, \forall i \in \partial\Omega\)$ However, using monocular predictions \(\hat{d}\) (which suffer from scale-collapse) as global gradient guidance creates severe distortions: anchoring on background MVS alone forces the solver to propagate scale corrections across the entire continuous field, expanding foreground pedestrians into implausible \(3.26\text{ m}\) giants (BG only). Simultaneously enforcing foreground monocular anchors and background MVS anchors in a single stage warps unanchored transition zones (such as building plinths) forward to satisfy the collapsed gradient field (FG+BG). To resolve this, the authors propose a decoupled two-stage strategy: Stage 1 solves the Poisson equation using solely filtered MVS background anchors, obtaining a metrically accurate distant background while discarding the distorted foreground. Stage 2 takes this completed background and the reliable MoGe-2 foreground as joint dense anchors, optimizing an edge-weighted objective: $\(\min_{d^*} \sum_{i \in \Omega} \sum_{j \in \mathcal{N}(i)} w_{ij} \left| (\log d^*_i - \log d^*_j) - (\log \hat{d}_i - \log \hat{d}_j) \right|^2\)$ where the weight \(w_{ij}\) vanishes across sharp depth discontinuities. This edge weighting preserves crisp foreground boundaries while locking background landmarks to their true physical dimensions without tearing or bleeding artifacts.
Loss & Training¶
Fine-tuning is conducted on the curated MetricScenes dataset containing 99,711 dense, metrically grounded frames. WildMoGe builds upon the MoGe-2 ViT-Large-Normal backbone. Training runs for 10,000 iterations with a batch size of 32 (approximately 3 epochs) using the original MoGe-2 data augmentations and image cropping regimes. To preserve learned geometric priors from extensive synthetic pre-training while adapting the metric scale distribution, the backbone is fine-tuned with a cautious learning rate of \(1 \times 10^{-6}\), while the remaining prediction heads are optimized with a learning rate of \(1 \times 10^{-5}\).
Key Experimental Results¶
Main Results¶
WildMoGe is benchmarked across two categories: 10 standard evaluation datasets (NYUv2, KITTI, ETH3D, iBims-1, GSO, Sintel, DDAD, DIODE, Spring, HAMMER) and the held-out MetricScenes test set comprising in-the-wild landmarks and stereo scenes.
Table 1: Quantitative evaluation of relative and metric geometry (selected from paper Table 1)
| Benchmark Setting / Method | Scale-inv. Point \(\text{Rel}_p\downarrow\) | Scale-inv. Depth \(\text{Rel}_d\downarrow\) | Metric Point (w/o GT Cam) \(\text{Rel}_p\downarrow\) | Metric Depth \(\text{Rel}_d\downarrow\) | Inlier Threshold \(\delta_{d1}\uparrow\) |
|---|---|---|---|---|---|
| Standard Benchmarks | |||||
| Metric3D v2 | - | 7.92 | - | 18.3 (w/ GT Cam) | 73.9% |
| UniDepth v2 | 11.6 | 8.61 | 10.1 | 21.3 | 75.3% |
| Depth Pro | 12.4 | 9.81 | 13.7 | 27.6 | 54.4% |
| MoGe-2 (Baseline) | 10.8 | 7.35 | 8.19 | 15.7 | 76.8% |
| WildMoGe (Ours) | 10.1 | 7.49 | 8.39 | 15.6 | 73.4% |
| MetricScenes Test Set (In-the-wild) | |||||
| MoGe-2 (Baseline) | 6.76 | 4.85 | 11.7 | 37.2 | 37.7% |
| WildMoGe (Ours) | 5.24 | 4.02 | 7.59 | 26.5 | 73.8% |
Note: On standard indoor and driving datasets, WildMoGe matches MoGe-2 within marginal variance. On the MetricScenes test set containing open-domain landmarks and vast scenes, WildMoGe improves metric depth inlier ratio \(\delta_{d1}\) from 37.7% to 73.8% and slashes relative metric depth error from 37.2% to 26.5%.
Ablation Study¶
Ablation experiments evaluate the unique contributions of individual data sources within MetricScenes toward overall geometric robustness, scale grounding, and boundary sharpness.
Table 2: Ablation of fine-tuning MoGe-2 on MetricScenes subsets (from paper Table 2)
| Subset Configuration | Standard \(\text{Rel}_p\downarrow\) | Standard \(\text{Rel}_d\downarrow\) | Boundary Sharpness \(F_1\uparrow\) | MetricScenes \(\text{Rel}_p\downarrow\) | MetricScenes \(\text{Rel}_d\downarrow\) | MetricScenes \(\delta_{d1}\uparrow\) |
|---|---|---|---|---|---|---|
| Stereo4D only | 11.3 | 7.50 | 15.6 | 6.23 | 34.0 | 39.3% |
| AerialMegaDepth only | 9.81 | 7.41 | 14.5 | 6.33 | 41.0 | 58.7% |
| MegaScenes only | 9.78 | 7.45 | 15.0 | 6.68 | 32.4 | 53.0% |
| AerialMD. + MegaSc. | 9.73 | 7.40 | 14.9 | 6.66 | 32.7 | 65.0% |
| All Subsets (Full Corpus) | 10.1 | 7.49 | 15.5 | 5.24 | 26.5 | 73.8% |
Key Findings¶
- Synergistic Multi-Source Complementarity: Stereo4D provides fine-grained disparity transitions that boost boundary sharpness (\(F_1=15.6\)); AerialMegaDepth acts as an authoritative large-scale metric anchor derived from Google Earth geometry; MegaScenes injects rich viewpoint and semantic variety. Combining all sources yields the lowest error (\(\text{Rel}_p=5.24\)) and highest inlier rate (73.8%), proving that unconstrained geometry requires multi-modal supervision.
- Physical Grounding vs. Scale Inflation Bias: When tested on the ETH3D courtyard, ground-truth desk legs stand 72 cm tall; baseline MoGe-2 overestimates this height at 81 cm, whereas WildMoGe predicts 71.6 cm. In typical indoor and street scenes, both models predict identical door heights (2.1 m) and car lengths (~3.2 m). This confirms that WildMoGe's gains reflect true physical grounding across distance regimes, rather than a trivial bias toward predicting larger numbers.
Highlights & Insights¶
- Hardware-Free Metric Grounding: Demonstrates that physical LiDAR hardware is not a prerequisite for building metric foundation datasets; geo-tagged panoramic mapping metadata and commodity stereo camera baselines can supervise absolute physical scale at planetary scale.
- Two-Stage Decoupled Depth Completion: Ingeniously addresses the failure of classical Poisson solvers under scale-inconsistent gradient guidance, providing an elegant decoupling paradigm ("solve background, reuse foreground, damp edges") applicable to general multi-modal depth fusion.
- Robustness Across Dramatic Distance Spans: Resolves scale-collapse across novel open-domain landmarks (e.g., Philadelphia Museum of Art at 31.4 m vs. 32.4 m ground truth, Piazza della Signoria at 46.7 m vs. 46.5 m ground truth), expanding the operating envelope of monocular 3D vision from local rooms to monumental outdoor environments.
Limitations & Future Work¶
- Featureless Environments: In open-domain scenes devoid of salient landmarks, recognizable objects, or textured surfaces (such as open deserts or vast oceanic expanses), estimating absolute metric scale from a single viewpoint remains inherently ambiguous.
- Pseudo-Ground-Truth Imperfections: Supervision depth maps are constructed via multi-view stereo and Poisson completion rather than direct time-of-flight LiDAR scans, leaving subtle residual noise in complex occlusions and ultra-thin structures.
- End-to-End Integration: Future directions include embedding Poisson boundary regularization as a differentiable neural layer within the model architecture, or leveraging Vision-Language Models (VLMs) to provide explicit semantic prior reasoning during depth decoding.
Related Work & Insights¶
- vs MoGe-2 / Metric3D v2: While existing foundation models adopt virtual camera normalizations or decoupled scale tokens, their training corpora remain biased toward driving corridors and indoor rooms, inevitably triggering scale-collapse on open-domain landmarks. WildMoGe directly resolves the underlying training data imbalance.
- vs UniDepth v2 / Depth Pro: UniDepth v2 uses self-prompted camera embeddings but still exhibits significant metric drift on distant monuments; Depth Pro produces sharp boundaries but often underestimates physical scale by orders of magnitude on far-field scenes. WildMoGe maintains both boundary sharpness and absolute physical scale fidelity.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Identifies and formalizes the scale-collapse phenomenon; presents an innovative pipeline to ground internet imagery to absolute metric scale without range sensors]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Comprehensive evaluations across 10 established benchmarks and a novel held-out in-the-wild test set, corroborated by physical landmark measurements]
- Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, rigorous mathematical exposition of the Poisson failure modes, and convincing visual and tabular evidence]
- Value: ⭐⭐⭐⭐⭐ [Provides a transformative dataset curation blueprint for 3D foundation models, autonomous systems, and embodied spatial AI]