Skip to content

Argus: Metric Panoramic 3D Reconstruction for Indoor Scenes

Conference: ECCV 2026
Paper: ECCV Official
Project: https://argus-paper.realsee.ai
Area: 3D Vision
Keywords: Panoramic 3D Reconstruction, Metric Scale, Feed-Forward Network, Covisibility Reference Selection, Geometric Factorization Supervision

TL;DR

Addressing reference drift and scale ambiguity in unordered panoramic image reconstruction, Argus introduces a learnable covisibility-guided reference selection module and overcomplete geometric factorization supervision, achieving state-of-the-art feed-forward metric indoor 3D reconstruction on the large-scale Realsee3D benchmark.

Background & Motivation

Panoramic cameras, with their continuous \(360^\circ \times 180^\circ\) horizontal and vertical fields of view, have rapidly emerged as the preferred imaging modality for indoor digital twinning, robotic spatial perception, and embodied navigation. Unlike narrow field-of-view perspective cameras, a single omnidirectional panorama observes almost all visible room surfaces, and a small collection of viewpoints establishes extensive spatial overlap. This characteristic naturally aligns with the sparse-view feed-forward paradigm by allowing complete scene geometry recovery from only a few captures. Nevertheless, modern feed-forward 3D reconstruction foundation models (such as DUSt3R, VGGT, and MapAnything) are overwhelmingly trained on perspective datasets. Due to severe equirectangular projection (ERP) polar distortion and non-uniform pixel sampling, directly transferring these models to panoramic inputs triggers severe geometric fragmentation and scale collapse. Moreover, research has long been bottlenecked by the absence of large-scale indoor panoramic RGB-D benchmarks with rigorous metric-scale annotations.

In realistic indoor panoramic survey workflows, captured images are typically sparse and delivered as an unordered set without sequential trajectory continuity. Standard multi-view feed-forward architectures anchor their global world coordinate frame to a pre-designated reference frame (often arbitrarily fixed to the first image). If that anchor happens to be an isolated viewpoint located in a room corner, behind a door, or near a occluding boundary, the limited cross-view visual overlap severely degrades geometric constraints, inducing catastrophic global pose drift and accumulated structural distortion. Concurrently, regressing 3D world-coordinate point clouds directly from 2D pixels in an end-to-end black box bundles perspective unprojection, depth prediction, relative rotation, and spatial translation into a single task. This entanglement creates conflicting gradients across multi-task objectives and fails to enforce strict geometric consistency between camera-local and world spaces.

This work targets this fundamental limitation by leveraging the intrinsic topological covisibility of panoramic imagery to establish an optimal coordinate anchor, alongside an explicit forward-and-inverse decomposition of the pixel-to-world transformation pipeline. Core idea: dynamically select the geometrically optimal reference view via a lightweight Covisibility Transformer to anchor the absolute metric world frame, combined with overcomplete geometric factorization and cross-coordinate joint supervision to achieve robust, single-pass feed-forward metric indoor 3D reconstruction.

Method

Overall Architecture

Argus takes an unordered set of sparse indoor ERP panoramic images as input and directly predicts absolute metric camera poses, dense metric depth maps, and multi-coordinate 3D point maps in a single feed-forward pass. The pipeline operates in three distinct phases: first, DINOv2 extracts per-frame patch tokens, which are augmented with covisibility tokens and fed into a lightweight Covisibility Transformer to predict global connectivity scores for dynamic reference selection; next, the reordered token sequence is processed by a deep Geometry Transformer featuring alternating intra-view and inter-view self-attention layers for comprehensive cross-view reasoning; finally, dedicated MLP and multi-scale DPT prediction heads regress camera poses, metric depth, and intermediate point clouds, while bidirectional cross-coordinate transformations enforce closed-loop consistency during training.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Unordered Panoramic Images<br/>ERP preprocessing and polar cropping"] --> B["Covisibility Reference View Selection<br/>Predict global connectivity and select optimal anchor"]
    B --> C["Cross-View Geometric Reasoning<br/>Geometry Transformer with alternating attention"]
    C --> D["Overcomplete Geometric Factorization<br/>Pose/depth/camera points/rotated points/world points"]
    D --> E["Bidirectional Cross-Coordinate Joint Constraints<br/>Closed-loop transformations and gradient synergy"]
    E --> F["Metric-Scale 3D Scene Reconstruction"]

Key Designs

1. Covisibility Reference View Selection: Eliminating Global Pose Drift in Unordered Capture Standard feed-forward architectures are highly sensitive to input sequence order, where an arbitrary boundary anchor leads to severe trajectory drift. Argus introduces a lightweight Covisibility Transformer with \(L_c = 2\) alternating attention layers (alternating self-attention within each view and across views). Taking patch tokens and dedicated covisibility tokens as input, an MLP predicts global covisibility scores \(\hat{\mathbf{C}} \in \mathbb{R}^N\). Ground-truth labels are computed from pose and depth visibility graphs using Dijkstra's shortest-path algorithm and supervised with binary cross-entropy loss \(\mathcal{L}_{\text{covis}}\). During inference, the reference frame \(\hat{I}\) is determined via an \(\arg\max\) operation and assigned a specialized reference camera token. Because \(\arg\max\) is permutation-invariant, the system consistently selects the visual hub near the room center regardless of how the input sequence is shuffled, stabilizing the optimization landscape and suppressing long-range pose drift.

2. Overcomplete Geometric Factorization: Explicitly Decoupling Pixel-to-World Mappings Directly predicting world-coordinate point maps conflates projection geometry, relative motion, and depth estimation into an entangled optimization problem. Leveraging the intrinsic properties of ERP panoramas, Argus decomposes the single-view mapping into four interpretable physical steps: mapping pixels \((u, v)\) to unit sphere directions \(P_u\), scaling by metric depth \(D\) to obtain camera-space points \(P_c = D \odot P_u\), rotating by camera rotation \(R\) to reference-aligned rotated points \(P_r = R P_c\), and translating by \(t\) into world coordinates \(P_w = P_r + t\), alongside exact inverse formulations. At the network output, independent DPT heads predict depth \(\hat{D}\), camera points \(\hat{P}_c\), rotated points \(\hat{P}_r\), and world points \(\hat{P}_w\). Each head predicts an associated uncertainty confidence map and activates 3D coordinates via an inverse log transform \(f(x) = \text{sign}(x) \cdot (\exp(|x|) - 1)\) to accommodate wide dynamic physical ranges without numerical collapse.

3. Bidirectional Cross-Coordinate Joint Constraints: Enforcing Multi-Branch Closed-Loop Consistency To prevent independent prediction heads from diverging into mutually inconsistent representations, Argus establishes an overcomplete joint supervision loss \(\mathcal{L}_{\text{joint}}\). The output from one branch is transformed using the predicted motion parameters of another branch to reconstruct intermediate geometric states and supervised against ground-truth targets: $$ \mathcal{L}{\text{joint}} = \mathcal{L}_p\bigl(\Phi}(\hat{D})\bigr) + \mathcal{Lp\bigl(\Phi}(\hat{Pc, \hat{R})\bigr) + \mathcal{L}_p\bigl(\Phi}(\hat{Pr, \hat{t})\bigr) + \mathcal{L}_p\bigl(\Phi}(\hat{Pw, \hat{t})\bigr) + \mathcal{L}_p\bigl(\Phi}(\hat{Pr, \hat{R})\bigr) + \mathcal{L}_d\bigl(\Phi_c)\bigr) $$ Because all transformations operate on real-time network predictions }(\hat{P\((\hat{D}, \hat{P}_c, \hat{P}_r, \hat{P}_w, \hat{R}, \hat{t})\), gradients circulate dynamically between the pose, depth, and spatial point heads. Importantly, during inference deployment, the intermediate point map heads can be completely deactivated, requiring only the pose and depth heads to generate final reconstructions with zero added computational overhead.

4. Direct Absolute Metric Scale Regression: Resolving Intrinsic Scale Ambiguity In perspective imaging, focal length and depth are inherently coupled, typically requiring extra scale-invariant alignment or auxiliary metric regression modules. In contrast, the equirectangular projection model defines a fixed geometric mapping where pixel coordinates \((u, v)\) map deterministically to spherical angles \(\theta = (\frac{v}{H} - 0.5)\pi\) and \(\phi = (\frac{u}{W} - 0.5)2\pi\). By training on Realsee3D's high-fidelity metric annotations (derived from LiDAR scans and physically based UE5 rendering), Argus simply normalizes ground-truth values by a constant scale factor (\(s = 10\)). The network directly learns absolute physical metric units without needing heuristic test-time scale recovery.

Loss & Training

Argus is optimized end-to-end using a compound loss function: $$ \mathcal{L} = 0.1 \cdot \mathcal{L}{\text{covis}} + \mathcal{L}}} + \mathcal{Ld + \mathcal{L}} + \mathcal{L{rp} + \mathcal{L} $$ The camera loss } + \mathcal{L}_{\text{joint}\(\mathcal{L}_{\text{cam}} = \mathcal{L}_q + \mathcal{L}_t\) measures L1 discrepancies between predicted and ground-truth quaternions and translation vectors, modulated by aleatoric uncertainty weights and an \(\alpha \log(\mathcal{C})\) regularizer (\(\alpha = 0.2\)). The depth loss \(\mathcal{L}_d\) incorporates both L2 distance and gradient consistency terms. The point map loss \(\mathcal{L}_p\) penalizes spatial coordinate L1 distances alongside surface normal cosine similarities \(\mathbf{n}(\cdot)\).

Training runs on 24 NVIDIA H20 GPUs (141 GB) for 36 hours over 99K iterations using the AdamW optimizer with cosine learning rate scheduling (peak learning rate 5e-5, 9.9K warmup steps). Batches contain 2 to 28 covisible views sampled per scene. Input panoramas are resized to \(560 \times 280\) and vertically cropped by removing the top and bottom 15% polar regions to a resolution of \(560 \times 196\). Data augmentations include color jittering, Gaussian blur, and random horizontal roll shifts (simulating random yaw rotation around the gravity axis).

Key Experimental Results

Main Results

On the Realsee3D benchmark (comprising 1,000 real-world scenes and 9,000 synthetic scenes), Argus is evaluated against adapted leading feed-forward architectures (VGGT360, MapAnything360, and \(\pi^3_{360}\)) across multi-view camera pose estimation, depth estimation, and 3D point cloud reconstruction.

Table 1: Comprehensive Multi-Task Performance on the Realsee3D Benchmark | Evaluation Task & Metric | Subset | VGGT360 | MapAnything360 | \(\pi^3_{360}\) | Argus (Ours) | Relative Advantage / Gain | | :--- | :--- | :---: | :---: | :---: | :---: | :---: | | Camera Pose ATE \(\downarrow\) (m) | Real | — | 0.134 | — | 0.096 | 28.4% error reduction | | Camera Pose ATE \(\downarrow\) (m) | Synthetic | — | 0.087 | — | 0.027 | 69.0% error reduction | | Pose Acceptance Rate A.R. \(\uparrow\) (%) | Real | — | 94.0 | — | 98.2 | +4.2% usability gain | | Pose Acceptance Rate A.R. \(\uparrow\) (%) | Synthetic | — | 99.3 | — | 99.8 | +0.5% usability gain | | Pose AUC@\(10^\circ\) \(\uparrow\) (%) | Real | 83.00 | 85.37 | 85.92 | 85.52 | Competitive with top baseline | | Pose AUC@\(10^\circ\) \(\uparrow\) (%) | Synthetic | 95.38 | 95.11 | 96.58 | 97.29 | Outperforms SOTA by +0.71% | | Metric Depth AbsRel \(\downarrow\) | Real | — | 0.070 | — | 0.050 | 28.6% relative improvement | | Metric Depth AbsRel \(\downarrow\) | Synthetic | — | 0.063 | — | 0.035 | 44.4% relative improvement | | Metric Depth \(\delta_1\) \(\uparrow\) (%) | Real | — | 46.31 | — | 66.28 | +19.97% threshold accuracy | | Metric Depth \(\delta_1\) \(\uparrow\) (%) | Synthetic | — | 74.64 | — | 91.24 | +16.60% threshold accuracy | | Metric Point Cloud Acc. \(\downarrow\) (m) | Real | — | 0.089 | — | 0.056 | 37.1% geometric precision gain | | Metric Point Cloud Comp. \(\downarrow\) (m) | Real | — | 0.075 | — | 0.063 | 16.0% completeness gain | | Metric Point Cloud N.C. \(\uparrow\) | Real | — | 0.825 | — | 0.894 | Higher normal consistency |

Ablation Study

Ablations on key architectural modules are conducted on the synthetic test subset under direct metric evaluation:

Table 2: Ablation Study of Key Modules on Realsee3D Synthetic Subset | Configuration | Depth AbsRel \(\downarrow\) | Depth \(\delta_1\) \(\uparrow\) (%) | Depth RMSE \(\downarrow\) | Point Acc. \(\downarrow\) | Point Comp. \(\downarrow\) | Point N.C. \(\uparrow\) | Pose ATE \(\downarrow\) (m) | Analysis & Insight | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :--- | | Full Model | 0.035 | 91.24 | 0.139 | 0.020 | 0.015 | 0.939 | 0.027 | Best overall geometric fidelity | | w/o Reference Selection | 0.037 | 90.45 | 0.142 | 0.024 | 0.017 | 0.923 | 0.060 | ATE surges 122% due to anchor drift | | w/o Joint Loss \(\mathcal{L}_{\text{joint}}\) | 0.047 | 90.71 | 0.158 | 0.020 | 0.015 | 0.937 | 0.049 | Consistency broken; depth & pose degrade | | w/o Camera Point Loss \(\mathcal{L}_{cp}\) | 0.055 | 89.90 | 0.161 | 0.020 | 0.015 | 0.936 | 0.047 | Loss of local 3D surface guidance | | w/o Rotated Point Loss \(\mathcal{L}_{rp}\) | 0.045 | 90.55 | 0.159 | 0.020 | 0.015 | 0.937 | 0.051 | Weaker relative orientation constraints | | w/o \(\mathcal{L}_{cp}\) & \(\mathcal{L}_{rp}\) | 0.055 | 89.75 | 0.160 | 0.020 | 0.015 | 0.936 | 0.048 | Optimization landscape deteriorates | | w/o All Point Losses | 0.075 | 88.52 | 0.169 | — | — | — | 0.061 | Degrades to naive pose+depth regression |

Key Findings

  • Reference Selection is Vital for Global Metric Stability: Disabling the covisibility reference selection module causes the absolute trajectory error (ATE) to surge from 0.027 m to 0.060 m (a 122% error increase). Selecting an anchor near the topological center of the scene prevents severe drift from peripheral viewpoints.
  • Camera-Space Point Supervision Strongly Benefits Depth Learning: Removing \(\mathcal{L}_{cp}\) degrades AbsRel significantly more than removing \(\mathcal{L}_{rp}\) (0.055 vs. 0.045). Even though camera points \(P_c\) and depth \(D\) are analytically interconvertible via ray vectors, explicit 3D distance and surface normal supervision in Euclidean space provides substantially smoother gradient landscapes than 2D depth regression alone.
  • Exceptional Runtime Efficiency: Argus processes 16 panoramic images in 0.66 seconds (3.54 GB peak GPU memory) and 128 images covering an entire house in 10.55 seconds (7.22 GB memory) on a single H20 GPU, demonstrating compelling practicality for large-scale spatial computing production pipelines.

Highlights & Insights

  • Topology-Guided Permutation Equivariance: By predicting global view connectivity and anchoring via an \(\arg\max\) operation, Argus achieves approximate permutation equivariance without incurring the prohibitive \(O(N^2)\) computational complexity of dense pairwise cross-attention.
  • Supervision Heavy, Inference Light: Incorporating overcomplete intermediate geometric representations and bidirectional cycle constraints during training reinforces multi-task synergy, yet allows intermediate heads to be trimmed at inference with zero runtime penalty.
  • Unlocking the Metric Panoramic Paradigm: Realsee3D addresses the persistent data bottleneck for omnidirectional 3D vision, validating that large-scale metric training enables feed-forward models to handle ERP distortions without hand-crafted spherical convolutions.

Limitations & Future Work

  • Generalization to Vast Open Environments: Model training is focused on structured indoor architectural environments; zero-shot transfer to unbounded outdoor scenes, drone aerial vistas, or dynamic streetscapes remains constrained.
  • Specular and Transparent Surfaces: While the synthetic subset covers diverse lighting conditions, real-world scenes with large floor-to-ceiling glass windows, full-length mirrors, or complex reflections present challenges for photometric covisibility estimation.
  • vs. VGGT / VGGT-Omega: VGGT established alternating attention for perspective feed-forward reconstruction, but its fixed reference frame severely degrades when applied to unordered panoramic captures; Argus introduces covisibility anchoring and geometric factorization, substantially outperforming VGGT360 across all geometric metrics.
  • vs. MapAnything: While MapAnything supports universal metric reconstruction, its unconstrained formulation exhibits noticeable pose and point map errors on indoor panoramas (e.g., 37% higher point error on Realsee3D Real); Argus demonstrates that modeling equirectangular projection geometry directly yields superior metric accuracy.
  • vs. \(\pi^3\): \(\pi^3\) adopts an equivariant pairwise architecture to handle unordered inputs, which suffers from high computational complexity and training instability; Argus achieves equivalent permutation robustness with significantly faster training convergence via covisibility-guided reference anchoring.

Rating

  • Novelty: ⭐⭐⭐⭐☆ Introduces learned covisibility anchoring and overcomplete geometric factorization to panoramic metric reconstruction.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive benchmarking on the 10K-scene Realsee3D dataset with detailed cross-task comparisons and ablations.
  • Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical formulations, clear pipeline presentation, and insightful empirical analysis.
  • Value: ⭐⭐⭐⭐⭐ Overcomes the metric barrier in panoramic 3D reconstruction with immediate utility for VR, digital twins, and embodied robotics.