CasaMaestro: Multi-View Panoramas for House-Scale 3D Reconstruction

Conference: ECCV 2026
Paper: ECCV Official
Code: https://george-attano.github.io/CasaMaestro
Area: 3D Vision
Keywords: panoramic 3D reconstruction, house-scale reconstruction, multi-view panoramas, camera pose estimation, cross-view attention

TL;DR

CasaMaestro is the first feedforward model for house-scale, extrinsic-free 3D reconstruction from multi-view panoramas, using only 20 to 50 sparse indoor 360° captures to jointly predict metric depth and global camera poses for fast, metric point-cloud generation across entire homes.

Background & Motivation

Home-deployed embodied AI systems are driving an urgent need for fast, metrically consistent 3D reconstruction of residential environments to support navigation, spatial interaction, and simulation-to-real transfer. While LiDAR scanning reliably yields high-quality 3D assets, it remains costly, time-consuming, and dependent on dedicated hardware. Conversely, emerging 3D vision foundation models (such as VGGT, π3, and Depth Anything 3) provide a compelling sensor-free alternative by lifting unconstrained 2D imagery into 3D. However, these models predominantly assume pinhole cameras, whose inherently narrow field of view requires thousands of densely captured frames to cover multiple rooms, inevitably triggering severe trajectory drift, accumulative alignment errors, and excessive compute overhead.

To improve efficiency, several studies have explored panoramic cameras, yet most remain restricted to monocular depth estimation without relative poses or to short-baseline, 2-to-5-view setups (such as PanoSplatt3R). Existing panoramic pose networks (such as PanoPose and SPR) are tailored for dense video streams with massive visual overlap, breaking down completely under sparse, multi-room sampling. The underlying challenge is that sparse residential captures introduce extreme viewpoint displacements and cross-room occlusions, requiring a network to simultaneously preserve fine-grained local representations for dense depth regression and establish globally consistent geometric alignment across wide-baseline panoramas.

The core idea of this paper is to adopt a minimalist DINOv2 backbone with alternating intra-view and cross-view self-attention, coupled with a lightweight cross-view attention pose decoder and physically consistent ERP rotation data augmentation, directly predicting metric depth and camera extrinsics from sparse uncalibrated panoramas.

Method

Overall Architecture

CasaMaestro takes \(V\) sparse, unposed equirectangular panoramic images (typically 20 to 50 views) as input. The framework operates across three key stages: first, a unified plain DINOv2 ViT backbone extracts multi-view visual representations by performing independent intra-view attention in early layers and alternating between intra-view and flattened cross-view attention in deeper layers; second, a multi-scale DPT head decodes per-view metric depth maps; third, a dedicated lightweight camera pose decoder leverages cross-view attention over condensed tokens to directly regress camera translations and rotation quaternions; finally, predicted metric depths and extrinsics are back-projected into a house-scale metric 3D point cloud.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Sparse Multi-View Panoramas<br/>V raw RGB frames, extrinsic-free"] --> B["Unified Multi-View Backbone<br/>Early intra-view, deep alternating cross-view"]
    B --> C["Feature Extraction & Decoupling<br/>Dense prediction features + global view tokens"]
    C --> D["DPT Depth Head<br/>Multi-scale fusion for metric depth"]
    C --> E["Cross-View Pose Decoder<br/>View-wise attention for decoupled global alignment"]
    D --> F["House-Scale 3D Point Cloud<br/>Metric, drift-free multi-room back-projection"]
    E --> F

Key Designs

1. Alternating Multi-View Backbone: Unifying Intra-View Representation and Cross-View Information Exchange

To reconstruct residential spaces from sparse wide-baseline panoramas, the feature extractor must preserve local high-frequency details despite equirectangular distortion while establishing long-range contextual links across distant rooms. Instead of adding complex spherical convolutions or multi-branch modules, CasaMaestro follows a minimalist plain ViT architecture based on DINOv2, enabling cross-view reasoning purely via token dimension reordering. Given an input tensor of patch tokens \(X^{(0)} \in \mathbb{R}^{B \times V \times N \times D}\) across batch size \(B\), views \(V\), patch count \(N\), and feature dimension \(D\), the first \(L_{\text{local}}\) layers fold the view dimension into the batch dimension (yielding \((BV) \times N \times D\)) and run standard self-attention independently per view. Starting at layer \(L_{\text{local}}\), the model alternates between local intra-view steps and global cross-view steps. In global steps, tokens are reshaped into \(B \times (VN) \times D\), allowing tokens from all views to interact globally with zero extra parameters.

2. Decoupled Cross-View Pose Decoder: Isolating Global Geometry from Dense Visual Features

A naive pose head directly regresses camera extrinsics from each view's token independently using an MLP. However, because the upstream backbone must focus primarily on generic dense representations for depth estimation, forcing it to fully resolve pairwise global geometric ambiguities compromises dense feature quality. CasaMaestro introduces a lightweight panoramic pose decoder that operates across the view dimension. The top-level features from the backbone are first projected into dimension \(D\) with LayerNorm, passed through a per-view residual MLP (with expansion ratio \(r\)) to condition the view tokens, and then processed by a multi-head cross-view self-attention layer across the \(V\) views. An optional mask \(M\) can filter out occluded or invalid capture regions. Finally, linear heads regress translation \(\hat{t}_{b,v} \in \mathbb{R}^3\) and unnormalized quaternion \(\tilde{q}_{b,v} \in \mathbb{R}^4\), with rotation strictly constrained to the unit sphere:

\[ \hat{q}_{b,v} = \frac{\tilde{q}_{b,v}}{\max(\|\tilde{q}_{b,v}\|_2, \epsilon)} \]

Because standard equirectangular panoramas have fixed fields of view (360° horizontal and 180° vertical), focal length and FoV prediction are omitted, making the pose head uniquely streamlined and numerically stable.

3. ERP Rotation Data Augmentation: Rigorous Geometry-Preserving Viewpoint Expansion

In existing indoor panoramic benchmarks, camera viewpoints within the same residence typically exhibit near-zero relative orientation diversity (near-identical yaw angles), hampering the model's ability to learn robust rotation reasoning. CasaMaestro introduces an explicit spherical ray-remapping augmentation for equirectangular projection data. For each view in synthetic training scenes, yaw \(\psi \sim \mathcal{U}(0, \pi)\), pitch \(\theta\), and roll \(\phi\) are independently sampled to form a rotation matrix \(R = R_z(\psi) R_y(\theta) R_x(\phi)\). For an output ERP image of size \(H \times W\), each pixel \((u, v)\) is converted to a spherical unit ray direction \(d_{\text{out}}\), rotated to source coordinates \(d_{\text{src}} = R d_{\text{out}}\), and mapped back to source pixel coordinates \((u_s, v_s)\) with bilinear interpolation and horizontal periodic boundary wrapping. RGB images and metric depth maps are warped synchronously, and camera extrinsic matrices are updated via right-multiplication:

\[ R_{c2w} \leftarrow R_{c2w} R \]

This strategy synthesizes diverse relative camera orientations while strictly honoring underlying 3D projective geometry.

Loss & Training

The network is trained end-to-end using the Adam optimizer across 4 NVIDIA A100 GPUs for 20 epochs with a per-GPU batch size of 1 and a learning rate of \(5 \times 10^{-5}\). The number of input views varies dynamically between 20 and 50, and the backbone is initialized from Depth Anything 3. The joint objective consists of a multi-scale depth regression loss \(\mathcal{L}_{\text{depth}}\) and a camera pose loss \(\mathcal{L}_{\text{pose}}\) balancing translation \(L_1\) distance and quaternion rotation error.

Key Experimental Results

Main Results

The authors evaluate camera pose estimation on the Realsee test split (both real-world captures in Realsee-Real and synthetic residences in Realsee-Syn). Metrics include Area Under the Curve (AUC) at 10°, 20°, and 30° thresholds, mean rotation error (degrees), mean translation error (centimeters), and overall mean pose error.

Table 1: Camera Pose Estimation Comparison on Realsee-Real (Source: Table 1)

Method	AUC@10 ↑	AUC@20 ↑	AUC@30 ↑	Rot. Mean (°) ↓	Trans. Mean (cm) ↓	Pose Mean ↓
PanoPose* [32]	0.042	0.115	0.166	62.20	58.27	90.12
SPR* [46]	0.086	0.145	0.191	59.43	56.12	86.23
VGGT [37]	0.189	0.313	0.388	37.74	32.31	47.65
VGGT-Finetune*	0.276	0.379	0.411	30.12	10.23	38.42
PI3 [41]	0.222	0.365	0.450	26.69	25.48	36.61
PI3-Finetune*	0.301	0.397	0.510	21.36	7.442	28.97
DepthAnything3 [15]	0.083	0.173	0.235	54.57	49.79	69.51
InfiniteVGGT [44]	0.271	0.412	0.487	29.82	4.488	30.01
CasaMaestro (Ours)	0.727	0.859	0.903	2.608	2.132	3.225

Note: * indicates methods fine-tuned or re-implemented on the Realsee training set. CasaMaestro achieves an AUC@30 of 0.903 on Realsee-Real, an 84% relative improvement over the best prior result (0.510), reducing mean rotation error from 21.36° to 2.61° and mean translation error from 7.44 cm to 2.13 cm.

Table 2: Zero-Shot Panoramic Metric Depth Estimation on Unseen Datasets (Source: Table 4)

Method	Replica (AbsRel ↓)	Replica (\(\delta_1\) ↑)	PanoSUNCG (AbsRel ↓)	PanoSUNCG (\(\delta_1\) ↑)	HM3D (AbsRel ↓)	HM3D (\(\delta_1\) ↑)
PanoFormer [29] (single-view)	0.076	0.945	0.054	0.978	0.151	0.742
DA2 [13] (single-view)	0.070	0.966	0.060	0.976	0.164	0.751
DAP [16] (single-view)	0.124	0.878	0.130	0.841	0.143	0.749
PanoPose* [32] (multi-view)	0.116	0.937	0.094	0.955	0.215	0.708
CasaMaestro (Ours)	0.058	0.970	0.042	0.988	0.105	0.921

Ablation Study

The authors systematically ablated model capacity (DINO-Base vs. DINO-Large), the cross-view pose decoder (PoseDecOurs vs. PoseDecLinear), and ERP rotation data augmentation (ERPAug).

Table 3: Ablation Study on Key Components (Source: Table 5)

#	Backbone	Pose Decoder	ERP Aug.	AUC@10 ↑	AUC@30 ↑	Rot. Mean (°) ↓	Trans. Mean (cm) ↓	AbsRel ↓	\(\delta_1\) ↑
①	DINO-Base	Linear Head	✕	0.302	0.568	24.39	17.85	0.092	0.920
②	DINO-Base	Ours (Cross-View)	✕	0.629	0.831	3.192	2.834	0.077	0.939
③	DINO-Base	Linear Head	✓	0.512	0.772	2.175	2.375	0.111	0.970
④	DINO-Base	Ours (Cross-View)	✓	0.688	0.891	2.547	2.404	0.075	0.956
⑤	DINO-Large	Linear Head	✕	0.297	0.598	24.41	16.32	0.077	0.939
⑥	DINO-Large	Ours (Cross-View)	✕	0.707	0.871	2.102	2.175	0.076	0.973
⑦	DINO-Large	Linear Head	✓	0.604	0.815	2.132	2.265	0.080	0.938
⑧ (Full)	DINO-Large	Ours (Cross-View)	✓	0.792	0.927	1.553	1.682	0.078	0.975

Key Findings

• The dedicated cross-view pose decoder is the single most critical architectural factor: on DINO-Base (comparing ① with ②), AUC@30 jumps from 0.568 to 0.831, and rotation error plummets from 24.39° to 3.19°. Training loss trajectories reveal that models with the cross-view pose decoder achieve convergence within 3 epochs that linear decoders fail to reach even after 10+ epochs.
• Simply scaling backbone parameters without specialized geometric decoding yields negligible gains (comparing ① with ⑤, AUC@10 is 0.302 vs 0.297), proving that wide-baseline pose regression bottlenecks reside in inter-view geometric reasoning rather than upstream feature extraction capacity.
• ERP rotation augmentation acts as an indispensable geometric regularizer: in the full model (⑥ vs ⑧), AUC@30 improves from 0.871 to 0.927, reducing rotation error to 1.55° and translation error to 1.68 cm while powering competitive zero-shot cross-dataset generalization.

Highlights & Insights

• Minimalist Multi-View Foundation: Avoids spherical convolutions or multi-branch architectures, relying solely on dynamic token dimension flattening inside a vanilla ViT to execute cross-view reasoning with zero parameter overhead.
• Decoupled Geometric Alignment: Isolates the global camera pose alignment burden from dense visual representation learning, allowing the lightweight view-attention decoder to achieve accurate camera poses without corrupting dense metric depth prediction.
• Physically Grounded ERP Data Augmentation: Synthesizes diverse camera orientations with paired ground truth depths and poses directly from 2D equirectangular rays, resolving the severe yaw-bias pervasive in indoor panorama datasets.

Limitations & Future Work

• Static Scene Assumption: The current formulation assumes rigid static environments; moving humans or pets could degrade cross-view geometric correspondence, suggesting the need for dynamic object segmentation and confidence masking.
• Extreme Disconnects and Multi-Floor Layouts: When rooms are completely occluded by closed doors or when navigating across floor levels with zero direct line-of-sight, purely visual feedforward alignment can experience local frame drift; incorporating architectural floor plan priors or lightweight global bundle adjustment could further improve global topological consistency.

Related Work & Insights

• vs Depth Anything 3 / VGGT: Pinhole foundation models demand dense video streams and suffer catastrophic drift over long indoor trajectories; CasaMaestro proves that multi-view panoramas enable drift-free, house-scale metric reconstruction with as few as 20–50 captures.
• vs PanoPose / SPR: Prior panoramic pose networks rely strictly on high video frame overlap and collapse under wide-baseline sparse captures; CasaMaestro eliminates the small-motion assumption through cross-view attention and explicit rotation augmentation.

Rating

• Novelty: ⭐⭐⭐⭐⭐ First feedforward, extrinsic-free foundation model for house-scale 3D metric reconstruction from sparse multi-view panoramas.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across real and synthetic benchmarks, extensive cross-dataset zero-shot transfers, and detailed ablation of every core module.
• Writing Quality: ⭐⭐⭐⭐⭐ Cohesive narrative, clean mathematical formulation, and well-structured empirical validation.
• Value: ⭐⭐⭐⭐⭐ Provides an efficient, high-fidelity asset acquisition paradigm for embodied AI simulation and digital twins.

Related Papers

• [ECCV 2026] Large-Scale High-Quality 3D Gaussian Head Reconstruction from Multi-View Captures
• [ECCV 2026] Capacity-Controlled Multi-View Stylization of 3D Gaussian Splatting
• [ECCV 2026] LinStereo: Linear-Complexity Global Attention for Multi-Scale Iterative Stereo Matching
• [ECCV 2026] Physically Grounded 3D Generative Reconstruction under Hand Occlusion using Proprioception and Multi-Contact Touch
• [ECCV 2026] Multi-View Foundation Models