GARDEN: Gravity-Aligned Reconstruction of Disentangled ENvironments from RGB images

Conference: ECCV 2026
Paper: ECCV Official
Code: https://sunjiahaovo.github.io/garden/
Area: 3D Vision
Keywords: gravity alignment / scene factorization / physics simulation / hybrid representation / 3D point classification

TL;DR

GARDEN presents an RGB-only physically grounded scene factorization framework that resolves gauge ambiguity via a learned Gravity-View coordinate system, reconstructs scene-specific rigid meshes with gravity-constrained 6-DoF poses, and eliminates duplicate geometry through conditional 3D point classification to deliver simulation-ready hybrid representations.

Background & Motivation

Synthesizing interactive 3D digital twins from casual multi-view RGB imagery is a cornerstone capability for Embodied AI and interactive physical simulation. However, prevailing 3D reconstruction systems based on Neural Radiance Fields (NeRF), 3D Gaussian Splatting (3DGS), or feed-forward multi-view foundation models (e.g., DepthAnything-3, VGGT) typically output monolithic "3D photographs." These representations couple foreground rigid objects and static background geometry into single holistic surfaces, lacking instance-level modularity. Crucially, their reconstructed coordinate systems are defined only up to an arbitrary global rotation without an explicit physical gravity axis, leaving gauge ambiguity unresolved and causing severe physical instability when entities are imported into dynamic simulators.

To introduce interactive structure, contemporary pipelines such as LiteReality and ACDC rely on a "detect-retrieve-replace" workflow: estimating 3D bounding boxes and substituting real objects with retrieved artist-designed CAD assets. While this recovers modular interactivity, CAD replacement fundamentally sacrifices scene-specific geometric authenticity because pre-modeled assets cannot match the idiosyncratic shapes and fine-grained variations of in-the-wild objects. Furthermore, chaining 3D layout prediction, database querying, and fine alignment introduces severe computational bottlenecks and error cascades across stages, impeding fast and scalable digital twin generation.

This paper tackles this dilemma by reframing reconstruction as physically-grounded scene factorization without external CAD retrieval. Gravity serves as a universal, invariant physical prior to anchor objects and resolve global gauge freedom. Core idea: construct a unified Gravity-View coordinate frame via cross-view camera token regression, under which amodal object mesh generation, gravity-constrained 6-DoF pose refinement, and conditional 3D point classification are orchestrated to output a structured hybrid representation of decoupled rigid bodies and clean visual backgrounds.

Method

Overall Architecture

GARDEN takes unposed multi-view RGB images as input and produces a simulation-ready hybrid representation comprising standalone rigid meshes with accurate 6-DoF poses and a cleaned static background (colored point clouds or 3DGS), seamlessly executable within physics engines like MuJoCo. The pipeline operates in four interconnected stages: 1. Multi-View Reconstruction with Gravity-View Alignment (GV-Align): Employs a multi-view geometric foundation model to extract global camera tokens, aggregates multi-view gravity clues via cross-attention to predict the reference view rotation \(\hat{R}_{c_{\text{ref}} \to \text{gv}}\), transforming the entire reconstruction into an upright Gravity-View (GV) physical frame; 2. Target-Driven Object Generation and Layout Estimation: Guided by a lightweight 2D prompt (user click or VLM detection), extracts high-precision 2D masks using SAM-3, reconstructs amodal 3D meshes using SAM-3D, and refines the 6-DoF rigid layout with FoundationPose under gravity-constrained rotational bounds; 3. Point-Conditional Background Disentanglement: Formulates redundant geometry removal as a conditional 3D point classification task, using a Transformer to cross-attend cropped scene points with surface-sampled mesh points to prune duplicate object points from the background; 4. Unified Physics Simulation: Identifies the support ground plane along the gravity vector, instantiates static floor collisions and dynamic rigid objects, and pairs them with high-fidelity visual backgrounds for stable MuJoCo simulation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-View RGB Images Input"] --> B["Gravity-View Alignment Module<br/>Cross-view attention regressing R_c2gv"]
    B --> C["GV-Aligned Reconstruction<br/>Canonical upright point cloud / 3DGS"]
    C --> D["Target-Driven Object Generation & Layout Estimation<br/>SAM-3D amodal mesh + gravity-constrained 6-DoF pose"]
    D --> E["Point-Conditional Background Disentanglement<br/>Transformer cross-attention point classifier"]
    E --> F["Unified Physics Simulation & Hybrid Rendering<br/>MuJoCo rigid body dynamics + pristine background"]

Key Designs

1. Gravity-View Alignment (GV-Align): Resolving Gauge Ambiguity via Universal Physical Priors
Conventional multi-view feed-forward backbones (e.g., DepthAnything-3, VGGT) orient reconstructions either arbitrarily or locked to an arbitrary reference camera frame, producing globally tilted geometries where floor and object points entangle along projected axes. GARDEN adopts the Gravity-View (GV) convention: given gravity direction \(g\) in camera coordinates, it establishes the canonical frame via \(y = g\), \(x = \frac{y \times v}{\|y \times v\|}\) (where \(v = [0, 0, 1]^\top\) is camera view direction, reverting to the camera's original \(x\)-axis upon collinearity), and \(z = x \times y\), forming the rotation matrix \(R_{c \to \text{gv}} = [x, y, z]^\top\). To regress this transformation purely from visual input, the network freezes DepthAnything-3 to obtain camera tokens \(F \in \mathbb{R}^{N \times C}\), takes the reference token \(f_{\text{ref}}\) as query, and applies a Context Transformer Decoder with full-sequence cross-attention to harvest gravity-aware geometric clues. An MLP head regresses a continuous 6D rotation mapped to an orthogonal 3x3 matrix \(\hat{R}_{c_{\text{ref}} \to \text{gv}}\), supervised with ground-truth synthetic extrinsics via L1 loss: $\(\mathcal{L}_{\text{rot}} = \|\hat{R}_{c_{\text{ref}} \to \text{gv}} - R_{c_{\text{ref}} \to \text{gv}}^{\text{gt}}\|_1\)$ This explicit alignment restores physical verticality and decouples vertical support planes from foreground geometry.

2. Gravity-Constrained 6-DoF Pose Refinement: Amodal Geometry with Reduced Search Space
Preserving idiosyncratic object geometry without CAD replacement requires robust amodal reconstruction. Given a prompt box, SAM-3 extracts a 2D mask which SAM-3D lifts into an amodal 3D mesh, completing occluded backside geometry. However, native SAM-3D spatial placement remains too coarse for collision-level contact dynamics. GARDEN integrates FoundationPose to optimize 6-DoF poses directly in the GV space. Because SAM-3D produces inherently axis-aligned meshes, aligning their vertical axis with the recovered gravity vector is deterministic. Consequently, FoundationPose's optimization over \(SE(3)\) is reduced: pitch and roll are locked, leaving only 3D translation and horizontal yaw angle to be resolved. This structural constraint dramatically shrinks the optimization search space and eliminates physically absurd tilted poses or floor penetrations.

3. Point-Conditional Background Disentanglement: Transformer-Based Duplicate Pruning
Superimposing standalone object meshes into the original scene creates duplicated geometry, causing ghosting artifacts and simulation collisions. Naive 3D bounding box cropping inevitably damages neighboring support surfaces (e.g., tables, floors), while 2D multi-view segmentation suffers from cross-view association drift in the presence of repetitive instances (e.g., multiple identical chairs). GARDEN frames this as conditional 3D point classification. The network takes two point inputs: scene points inside a relaxed bounding box, and condition points uniformly sampled from the reconstructed object mesh surface. A 6-block Transformer with interleaved self- and cross-attention maps both sets into a shared embedding space, predicting point-level binary probabilities to filter out target-object points at a threshold of 0.5. To bridge the synthetic-to-real gap, training on InternScenes incorporates artifact-oriented augmentations (multi-layer ghosting, thin-structure deformation, and pose jitter), ensuring robust transfer to casual real scans.

Loss & Training

The framework employs modular training: 1. Gravity-View Alignment Module: Trained for 10k steps on TartanAir, Hypersim, and Virtual KITTI 2 with AdamW, peak learning rate \(5 \times 10^{-6}\), and linear warm-up of 1k steps using rotation L1 loss \(\mathcal{L}_{\text{rot}}\); 2. Conditional Point Classification Network: Trained for 2k steps on InternScenes using Binary Cross-Entropy (BCE) loss at base learning rate \(5 \times 10^{-4}\) with batch size 48; 3. Inference & Simulation Integration: Input images are resized to a maximum dimension of 504. FoundationPose refines layouts over 5 iterations. Physics execution runs in MuJoCo with 0.001s timestep, 50 Newton solver iterations per step, and gravity vector \(g_{\text{sim}} = [0, 9.81, 0]^\top\).

Key Experimental Results

Main Results

To evaluate simulation-readiness rather than static visual tricks, GARDEN introduces a rigorous post-simulation evaluation protocol: reconstructed assets are imported into MuJoCo, simulated under gravity for 10 seconds, and then rendered to assess image-space discrepancies. Floating objects, interpenetrations, or unstable poses are immediately exposed and penalized.

Table 1: Object-Centric Material and Geometry Recovery (LiteReality Benchmark)

Method	RMSE ↓	SSIM ↑	LPIPS ↓	Evaluation Protocol
PhotoShape	0.3225	0.2371	0.6558	Static cropped bbox
MIR	0.2377	0.3981	0.6111	Static cropped bbox
MIR + AO	0.2156	0.4203	0.5899	Static cropped bbox
Sem&Vis (CAD retrieval)	0.2835	0.3758	0.6362	Static cropped bbox
LiteReality	0.2163	0.4353	0.5854	Static cropped bbox
GARDEN (3DGS, post-sim)	0.1880	0.4181	0.5035	10s MuJoCo simulation
GARDEN (point cloud, post-sim)	0.1887	0.4240	0.4444	10s MuJoCo simulation
GARDEN (3DGS, static baseline)	0.1796	0.4494	0.4182	Static without simulation
GARDEN (point cloud, static baseline)	0.1736	0.4656	0.3680	Static without simulation

Table 2: Holistic Graphics-Ready Scene Reconstruction (Full-Scene Novel View Synthesis)

Method	Full-Scene RMSE ↓	Full-Scene SSIM ↑	Full-Scene LPIPS ↓	Setup
Phone2Proc	0.3604	0.5512	0.7338	Holistic scene
ACDC	0.3653	0.5531	0.7364	Holistic scene
ACDC + Sem&Vis	0.3226	0.5425	0.6717	Holistic scene
ACDC + MIR	0.3046	0.5492	0.6648	Holistic scene
LiteReality	0.2664	0.5818	0.6522	Static graphics assets
GARDEN (3DGS, post-sim)	0.1670	0.5573	0.4709	10s MuJoCo simulation
GARDEN (point cloud, post-sim)	0.1616	0.5735	0.4081	10s MuJoCo simulation
GARDEN (3DGS, static baseline)	0.1593	0.5718	0.4379	Static without simulation
GARDEN (point cloud, static baseline)	0.1511	0.5916	0.3770	Static without simulation

Ablation Study

The paper validates gravity alignment accuracy and inspects how gravity constraints influence rendering and point cloud geometry in local object regions.

Table 3: Gravity Estimation Angular Error (Degrees) on Held-Out Test Scenes

Method	Hypersim Mean ↓	Hypersim P90 ↓	Hypersim Fail@10° (%) ↓	TartanAir Mean ↓	TartanAir P90 ↓	TartanAir Fail@10° (%) ↓
Plane-RANSAC	24.87°	89.68°	26.7%	19.26°	83.22°	34.4%
Normal Clustering	30.98°	89.77°	33.3%	21.33°	88.70°	37.5%
COLMAP Manhattan	2.62°	6.38°	0.0%	8.37°	15.91°	12.5%
GeoCalib	7.59°	4.79°	6.7%	3.01°	6.50°	9.4%
GeoCalib + RANSAC	1.71°	3.79°	0.0%	2.69°	5.91°	9.4%
GARDEN GV Predictor	1.40°	1.90°	0.0%	1.56°	3.48°	0.0%

Table 4: Factorization and Placement Ablation on Hypersim (Local Object Region)

Configuration	Background	RMSE ↓	SSIM ↑	LPIPS ↓	Comp. ↓	Acc. ↓	Normal Consist. (N.C.) ↑
DA3 (Monolithic Backbone)	Point Cloud	0.2788	0.3482	0.5311	0.0516	0.0082	0.9129
w/o GV, full FP pose w/o sim	Point Cloud	0.2961	0.3211	0.5917	0.0447	0.0726	0.7320
w/o GV, full FP pose w/ sim	Point Cloud	0.3002	0.3096	0.5929	0.0743	0.3132	0.7078
w/ GV, full FP pose w/o sim	Point Cloud	0.2753	0.3562	0.5351	0.0470	0.0674	0.7275
GARDEN Full	Point Cloud	0.2751	0.3565	0.5338	0.0387	0.0658	0.7497
DA3 (Monolithic Backbone)	3DGS	0.3068	0.3141	0.6623	-	-	-
w/o GV, full FP pose w/o sim	3DGS	0.3134	0.3000	0.6617	-	-	-
w/o GV, full FP pose w/ sim	3DGS	0.3171	0.2885	0.6725	-	-	-
GARDEN Full	3DGS	0.2840	0.3809	0.5575	-	-	-

Key Findings

1. Gravity Alignment is Critical for Dynamic Stability: Disabling GV alignment under simulation (w/o GV, full FP pose w/ sim) causes geometric Accuracy (Acc.) to collapse drastically from 0.0658 to 0.3132, and Completeness (Comp.) to degrade to 0.0743. Without a physically grounded vertical axis, arbitrary global tilt causes unconstrained objects to slide off tables or tip over during simulation.
2. Substantial Latency Speedup (7.7× Faster): By replacing cumbersome 3D layout detection and CAD model retrieval with feed-forward amodal generation, GARDEN slashes end-to-end inference time across 5 benchmark scenes from an average of 4332.8s (LiteReality) to 560.2s (Table 5).
3. Decoupled Completeness Over Monolithic Baselines: Compared to the monolithic DA3 baseline, GARDEN improves geometric completeness from 0.0516 to 0.0387, proving that amodal 3D mesh synthesis successfully fills occluded object geometry while conditional point classification removes duplicated background points without eroding surrounding context.

Highlights & Insights

• Gravity as an End-to-End Coordinate Anchor: Rather than treating gravity as an isolated post-processing rotation, GARDEN weaves gravity through the whole pipeline: camera token decoding, ground plane detection, 6-DoF search space reduction (locking pitch/roll), and dynamic simulator initialization.
• Mesh-Conditioned Transformer Disentanglement: Leveraging object surface samples as geometric condition tokens within a cross-attention Transformer sidesteps both bounding-box over-cropping and multi-view semantic drift, enabling clean point-level separation on complex indoor scenes.
• Simulation-Ready Stress-Testing: Elevating the benchmarking standard from static image synthesis to a 10-second post-simulation rendering benchmark sets a rigorous new precedent for Embodied AI digital twin generation.

Limitations & Future Work

• Author-Acknowledged Limitations: Target objects currently require interactive 2D bounding box initialization or open-vocabulary grounding prompts; the system does not yet support fully autonomous all-object scene-wide decomposition. Additionally, it targets rigid bodies and does not support articulated mechanisms (e.g., cabinets, doors) or soft deformable items.
• Identified Limitations: Complete reliance on single-view amodal priors (SAM-3D) for occluded backsides can hallucinate incorrect geometry for highly asymmetric or long-tailed shapes. Furthermore, the pipeline lacks explicit PBR material decomposition (albedo, roughness, specular), relying on point/Gaussian appearance for visual backdrop rendering.
• Future Directions: Integrating active embodied exploration to resolve occluded ambiguities, extending the conditional classification to joint articulation tree estimation, and incorporating physically-based inverse rendering.

Related Work & Insights

• vs LiteReality [Huang et al., NeurIPS 2024]: LiteReality mandates RGB-D scans and relies on heavy CAD retrieval, averaging 4330+ seconds per scene while losing unique geometric details; GARDEN operates on casual RGB images, runs in 560 seconds (7.7× speedup), and preserves exact instance geometry via generative amodal meshes.
• vs DepthAnything-3 [Lin et al., 2025] & VGGT [Wang et al., 2025]: State-of-the-art multi-view models generate monolithic, physically ungrounded point clouds subject to arbitrary rotation; GARDEN leverages DepthAnything-3's representation as a foundation, adding gravity alignment and conditional disentanglement to unlock physics simulation.
• vs GVHMR [Shen et al., SIGGRAPH Asia 2024]: While GVHMR pioneered Gravity-View coordinates to stabilize human motion recovery, GARDEN generalizes the gravity-aligned physical frame to full 3D scene factorization and multi-body physics simulation.

Rating

• Novelty: ⭐⭐⭐⭐ [Elegant physical framing using gravity to resolve visual gauge freedom; CAD-free hybrid factorization]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Pioneering 10-second post-simulation evaluation protocol; exhaustive ablations on gravity and latency]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear problem formulation, well-structured figures, and mathematically grounded designs]
• Value: ⭐⭐⭐⭐⭐ [Provides a practical, highly efficient bridge connecting casual RGB video capture to embodied AI simulation]

Related Papers

• [ECCV 2026] Gravity-aware partially calibrated absolute pose estimation from affine- or rotation-covariant features
• [ECCV 2026] COSY: Compositional 3DGS Synthesis for Disentangled Human Head Editing
• [ECCV 2026] Edit3r: Instant 3D Scene Editing from Sparse Unposed Images
• [ECCV 2026] RGB-Pointmap Pretraining for Unified 3D Scene Understanding
• [ECCV 2026] APT: Anchor-aligned Perturbations for Tamper Localization in Fully Regenerated Images