Scene Generation at Absolute Scale: Utilizing Semantic and Geometric Guidance From Text for Accurate and Interpretable 3D Indoor Scene Generation

Conference: ECCV 2026
Paper: ECCV Official
Project Page: https://d3ixi.github.io/GuidedSceneGen/
Area: 3D Vision
Keywords: 3D Indoor Scene Generation / Absolute Metric Scale / Panoramic Diffusion / 3D Gaussian Splatting / Camera Trajectory Optimization

TL;DR

GuidedSceneGen pioneers an absolute-scale text-to-3D indoor scene generation pipeline by leveraging a 3D metric proxy layout, multi-modal semantic/depth-guided panoramic diffusion, and collision-aware camera trajectory sampling to reconstruct fully navigable, semantically interpretable 3D Gaussian Splatting scenes without external pose estimation.

Background & Motivation

Generating high-fidelity, freely explorable 3D indoor scenes directly from natural language text is a foundational milestone for computer vision, computer graphics, and embodied AI. However, existing text-to-3D scene generation paradigms suffer from severe scale ambiguity and progressive geometric drift. Mainstream pipelines predominantly rely on iterative 2D-to-3D lifting and inpainting (such as Text2Room and SceneScape) or panoramic image outpainting and layering (such as LayerPano3D, DreamScene360, and WorldExplorer). Operating in unconstrained or relative coordinate frames, these methods accumulate geometric distortions, floaters, and appearance inconsistencies as exploratory camera trajectories lengthen. Crucially, the generated scenes lack physical metric scale, leaving object dimensions distorted and rendering them unusable for downstream robotic path planning or physical interaction.

The underlying tension arises from the fundamental gap between generative 2D/video diffusion models and physical 3D scene understanding: diffusion models excel at producing detailed visual textures and local multi-view priors within normalized latent spaces, yet they inherently lack absolute physical metric awareness. Conversely, downstream robotic manipulation, navigation, and CAD modeling strictly require a unified metric coordinate frame along with explicit 9D bounding boxes (3D position, orientation, and metric dimensions) and categorical semantic segmentations. Prior proxy-guided approaches (such as SceneCraft and ControlRoom3D) rely on pre-defined axis-aligned bounding boxes (AABBs) or discrete CAD retrieval, preventing an automated, end-to-end transfer of learned geometry and object poses.

This paper tackles the challenge from a proactive standpoint: rather than attempting post-hoc geometry recovery from uncalibrated 2D views, it establishes an explicit global 3D proxy layout from the input prompt at the very beginning of the generation pipeline. This proxy anchors the entire process in an absolute world coordinate frame and continuously feeds metric depth and semantic constraints into subsequent panoramic diffusion and camera trajectory sampling. Core idea: Grounding the generation in a text-predicted metric 3D proxy layout, the method couples semantic- and depth-controlled panoramic diffusion with collision-aware novel-view video sampling, achieving seamless 3D Gaussian Splatting reconstruction in an absolute world frame with accurate 9D object poses and modular multi-room extendability.

Method

Overall Architecture

GuidedSceneGen adopts a coarse-to-fine generation pipeline that operates strictly within an absolute metric coordinate frame. Given an input textual scene prompt, the framework first estimates a global 3D proxy layout containing room boundaries and 9D-oriented object instances; it then renders absolute metric depth and NYUv2-40 semantic cubemaps to condition a multi-view panoramic diffusion model, producing a globally consistent \(360^\circ\) RGB panorama. To discover occluded areas, the method executes a grid-partitioned, collision-aware trajectory planning strategy and optimizes the camera scale parameter of a video diffusion model via bisection search on metric depth. Finally, all synthesized RGB frames are combined with their true metric camera poses to optimize a 3D Gaussian Splatting (3DGS) radiance field, enabling precise 9D pose transfer, 3D semantic segmentation, and seamless scene expansion.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Text Prompt Input"] --> B["Text-to-3D Global Metric Proxy Layout Estimation<br/>Establish metric world frame & 9D object priors"]
    B --> C["Semantic & Geometric-Guided Panoramic Diffusion Generation<br/>Multi-ControlNet driving 6-view cubemap"]
    C --> D["Collision-Aware Camera Trajectory Planning<br/>Grid-based 4-quadrant sampling & 0.3m depth filtering"]
    D --> E["Depth-Guided Bisection Video Novel View Synthesis<br/>Minimize L1 depth error to calibrate SVC camera scale"]
    E --> F["Absolute-Scale 3DGS Scene Reconstruction<br/>Fusing 338 ground-truth posed views with geometric losses"]

Key Designs

1. Text-to-3D Global Metric Proxy Layout Estimation: Establishing Absolute World Coordinates and 9D Object Priors Prior generative pipelines accumulate irreversible spatial drift because they operate in ungrounded relative spaces. GuidedSceneGen utilizes Holodeck to convert textual descriptions into diverse 3D indoor room layouts defined in a genuine metric coordinate system. The generated proxy delineates room structural boundaries while equipping each object instance with a complete 9D pose (3D location, 3D metric scale, and 3D orientation). From the scene origin, the system renders a \(360^\circ \times 180^\circ\) cubemap consisting of 6 orthogonal perspective faces, outputting metric depth maps along with dense semantic maps adhering to the NYUv2-40 taxonomy. This proxy geometry serves as an invariant geometric and semantic scaffold throughout all subsequent synthesis steps.

2. Semantic- and Geometric-Guided Panoramic Diffusion: Suppressing Spherical Distortions via Multi-ControlNet Cubemaps Standard equirectangular representations (ERP) suffer from severe polar stretching, which frequently degrades diffusion model outputs into distorted geometries. The framework represents panoramas as 6-face cubemaps to maintain uniform pixel distributions across perspective views. Built on Stable Diffusion 2, the backbone incorporates inflated self- and cross-attention modules that reshape feature tokens from \(b \times (hw) \times l\) to \(b \times (t \cdot hw) \times l\) across \(t=6\) views, enabling pre-trained 2D weights to share inter-view spatial context. To teach the model spatial connectivity between adjacent cube faces, 3D coordinate positional encodings (\(xyz\)-positional encoding) are injected into pixel tokens. Furthermore, a Multi-ControlNet architecture is integrated by initializing two separate ControlNet branches for depth and semantic maps from the trained multi-view U-Net, firmly enforcing that the synthesized \(360^\circ\) panorama strictly respects the proxy's spatial boundaries and semantic layout.

3. Collision-Aware Camera Trajectory Planning and Depth-Guided Bisection Alignment: Balancing Blind-Spot Exploration and Metric Consistency Unprojecting the initial panorama leaves extensive occluded regions behind furniture and room corners. Naive unconstrained view generation incurs prohibitive computational costs and introduces severe multi-view inconsistencies. To resolve this, the method divides the room into four equal spatial quadrants and plans sparse circular camera trajectories oriented toward each quadrant's center. Using proxy depth renderings, candidate camera poses within 0.3 meters of any wall, ceiling, or furniture are discarded, preventing degenerated frames caused by visual clipping or near-field collision. Because the underlying video diffusion model (Stable Virtual Camera, SVC) operates within an internal \([-2, 2]^3\) normalized cube, directly executing trajectories causes severe scaling mismatch. The paper adapts a bisection search over the camera-scale parameter \(\theta_{cs} \in [0.1, 2.0]\) by minimizing the \(\ell_1\) depth difference against the proxy's metric depth:

\[\theta^*_{cs} = \arg\min_{\theta_{cs}} \mathcal{L}_{\text{dpt}}, \quad \mathcal{L}_{\text{dpt}} = \frac{1}{N} \sum_{i=1}^N \| D^i_{\text{gt}} - D^i_{\theta_{cs}} \|_1\]

where \(D^i_{\theta_{cs}}\) is the metric depth predicted by Depth Anything v2 on the video frames generated under parameter \(\theta_{cs}\), and \(D^i_{\text{gt}}\) is the ground-truth metric depth rendered from the 3D proxy at pose \(i\). This replaces brute-force linear sweeping with efficient convergence, ensuring that synthesized novel views match target metric poses accurately.

4. Absolute-Scale 3DGS Scene Reconstruction: Pose-Free Radiance Field Optimization Because all synthesized views are geometrically grounded in the global proxy frame, reconstruction proceeds without external Structure-from-Motion (SfM) or pose optimization tools like COLMAP. The training dataset compiles 168 novel-view frames (4 trajectories \(\times\) 42 frames) alongside 170 sampled perspective views from the initial panorama, yielding a total of 338 views with ground-truth metric poses and intrinsics. The 3DGS radiance field is optimized using a composite objective:

\[\mathcal{L} = \mathcal{L}_{\text{3DGS}} + \mathcal{L}_{\text{geom}} + \mathcal{L}_{\text{NN}} + \mathcal{L}_{\text{depth}}\]

where \(\mathcal{L}_{\text{3DGS}}\) is the standard photometric loss (\(\ell_1\) combined with D-SSIM), \(\mathcal{L}_{\text{geom}}\) enforces surface normal, geometric, and multi-view photometric consistency, \(\mathcal{L}_{\text{NN}}\) minimizes \(k\)-nearest-neighbor distances between Gaussian centroids and the initial proxy point cloud, and \(\mathcal{L}_{\text{depth}}\) supervises rendering using metric depth maps. The resulting 3DGS scene inherits the 9D bounding boxes and semantic labels directly, and effortlessly accommodates multi-room incremental expansion by removing Gaussian primitives belonging to connecting doors/walls without retraining previous scenes.

Loss & Training

The panoramic diffusion generator is trained in two consecutive stages: first, the multi-view inflated U-Net is trained for 120k iterations with batch size 2 using AdamW (\(\beta_1=0.9, \beta_2=0.999, \text{weight decay}=0.01\)) and a base learning rate of 1e-5 (linearly warmed up over 10k steps) under \(v\)-prediction with a 1000-step DDPM schedule on datasets including Structured3D and Matterport3D; second, the depth and semantic Multi-ControlNet adapters are fine-tuned for 90k iterations with batch size 1 on Structured3D. Inference executes 50-step DDIM sampling with a classifier-free guidance (CFG) scale of 8.0. 3DGS optimization directly fits the 338 ground-truth posed views in the metric coordinate frame.

Key Experimental Results

Main Results

Evaluated on complex textual scene descriptions from Structured3D, the table compares novel view synthesis (NVS) visual quality (IS), multi-view geometric consistency (MEt3R, lower is better), user study scores across Perceptual Quality (PQ), 3D Consistency (3DC), and Overall Scene Quality (SQ) on a 1–5 scale, alongside average per-scene runtime.

Method	IS ↑	MEt3R ↓	PQ ↑	3DC ↑	SQ ↑	Mean Runtime
DreamScene360 (ECCV 2024)	1.91	0.026	2.34	2.44	2.38	0.5h
LayerPano3D (SIGGRAPH 2025)	2.00	0.031	2.08	1.79	1.94	0.2h
WorldExplorer (SIGGRAPH Asia 2025)	2.27	0.033	1.42	1.51	1.35	7.0h
GuidedSceneGen (Ours)	2.22	0.024	3.03	3.05	3.01	0.75h

Ablation Study

Ablation of guidance modalities for the panoramic image generator (evaluated on Structured3D text descriptions):

Config	CLIP Score ↑	Q-Align ↑	CLIP-IQA+ ↑	IS ↑	Note
Ours (text only)	31.17	4.45	0.69	3.33	Baseline multi-view inflated U-Net
Ours (text + Depth-ControlNet)	31.85	4.49	0.70	3.33	Geometric boundary constraint
Ours (text + Semantic-ControlNet)	32.23	4.49	0.70	3.41	Layout & semantic structure grounding
Ours (full, text + Multi-ControlNet)	32.48	4.54	0.71	3.26	Dual depth & semantic guidance

Quantitative evaluation of absolute metric scale and geometric alignment (comparing 3D proxy vs. reconstructed 3DGS scene):

Evaluation Aspect	Cam. Transl. RMSE [m] ↓	Cam. Transl. Median [m] ↓	Cam. Rot. RMSE [°] ↓	Cam. Rot. Median [°] ↓	Depth RMSE [m] ↓	Depth AbsRel ↓
3D Proxy vs. 3DGS Alignment	0.036	0.032	1.05	0.77	0.117	0.014

Key Findings

• Superior 3D Consistency and User Preference: GuidedSceneGen achieves the lowest MEt3R error (0.024), demonstrating robust multi-view geometric stability. In user evaluations, it secures top marks across PQ (3.03), 3DC (3.05), and SQ (3.01), significantly outperforming DreamScene360 and LayerPano3D, while WorldExplorer collapses under long-range geometric drift (SQ: 1.35).
• Nearly 10× Computational Speedup: By relying on 4 targeted quadrant trajectories (338 total views) instead of unconstrained, exhaustive exploratory sampling, GuidedSceneGen slashes runtime from 7.0 hours (WorldExplorer) down to 0.75 hours per scene (a 9.3× acceleration) on a single consumer GPU.
• Centimeter-Level Absolute Metric Precision: Alignment verification demonstrates a median translation error of only 3.2 cm, a median rotation error of 0.77°, and a depth absolute relative error (AbsRel) of 1.4%, rigorously proving that the generated 3DGS representation faithfully adheres to physical metric ground-truth.

Highlights & Insights

• Promoting 3D Proxy to an End-to-End Metric Anchor: Unlike prior works that rely on post-hoc uncalibrated registration or relative camera scaling, this framework uses text-predicted 3D layouts as an invariant metric foundation, completely eliminating error-prone SfM and pose estimation.
• Depth-Supervised Bisection Scale Alignment: The paper identifies the fundamental coordinate gap between camera-controlled video diffusion models and real-world dimensions, formulating an elegant bisection search over metric depth differences that prevents visual clipping and collision artifacts.
• Modular Multi-Room Expansion: Because each room is synthesized in absolute global coordinates, multi-room scenes can be synthesized iteratively and independently; seamlessly merging environments requires only pruning the Gaussian primitives corresponding to connector walls or doors.

Limitations & Future Work

• Reliance on Structured Indoor Assumptions: The four-quadrant exploration pattern and cubemap representations assume bounded indoor rooms with planar walls and floors, making direct application to unbounded, irregular outdoor environments non-trivial.
• Coarse-Grained Geometry for Small Objects: The 3D proxy provides furniture-level 9D bounding boxes, providing weaker direct geometric guidance for small table-top decorative items or fine wireframes.
• Sensitivity to Layout Generation Errors: If the upstream text-to-layout generator produces self-intersecting furniture or abnormal room dimensions for unusual prompts, downstream depth guidance and trajectory estimation inherit these spatial inaccuracies.

Related Work & Insights

• vs WorldExplorer: WorldExplorer couples panoramic synthesis with video diffusion via autoregressive exploration, but its ungrounded trajectory causes severe geometric drifting and requires 7 hours per scene; GuidedSceneGen utilizes explicit 3D proxy constraints and collision-aware planning, achieving a ~10× speedup without geometric drift.
• vs LayerPano3D: LayerPano3D Progressive inpainting creates layered flat planes from panoramic images, but lacks authentic 3D spatial depth and suffers from visible edge tearing under off-center viewpoints; GuidedSceneGen reconstructs full volumetric geometry via 3DGS under rigorous metric supervision.
• vs DreamScene360: DreamScene360 optimizes 3DGS by perturbing views locally around the center and blurs out unobserved blind spots; GuidedSceneGen actively plans exploratory paths to generate true novel content for previously occluded regions.

Rating

• Novelty: ⭐⭐⭐⭐ [First text-to-3D framework preserving end-to-end absolute metric scale, paired with depth-guided bisection video trajectory calibration]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across panoramic 2D metrics, multi-view consistency, metric alignment accuracy, runtime, and user studies]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear structural flow, precise mathematical formulations, and rigorous empirical analysis]
• Value: ⭐⭐⭐⭐⭐ [Critical enabler for robotics, embodied AI, and interactive CAD environments needing metric-accurate 3D scene synthesis]