Taming LLMs for Codematic Indoor Scene Generation

Conference: ECCV 2026
Paper: ECCV Official
Code: To be released
Area: 3D Vision
Keywords: 3D Scene Generation, Codematic Layout Representation, LLM Fine-tuning, Conditional Mutual Information, Language Planning

TL;DR

Addressing the core issue where fine-tuning LLMs for indoor 3D scene code generation leads to prompt neglect caused by token-heavy layout history, SceneSpinner curates the first large-scale open-set dataset of 52K rooms and introduces a language-based planning stage alongside a Conditional Mutual Information (CMI) regularization objective, substantially improving fine-grained layout precision and instruction-following fidelity.

Background & Motivation

Generating physically plausible and realistic 3D indoor layouts from natural language descriptions is a foundational challenge in computer graphics and embodied AI simulation. Early procedural approaches relied heavily on manual heuristic rules and hardcoded constraints, which fundamentally struggled to generalize to open-ended user requests or produce diverse configurations. Later data-driven approaches employed specialized diffusion or autoregressive architectures to learn layout distributions from 3D datasets. However, these specialized networks are bounded by closed-vocabulary categories and lack holistic semantic reasoning, preventing them from supporting flexible, open-vocabulary instructions. With the rise of large language models (LLMs) equipped with vast common sense, researchers have turned to prompting LLMs to directly generate structured scene codes or utilizing multi-agent reflection loops with physical verifiers. Yet, empirical evidence shows that downstream refinement strategies critically hinge on the initial layout proposal quality; even frontier models like GPT-4o fail frequently in fine-grained 3D coordinates, rotations, and scales without specialized grounding.

Attempting to fine-tune LLMs directly to generate structured scene representations (codematic indoor scene generation) reveals two fundamental roadblocks. First, existing 3D scene datasets are scarce, fragmented, and heterogeneous: real-world scans like Matterport3D suffer from severe geometric noise and missing objects, whereas synthetic CAD datasets like 3D-FRONT and Structured3D are strictly limited to closed-set categorical labels. None provides the complete combination of open-vocabulary textual descriptions, precise 7-DoF bounding boxes (position, scale, yaw angle), and multi-perspective holistic scene captions required for effective LLM instruction tuning. Second, an intrinsic information imbalance arises during autoregressive generation: a user prompt is typically concise (around 10 tokens), while the generated layout code history quickly balloons to hundreds or thousands of tokens. Because the layout code tokens are statistically self-similar to the target distribution and vastly outnumber the initial instruction, the LLM's attention becomes completely overwhelmed by previous layout code, causing it to disregard specific user constraints (for instance, ignoring an explicit instruction for a bunk bed and desk by defaulting to a standard double bed).

To establish robust spatial intuition within LLMs, solutions must address both data infrastructure and attentional dynamics. Core idea: unify and re-annotate heterogeneous 3D scene datasets into the first large-scale open-vocabulary corpus of 52K rooms, and propose SceneSpinner, which bridges the semantic-token gap via a chain-of-thought language planning stage and explicitly prevents instruction neglect through a Conditional Mutual Information (CMI) regularization training recipe.

Method

Overall Architecture

SceneSpinner leverages Qwen3-4B-Instruct as its foundation model to directly translate natural language user inputs into executable 3D scene code (comprising wall geometries, door/window fixtures, and 7-DoF bounding boxes accompanied by open-vocabulary object descriptions), which is subsequently rendered into a complete 3D scene via a 3D asset renderer. The overall pipeline is driven by three interconnected components: unified multi-source open-set data engineering, chain-of-thought language planning, and dual-branch CMI-regularized training.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Heterogeneous Datasets<br/>(SpatialLM / 3D-FRONT / MP3D etc.)"] --> B["Unified Open-Set Data Engineering<br/>Coordinate normalization + heuristic filtering + hierarchical VLM captions"]
    B --> C["Language Planning Stage<br/>Designer CoT &lt;think&gt; constructing spatial semantic anchors"]
    C --> D["CMI Regularization Training Recipe<br/>Dual-branch forward passes + KL divergence penalty bounding CMI"]
    D --> E["Structured Scene Code Output &lt;scene&gt;...&lt;/scene&gt;<br/>(Walls, fixture objects, 7-DoF open-set bounding boxes)"]

Key Designs

1. Unified Open-Set Data Engineering: building the first open-vocabulary 3D codematic corpus

Existing indoor datasets vary wildly in data format and completely lack open-vocabulary textual descriptions, making them unsuitable for LLM instruction tuning. This module unifies five heterogeneous sources: SpatialLM, Structured3D, Matterport3D, 3D-FRONT, and InfiniGEN, spanning real-world scans, synthetic procedural scenes, and professional designs. First, all scenes are standardized into a canonical coordinate frame: the geometric center is set to the origin \((0,0,0)\), the Y-axis aligns with gravity (floor plane at \(y=0\)), \(+X/-X\) map to right/left, \(+Z/-Z\) represent back/front, and object front faces align with \(+Z\). Next, a three-fold geometric quality filter eliminates low-quality data: rooms with fewer than 3 objects, scenes with oversized furniture occupying over one-third of the total volume, and rooms containing invalid object collisions are pruned. Finally, a hierarchical captioning pipeline generates rich text descriptions: mesh-bearing objects are rendered across four canonical views and captioned by a Vision-Language Model (VLM), while mesh-less scanned instances are highlighted via projected 2D bounding boxes for VLM captioning. For scene-level descriptions, a template-based engine procedurally creates spatial relation prompts (directions and distances) on-the-fly, while a VLM-based engine synthesizes holistic captions capturing room category, layout functionality, and design intent, yielding 52,071 valid rooms, 281,586 pairs, and 661,614 open-set object descriptions.

2. Language Planning Stage: bridging the autoregressive token density imbalance

During autoregressive layout generation, concise user instructions are easily drowned out by lengthy sequences of previously generated code tokens. Inspired by chain-of-thought reasoning, this design decomposes the generation pipeline into an explicit three-stage sequence: "Instruction \(\rightarrow\) Plan \(\rightarrow\) Code". Before emitting any coordinate code, the model is prompted to articulate an interior designer's cognitive thought process inside <think> tags: establishing architectural footprints and room boundaries, positioning primary doors and windows for natural lighting, structuring functional zones and furniture orientations, and logically inferring unmentioned but essential everyday items. To generate high-quality supervision, VLM is prompted with ground-truth layouts and renderings under few-shot guidance to synthesize coherent planning rationales. This intermediate textual plan expands the input context to a token volume comparable to the output layout code, balancing conditional attention distributions and establishing explicit spatial keywords (e.g., 'right', 'corner', 'center') that act as semantic anchors grounding subsequent geometric coordinates.

3. CMI Regularization Training Recipe: quantifying and constraining attentional drift

Even with intermediate textual planning, autoregressive models can still develop shortcut behaviors that over-rely on layout history while ignoring nuanced instruction details. To rigorously quantify and constrain the model's dependence on the instruction, this design introduces Conditional Mutual Information (CMI) from information theory. The CMI between the generated layout action \(a_{\text{code}}\) and the text condition \(c_{\text{text}}\) given layout history \(c_{\text{code}}\) is formulated as:

\[I(a_{\text{code}}; c_{\text{text}} \mid c_{\text{code}}) = H(a_{\text{code}} \mid c_{\text{code}}) - H(a_{\text{code}} \mid c_{\text{text}}, c_{\text{code}})\]

Applying the probability chain rule, this mutual information is mathematically equivalent to the expected Kullback-Leibler (KL) divergence between the action posterior conditioned on both text and code, and the posterior conditioned on code alone:

\[I(a_{\text{code}}; c_{\text{text}} \mid c_{\text{code}}) = \mathbb{E} \left[ D_{\text{KL}}\left( p(a_{\text{code}} \mid c_{\text{text}}, c_{\text{code}}) \parallel p(a_{\text{code}} \mid c_{\text{code}}) \right) \right]\]

In next-token prediction, the action space corresponds to the discrete vocabulary, and conditional probabilities are directly given by applying softmax to output logits. Because completely dropping \(c_{\text{text}}\) degrades natural text distributions and incurs useless forward overhead, an efficient substitution strategy is proposed: instead of zeroing out text, \(c_{\text{text}}\) is replaced with a coarse-grained room category label \(c_{\text{label}}\) (e.g., "bedroom"). Running a single batched forward pass over both the specific instruction \(c_{\text{text}}\) and the generic label \(c_{\text{label}}\) computes cross-entropy against ground-truth layout codes for both paths, while penalizing negative KL divergence explicitly forces the model to maintain sensitive dependence on fine-grained instructions.

Loss & Training

During training, each batch samples a room from the corpus to construct the detailed condition input \(I_{\text{long}}\) (comprising user instructions and probabilistically included planning sequences) and the coarse category condition input \(I_{\text{short}}\). Forwarding both sequences through the LLM produces output logits, from which cross-entropy losses \(\mathcal{L}_{\text{long}}\) and \(\mathcal{L}_{\text{short}}\) are computed against ground-truth layout tokens. Slicing the action logits corresponding to layout code gives \(z_{\text{long}}\) and \(z_{\text{short}}\). Applying a stop-gradient (\(\text{sg}\)) operation to the short-condition logits yields the KL divergence regularization term:

\[\mathcal{L}_{\text{kl}} = D_{\text{KL}}\left( \text{softmax}(z_{\text{long}}) \parallel \text{softmax}(\text{sg}(z_{\text{short}})) \right)\]

The overall optimization objective is defined as:

\[\mathcal{L} = \mathcal{L}_{\text{long}} + \beta \mathcal{L}_{\text{short}} - \lambda \mathcal{L}_{\text{kl}}\]

where \(\beta\) and \(\lambda\) are loss balancing hyperparameters. Maximizing mutual information via \(-\lambda \mathcal{L}_{\text{kl}}\) prevents the LLM from neglecting user prompts during deep autoregression. Fine-tuning builds upon Qwen3-4B-Instruct, incorporating data mixtures from Mixture of Thought and Ultra-Chat to preserve general language comprehension and mathematical reasoning capabilities.

Key Experimental Results

Main Results

The evaluation benchmark consists of 120 diverse prompts encompassing various room types, multi-perspective layouts, and complex design purposes. Evaluation metrics include Instruction Following score (Ins_Follow, evaluated by GPT-4o on a 1-10 scale), Layout Fidelity and Completeness score (Lay_Fidelity&Comp, evaluated by GPT-4o for physical plausibility and completeness on a 1-10 scale), and CLIP Score evaluated on top-down views synthesized via the open-source Cube 3D generation framework.

Model	Paradigm	Ins_Follow ↑	Lay_Fidelity&Comp ↑	CLIP_Score ↑
M3DLayout-Diff	Diffusion (Closed-set)	3.058	3.691	0.182
M3DLayout-AR	Autoregressive (Closed-set)	3.608	3.908	0.187
Ctrl-Room	Constrained Diffusion	4.841	5.291	0.187
I-Design	LLM Agent (Few-shot)	4.222	4.533	0.167
HoloDeck	LLM Agent (Modular)	5.722	6.037	0.259
SceneSpinner (Ours)	LLM Fine-tuning + Plan + CMI	7.852	7.191	0.260

Ablation Study

The ablation quantitatively inspects the individual and combined contributions of the language planning phase and the CMI regularization recipe.

Config	Planning Phase	CMI Recipe	Ins_Follow ↑	Lay_Fidelity&Comp ↑	Note
Vanilla Fine-tuning	\(\times\)	\(\times\)	6.371	6.557	Suffers from token imbalance & history dominance
CMI Recipe Only	\(\times\)	\(\checkmark\)	6.988	6.806	Explicitly constrains condition dependence
Full Model (Ours)	\(\checkmark\)	\(\checkmark\)	7.631	7.157	Synergistic combination yields highest performance

Key Findings

• Substantial improvements over SOTA: SceneSpinner outperforms the strongest modular agent baseline (HoloDeck) by 37.2% in instruction following (7.852 vs. 5.722) and by 19.1% in layout fidelity and completeness (7.191 vs. 6.037), confirming that instilling intrinsic spatial coding ability into LLMs is far more effective than relying on iterative external agent corrections on suboptimal proposals.
• Attention grounding via semantic anchors: Attention visualization confirms that without the planning phase, LLM attention quickly drifts from prompt tokens to repetitive layout code syntax. With planning, the model consistently attends to spatial keywords like "right", "corner", and "center" in the generated plan, using them as semantic anchors to guide subsequent 3D coordinate generation.
• Decoupling prevention through CMI: In models trained without CMI regularization, a notable failure mode emerges where the LLM produces a completely correct chain-of-thought rationale but fails to follow it in the emitted bounding box code; incorporating the KL regularization effectively binds internal reasoning with final geometric layout outputs.

Highlights & Insights

• Foundational open-vocabulary 3D corpus: Systematically harmonizing five heterogeneous datasets into 52K rooms with open-vocabulary instance captions resolves the long-standing data scarcity bottleneck for LLM-based 3D scene synthesis.
• Diagnosing autoregressive token imbalance: Pinpointing the underlying structural flaw where short user instructions are swamped by long layout sequences provides a critical theoretical perspective for LLM code generation tasks.
• Elegant information-theoretic regularization: Formulating the instruction-dependence objective via Conditional Mutual Information and executing it via a computationally efficient dual-branch short-label contrast avoids artificial zero-token masks while preventing instruction fading.

Limitations & Future Work

• Dependency on downstream asset generation: SceneSpinner specializes in generating structured 3D layout coordinates and semantic descriptions; final visual fidelity depends on external asset retrieval or generative models (such as Cube), where rare or specialized furniture items may lack high-fidelity 3D meshes.
• Absence of explicit physics simulation loops: While common-sense layout collisions and floating objects are largely mitigated through fine-tuning, the current feed-forward generation does not perform hard-boundary physical simulation for micro-level contact alignments or complex physical interactions.
• Scaling to multi-room house-level planning: The current formulation primarily addresses single-room layouts; extending the planning hierarchy to multi-room apartments and structural multi-story building topologies represents a promising future avenue.

Related Work & Insights

• vs I-Design / HoloDeck: I-Design and HoloDeck use training-free prompting and modular agent verifiers. However, foundation LLMs without specialized fine-tuning lack grounded geometric intuition, frequently generating unrealistic bounding boxes that verification loops fail to rescue. SceneSpinner directly fine-tunes the base LLM, providing high-quality initial layout proposals natively.
• vs Ctrl-Room / M3DLayout: Traditional diffusion and autoregressive layout synthesizers operate over rigid, closed-set category vocabularies and struggle with nuanced descriptive constraints. SceneSpinner leverages the generative open-vocabulary power of LLMs, seamlessly accommodating intricate descriptive prompts and exact metric dimension constraints.

Rating

• Novelty: ⭐⭐⭐⭐☆ Uncovers the information imbalance mechanism in autoregressive scene coding and introduces a principled CMI-driven planning paradigm.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive 52K-scene data curation, rigorous multi-dimensional GPT-4o evaluations, detailed attention maps, and training dynamics analysis.
• Writing Quality: ⭐⭐⭐⭐⭐ Well-structured narrative with rigorous information-theoretic derivations and insightful empirical diagnostics.
• Value: ⭐⭐⭐⭐⭐ Establishes a reproducible benchmark and methodology for fine-tuning LLMs as native 3D spatial layout generators.