LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models

Conference: CVPR 2025
arXiv: 2412.02193
Code: Project Page
Area: Multimodal VLMs
Keywords: 3D Layout Generation, Differentiable Optimization, Spatial Relations, VLM Spatial Reasoning, Open-vocabulary Scene Synthesis

TL;DR

This paper proposes LayoutVLM, which leverages the semantic knowledge of VLMs to generate a dual scene layout representation containing numerical pose estimations and spatial relation constraints. By jointly optimizing semantic objectives and physical plausibility constraints via differentiable optimization, it significantly outperforms existing methods across 11 room types.

Background & Motivation

Background:
Open-vocabulary 3D indoor scene layout generation is a core task in robotics and simulation. Recently, LLMs/LMMs have been employed to generate diverse scene layouts based on natural language instructions.

Limitations of Prior Work:
1. Methods directly predicting numerical poses like LayoutGPT exhibit strong semantic alignment but frequently incur physical implausibility issues such as object collisions and boundary violations. 2. Holodeck improves physical plausibility by predicting spatial relations coupled with constraint satisfaction search, but struggles to find feasible solutions in dense scenes with numerous objects. 3. Traditional methods rely on predefined object categories and fixed placement patterns, failing to achieve true open-vocabulary scene generation. 4. Existing LLM-based methods lack direct utilization of visual information, relying solely on pure text for spatial reasoning.

Key Challenge:
3D layout generation must simultaneously satisfy physical plausibility (no collisions, inside boundaries) and semantic consistency (aligning with language instructions). Existing methods often suffer from one or the other.

Goal:
To design an open-vocabulary 3D layout generation method that ensures both physical plausibility and semantic alignment.

Key Insight:
Numerical poses and spatial relations are treated as complementary dual representations—numerical poses provide initial values for optimization, while spatial relations preserve semantics during optimization in the form of differentiable objective functions.

Core Idea:
VLMs generate both initial object poses and spatial relation constraints simultaneously. Differentiable optimization is then leveraged to adjust poses to achieve physical plausibility while preserving semantics.

Method

Overall Architecture

The workflow of LayoutVLM is as follows: 1. Generate textual descriptions and orientation annotations for each 3D object using a VLM. 2. Group objects by functionality using an LLM. 3. Generate scene layout representations (numerical poses + spatial relations) group-by-group using a VLM. 4. Filter unreliable spatial relations using self-consistency decoding. 5. Jointly optimize semantic and physical objectives via differentiable optimization.

Final optimization objective: \(\arg\min_{\{p_i\}_{i=1}^{N}}(\mathcal{L}_{\text{semantic}} + \mathcal{L}_{\text{physics}})\)

Key Designs

Key Design 1: Dual Scene Layout Representation

Function: To design a scene representation that both expresses rich semantics and supports precise physical optimization.

Mechanism: The scene layout representation consists of two parts: - Numerical Pose Estimation \(\{\hat{p}_i\}_{i=1}^{N}\): The 3D position \((x,y,z)\) and rotation angle \(\theta\) around the z-axis for each object, serving as the initial solution for optimization. - Differentiable Spatial Relations: Five types of spatial relation constraints, each corresponding to a differentiable objective function: - distance: The distance between two objects should be within the range \([d_{\min}, d_{\max}]\). - on_top_of: One object is placed on top of another. - align_with: Two objects are aligned at a specified angle. - point_towards: One object points towards another. - against_wall: The object is placed against a wall.

Design Motivation: Pure numerical methods (LayoutGPT) achieve good semantics but poor physical plausibility; pure constraint-based methods (Holodeck) are difficult to solve in complex scenes. The dual representation is complementary—the initial poses ensure the correct optimization direction, while the spatial relations maintain semantics during optimization without being corrupted.

Key Design 2: Visual Prompting and Self-Consistency Decoding

Function: To improve the accuracy and reliability of the scene representation generated by the VLM.

Visual Prompting: - Annotating coordinate grids (every 2 meters) on the 3D scene render to help the VLM estimate scales. - Visualizing coordinate axes to maintain consistent spatial reference. - Drawing orientation arrows on objects to assist in generating rotation constraints. - Re-rendering the scene after group-by-group placement, allowing the VLM to see the occupied regions.

Self-Consistency Decoding: Spatial relations generated by the VLM might be inconsistent with numerical poses. Only the spatial relations already satisfied in the initial poses are preserved: $\(\mathcal{L}_{\text{semantic}} = \sum_{\mathcal{L} \in \mathcal{R}} \mathbb{1}[\mathcal{L}_i(\hat{p}_i, \hat{p}_j, \lambda) \leq \epsilon] \cdot \mathcal{L}_i(p_i, p_j, \lambda)\)$

Design Motivation: While a VLM can be accurate when predicting spatial relations for individual pairs of objects, it struggles to guarantee global consistency. Self-consistency decoding filters unreliable constraints by requiring mutual consistency between the dual representations.

Key Design 3: Differentiable Physical Optimization

Function: To ensure the physical plausibility of the layout through gradient optimization.

Mechanism: Using Distance-IoU loss for collision avoidance: $\(\mathcal{L}_{\text{physics}} = \sum_{i=1}^{N}\sum_{j \neq i}^{N} \mathcal{L}_{\text{DIoU}}(p_i, p_j, b_i, b_j)\)$

Optimizing via Projected Gradient Descent (PGD), where objects are projected back inside the room boundaries every fixed number of iterations.

VLM Fine-tuning: Scene layout representations can be automatically extracted from the 3D-Front dataset (~9,000 rooms) as training data to fine-tune open-source VLMs (e.g., LLaVA-NeXT-Interleave), significantly enhancing their spatial reasoning capabilities.

Design Motivation: Traditional constraint satisfaction search often fails in dense scenes, whereas gradient-based optimization is more robust and scalable.

Key Experimental Results

Main Results: Average Performance across 11 Room Types

Method	CF↑	IB↑	Pos.↑	Rot.↑	PSA↑
LayoutGPT	83.8	24.2	80.8	78.0	16.6
Holodeck	77.8	8.1	62.8	55.6	5.6
I-Design	76.8	34.3	68.3	62.8	18.0
LayoutVLM	81.8	94.9	77.5	73.2	58.8

• PSA (Physical-Semantic Alignment score) improves by 40.8 points compared to the best baseline.
• In-Boundary (IB) score improves from 34.3% to 94.9%.
• Achieves the best PSA score across all 11 room types.

Ablation Study

• Self-consistency decoding improves the PSA from 50.4 to 58.8 (+8.4).
• Removing spatial relations and relying solely on numerical poses drops the PSA by approximately 15 points.
• Removing numerical initialization and relying solely on spatial relations causes the optimization to easily get stuck in local optima.

VLM Fine-Tuning Results

• GPT-4o fine-tuning further improves PSA.
• LLaVA-NeXT fine-tuning improves the model from being nearly unusable to highly competitive.

Highlights & Insights

1. Exquisite Dual Representation: Numerical poses and spatial relations complement and mutually verify each other, avoiding the physical issues of pure numerical methods and overcoming the tractability challenges of pure constraint-based methods.
2. Clever Self-Consistency Decoding: Leveraging the consistency between two representations to filter unreliable constraints serves as an elegant solution for VLM uncertainty handling.
3. Differentiable Optimization Replacing Search: Substituting gradient-driven optimization for constraint satisfaction search greatly enhances scalability.
4. Solid Visual Prompt Engineering: Visual annotations such as coordinate grids and orientation arrows significantly improve VLM spatial reasoning.
5. Automated Representation Extraction: Scene layout representations can be automatically extracted from existing datasets for fine-tuning without requiring manual annotation.

Limitations & Future Work

1. Reliance on closed-source VLMs like GPT-4o, resulting in high costs and irreproducibility.
2. The strategy of placing objects group-by-group may lead to insufficient coordination between groups.
3. Only single-axis rotation around the z-axis is supported, making it unable to handle complex rotations.
4. Physical constraints only consider collisions and boundaries, without modeling gravity, support relations, etc.
5. Evaluation metrics rely on GPT-4o scoring, which may introduce subjectivity into the evaluation.

Related Work & Insights

• LayoutGPT [Feng et al.]: Directly generates numerical poses with LLMs, showing good semantics but poor physical plausibility.
• Holodeck [Yang et al.]: Spatial relations + constraint satisfaction search, leading to poor scalability.
• I-Design [Hu et al.]: Iterative LLM layout generation.
• Insight: Combining the semantic knowledge of VLMs with differentiable optimization can be extended to other spatial reasoning tasks, such as robotic manipulation planning. The idea of self-consistency decoding is valuable for all VLM generation tasks.

Rating

⭐⭐⭐⭐ (4/5)

Reason: The problem is clearly defined, the design scheme of dual representation + differentiable optimization is elegant, and the self-consistency decoding method is novel. Experiments demonstrate a qualitative leap in physical plausibility. The primary limitations lie in the dependency on closed-source models and the subjectivity of the evaluation framework.

Related Papers

• [CVPR 2026] HOG-Layout: Hierarchical 3D Scene Generation, Optimization and Editing via Vision-Language Models
• [CVPR 2025] RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics
• [CVPR 2025] Debiasing Multimodal Large Language Models via Noise-Aware Preference Optimization
• [CVPR 2025] Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment
• [CVPR 2025] Generalized Few-Shot 3D Point Cloud Segmentation with Vision-Language Model