3D-Layout-R1: Structured Reasoning for Language-Instructed Spatial Editing

Conference: ECCV 2026
Paper: Official ECCV page
PDF: Full paper
Area: Vision-Language Reasoning / 3D Layout Editing
Keywords: Structured reasoning, scene-graph editing, 3D sorting, geometric rewards, GRPO

TL;DR

3D-Layout-R1 trains models to update JSON 3D scene graphs while reasoning, then refines their outputs with IoU, collision, and format rewards through GRPO, raising Qwen3-8B's mean IoU on perfect-input sorting from 0.850 with Vanilla GRPO to 0.924, although gains and metric trade-offs depend substantially on the task.

Background & Motivation

Understanding that a chair is to the right of a desk is different from executing an instruction to move it there while aligning it with a sofa. SpatialRGPT, SpatialLLM, and 3D-R1 primarily strengthen spatial question answering. Editing a layout additionally requires identifying what should change, preserving the attributes of unaffected objects, and converting language relations into numerical positions and orientations. Sequential instructions make this harder: after moving a reference object, later operations must use its updated state.

Existing layout systems often let a language model propose high-level constraints and delegate final object poses to an external solver or differentiable optimizer. This incorporates explicit geometric rules but separates instruction interpretation from layout computation. Letting a model write an unrestricted reasoning trace does not necessarily solve the problem either. In sorting, for example, a gap may be confused with a center-to-center distance, or a group's width may be calculated without its internal spaces. If intermediate reasoning remains entirely verbal, these ambiguities may surface only when the model finally produces coordinates.

The paper therefore uses the scene graph both as input and as an evolving working state. Intermediate outputs are not merely explanations: they contain object identifiers and numerical JSON edits that can be inspected. Core idea: first teach editing through supervised trajectories with explicit scene-graph updates, then optimize target geometry overlap, internal collision avoidance, and output formatting so that reasoning produces checkable changes to the 3D state.

Method

Overall Architecture

Inputs comprise a natural-language instruction, an initial 3D scene graph, and scene images when a visual backbone uses them. The model outputs an updated complete graph together with a structured reasoning trace containing intermediate edits. It directly generates the final coordinates rather than delegating them to an external layout optimizer. A simulator or visual grounding module supplies the initial representation, while robot execution remains a downstream component.

Training consists of data construction, structured CoT-SFT for a cold start, and geometric-reward GRPO. The teacher receives the target graph when constructing supervised traces; the target graph is not an inference input. Likewise, the reward branch below is used only in training, not to correct predictions with ground truth during deployment.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Images, initial graph,<br/>and language instruction"] --> Data["Controlled Layouts<br/>and Teacher Traces"]
    Target["Target scene graph<br/>Training only"] --> Data
    Data --> Trace["Structured Graph-Edit<br/>Cold Start"]
    Trace --> Reward["Geometric and Format<br/>Reward Optimization"]
    Target --> Reward
    Reward --> Model["Trained model"]
    Model --> Output["Stepwise graph edits<br/>and final scene graph"]
    Input -->|Inference inputs| Model

Key Designs

1. Controlled Layouts and Teacher Traces: make the editing target sufficiently determinate before supervising the path to it

The representation is a directed graph whose nodes are objects and supporting regions, with edges encoding contact or containment. Its serialized form is a JSON dictionary keyed by integer node IDs. Attributes include object type, 3D center, axis-aligned length/width/height, roll/pitch/yaw, and an optional textual caption. Numerical attributes support precise computation, captions help resolve references, and stable IDs preserve object identity across successive edits. The task is therefore an object-specific modification of an existing state, not unconstrained regeneration of a room.

The authors report 15k scenes overall and explicitly describe 10k synthetic sorting scenes and 1k Blender-rendered spatial-alignment scenes used for training. The cached paper does not fully enumerate the other task sizes or train/test splits, so the remainder should not simply be labeled the room-editing training set. Sorting instructions jointly specify grouping attributes, within-group sort attributes, group order, total span, and gaps. Alignment examples start from an organized category grid and randomly translate or rotate only a subset of objects; the model must restore those objects while leaving correctly placed ones unchanged.

Room editing draws on the InstructScene instruction corpus. A relation such as placing an object next to a sofa can admit many valid outputs, making a single target coordinate a potentially misleading supervisory signal. The authors add distances from each object to be inserted to two or three existing reference objects, narrowing the feasible placement set and describing the target as almost uniquely determined. Reducing ambiguity is not the same as mathematically guaranteeing uniqueness: distance constraints alone can retain multiple solutions in 3D, requiring semantic relations, support surfaces, or other conditions to further constrain placement.

Initial graphs are obtained through a rule-based auto-labeling framework using Blender annotations. DeepSeek-R1 receives the instruction, initial graph, and target graph, then generates an explanation connecting the initial and final states through object-level grouping, sorting, relational, and distance operations. These traces are training data, not independently proven programs. A target-conditioned teacher can produce a post-hoc explanation, and the cached paper does not provide a complete trace-filtering or step-verification protocol.

2. Structured Graph-Edit Cold Start: use intermediate states to store computational results rather than explanations alone

Answer-only SFT teaches a jump from inputs to final coordinates, leaving dependencies between ordering, group-width calculation, and sequential placement implicit. Here, CoT-SFT trains traces with explicit JSON updates: determine groups and ordering, allocate their spatial intervals, and then write object centers back into the graph. Once a group of spheres is placed, its numerical state can inform where the following cylinder group begins. This reduces the need to repeatedly reinterpret what a gap means in free-form prose.

Alignment uses the same state-update interface but different operations. The model infers a category grid, compares object centers to identify deviations, and recovers positions through interpolation along grid lines or extension from an endpoint. Room editing instead updates poses according to relational and distance constraints. The commonality is the representation and training interface, not a single sorting algorithm imposed on all three tasks. The trained model generates these operations; the pipeline does not imply a separate hard-coded executor that performs the numerical reasoning for it.

Formatting couples the explanation with a parseable result. Responses must have a matched <think> tag pair, at least one valid JSON block within that section, and a second valid JSON block in the final answer segment. The format reward checks this pattern and reduces the score for deviations such as incorrect tags or braces. Parseability establishes interface compliance, however, not the correctness of every intermediate step or the faithfulness of the explanation to the model's internal computation.

3. Geometric and Format Reward Optimization: refine parseable edits toward the correct layout

CoT-SFT supplies an initial policy capable of producing roughly correct layouts. GRPO then samples multiple candidates per input and updates the policy using their relative rewards within the group. The reward adds target IoU, collision avoidance within the predicted graph, and format compliance. The following restates the relationship described in Section 3.3; the cached main text does not specify numerical reward weights:

\[ r = r_{\mathrm{IoU}} + \lambda_{\mathrm{Coll}}\,\mathrm{Coll}(G_{\mathrm{pred}}) + \lambda_{\mathrm{Fmt}}\,\mathrm{Fmt}(G_{\mathrm{pred}}). \]

IoU calculation first establishes node correspondences. It attempts semantic matching when descriptions are available and otherwise selects the ground-truth node with the highest IoU if a unique semantic match cannot be found. For matched axis-aligned boxes, 3D IoU is their intersection volume divided by their union volume. The reward averages the matches, with the printed formula normalizing by the number of predicted nodes. Compared with binary exact-coordinate success, this distinguishes partial from substantial overlap; compared with center distance alone, it also reflects box extent. Nevertheless, fully disjoint boxes can all receive zero IoU, so a continuous geometric reward does not guarantee informative feedback for every distant candidate. The cached text does not completely specify one-to-one matching or the treatment of missing and extra nodes.

The collision term examines unordered pairs of non-container objects and counts an overlap only when the axis-aligned intersection volume exceeds a small tolerance. This avoids penalizing tiny numerical artifacts. The paper defines the collision-free score as:

\[ \mathrm{Coll}(G)=1-\frac{|\mathcal{C}|}{N}, \]

Here, \(\mathcal{C}\) is the set of colliding object pairs and \(N\) is the number of non-container objects. A collision-free layout scores 1. This is neither the fraction of collision-free objects nor a normalization by the total number of pairs; under the stated formula it can become negative if colliding pairs outnumber objects. Excluding containers avoids treating legitimate containment as interpenetration, but box-level overlap still does not establish mesh-level feasibility, stable support, or collision-free motion.

A Worked Example

The sorting instruction in Figure 2 groups objects by shape, sorts each group by increasing height, and places the groups in the order box, sphere, cylinder, cone. It specifies a total width of 12.506 meters, a 0.200-meter inter-group gap, a 0.05-meter within-group gap, y = 0 for all objects, and z equal to half the object's height. This illustrates why allocating intervals before computing centers is clearer than guessing an entire coordinate list.

First, the model identifies box nodes 4, 5, and 7 with heights of 0.799, 1.108, and 1.293 meters, establishing their order. Next, it determines group intervals: the box group occupies [-6.253, -3.783] meters along x, while the sphere group begins at -3.583 meters, leaving exactly 0.200 meters after the previous group's right edge.

The model then writes sphere centers into the graph: the figure gives x = -3.179 for node 3, -1.992 for node 8, and -0.320 for node 6. Adjacent center distances account for both objects' half-widths as well as the 0.05-meter gap; the gap is not a fixed center-to-center offset. Later groups continue from the allocated intervals, and the model returns a complete graph while the intermediate JSON preserves the ordering and coordinate calculations. These values come from Figure 2, not an additional constructed experiment.

Loss & Training

The supervised stage learns the teacher's structured reasoning and final-state outputs. The RL stage uses GRPO with group-normalized rewards, clipped policy updates, and KL regularization toward a reference policy. This note does not reproduce the corrupted GRPO equation in the text extraction, avoiding the presentation of extraction errors as algorithm definitions.

Post-training experiments use Qwen2.5-7B, Qwen2.5-VL-7B, Qwen3-8B, and Cosmos-Reason1-7B backbones, comparing Ans-SFT, CoT-SFT, Vanilla GRPO, and the full method. The cached main text does not specify learning rates, batch sizes, sampling group sizes, training steps, collision tolerances, or reward weights. It therefore cannot support a directly reproducible training configuration on its own.

Key Experimental Results

Main Results

Higher mean IoU is better. Ctr. Dist. is the average Euclidean distance between matched box centers, with lower values preferred; the original table header does not state its unit. [email protected] is the fraction of nodes whose IoU is at least 0.5. Col. Free follows the collision formula above, and Edit Dist. is the average Levenshtein distance between predicted and ground-truth object orders. The selected comparisons below come from Tables 2, 3, and 4. Zero-shot large models and domain-post-trained smaller models do not constitute an equal-training-data comparison.

Task	Model and setting	Mean IoU	Ctr. Dist.	Additional metric
Sorting, perfect input	LayoutGPT (Gemini-2.5)	0.781	0.471	Col. Free 0.982
Sorting, perfect input	Qwen3-8B CoT-SFT	0.635	0.513	Edit Dist. 1.610
Sorting, perfect input	Qwen3-8B Vanilla GRPO	0.850	0.274	Edit Dist. 0.684
Sorting, perfect input	Qwen3-8B Ours	0.924	0.145	Col. Free 1.000; Edit Dist. 0.510
Spatial alignment	Qwen3-VL-235B-Instruct, zero-shot	0.786	0.177	[email protected] 0.787
Spatial alignment	Qwen2.5-7B CoT-SFT	0.764	0.189	[email protected] 0.758
Spatial alignment	Qwen2.5-7B Ours	0.809	0.176	[email protected] 0.803
Room editing	Gemini 2.5 Pro, zero-shot	0.591	0.621	[email protected] 0.587
Room editing	Qwen3-8B CoT-SFT	0.972	0.026	[email protected] 0.972
Room editing	Qwen3-8B Ours	0.983	0.031	[email protected] 0.983

On sorting, the full method gains 0.074 mean IoU over Vanilla GRPO with the same backbone, equivalent to 7.4 percentage points, not a 7.4% relative improvement. On alignment, Qwen2.5-7B gains 0.045 over its CoT-SFT baseline. In room editing, IoU improves by 0.011 but center error increases from 0.026 to 0.031, so it would be incorrect to claim that every metric improves.

Ablation Study

The cache supports comparisons between training strategies and an analysis of input robustness. It does not contain a complete numerical table individually removing the IoU, collision, and format rewards. Accordingly, the ablation below is limited to training-stage comparisons, without inventing reward-component ablations.

Analysis setting	Model / strategy	Mean IoU	Col. Free
Sorting, perfect input	Qwen3-8B Ans-SFT	0.552	0.700
Sorting, perfect input	Qwen3-8B CoT-SFT	0.635	0.854
Sorting, perfect input	Qwen3-8B Vanilla GRPO	0.850	0.954
Sorting, perfect input	Qwen3-8B Ours	0.924	1.000
Sorting, caption-free	Qwen3-8B Ours, reference	0.209	0.984
Sorting, caption-free	Qwen2.5-VL-7B CoT-SFT	0.641	0.829
Sorting, caption-free	Qwen2.5-VL-7B Ours	0.787	0.979
Sorting, noisy input	Qwen2.5-VL-7B CoT-SFT	0.773	0.844
Sorting, noisy input	Qwen2.5-VL-7B Ours	0.822	0.980

The caption-free setting removes node captions. Noisy input adds 5% bounding-box jitter and imposes partial visibility. Values come from Tables 1 and 2; perfect-input and impaired-perception results should not be merged into a single undifferentiated ranking.

Key Findings

• Structured initialization and RL are complementary on sorting: the full method exceeds answer-only supervision, CoT-SFT, and Vanilla GRPO. These comparisons do not isolate the contribution of each reward component.
• Accurate scene graphs already supply substantial information. The language-only Qwen3-8B reaches 0.924 on perfect-input sorting but only 0.209 without captions, so its ideal-input performance should not be interpreted as evidence of strong visual perception.
• Qwen2.5-VL-7B reaches 0.787 in the caption-free setting, supporting the value of visual grounding when information is missing. Cross-backbone comparisons are not, however, a strictly controlled single-variable visual ablation.
• CoT-SFT already achieves 0.972 IoU on room editing. The authors attribute the smaller RL headroom to each example changing only a small part of a scene; this also suggests that whole-scene averages may dilute errors on the objects actually being edited.

Highlights & Insights

• Intermediate state serves both as reasoning material and inspectable output. Recording object IDs, group intervals, and coordinates exposes calculation errors or incorrect references more directly than increasing explanation length alone.
• Dataset evaluability and training objectives are designed together. Adding reference distances reduces ambiguity in room-editing supervision, while also making the task more constrained than open-ended interior design.
• Robustness experiments clarify the roles of language and vision. Numerical reasoning with a complete graph does not establish equivalent performance when semantics are missing or perception is noisy.

Limitations & Future Work

• The authors explicitly position the real-robot demonstrations as high-level goal-state generation. Qwen3-VL extracts boxes, while known grasping skills and rule-based goal-conditioned control execute the task. The model does not explicitly handle continuous dynamics or motion-path collisions, so these demonstrations do not establish end-to-end safe manipulation.
• Most evidence comes from synthetic layouts and distance-constrained room editing. Warehouse, multi-turn, and real tabletop examples are largely qualitative; the cache provides no large-scale success rates, confidence intervals, or long-horizon error-accumulation statistics for them.
• IoU and collision checks use axis-aligned boxes even though graphs contain rotation parameters. The cached text does not explain how orientation recovery is comprehensively evaluated. High IoU or Col. Free does not establish mesh-level non-interpenetration, stable support, or satisfaction of every language relation.
• The main text lacks training hyperparameters, complete splits, and individual reward ablations; one-to-one matching constraints are also unspecified. These details are needed to reproduce or interpret the scores, and the abstract's average gains should not be presented as uniform benefits across tasks.
• In the cached Figure 4, the chair instruction requests a position to the right, while the displayed trace says behind. Figure 5 specifies a 0.05-meter gap but its example calculation uses 0.03 meters. The text cache alone cannot distinguish source-figure errors from extraction problems, so neither figure is treated here as proof of constraint-by-constraint correctness.

Related Work & Insights

• Compared with SpatialRGPT / SpatialLLM / 3D-R1: These primarily target spatial understanding or question answering. This paper additionally predicts actionable scene-state changes, without thereby solving all underlying perception and control problems.
• Compared with LayoutGPT / LayoutVLM / Holodeck: Related layout approaches use language planning together with external constraints or optimization. This paper emphasizes direct, stepwise graph edits generated within the model. Removing an external layout solver does not automatically inherit its hard-constraint guarantees.
• Compared with InstructScene: The method reuses its instruction corpus and adds reference distances, turning the objective into constrained editing of an existing scene. Data provenance, task adaptation, and evaluation objectives should be distinguished rather than equating high IoU with general scene-generation capability.

Rating

• Novelty: 4/5. Combines explicit 3D graph-edit trajectories with geometric-reward post-training, with the main contribution in the joint design of the task, representation, and training interface.
• Experimental Thoroughness: 3/5. Covers three tasks, multiple backbones, and degraded inputs, but lacks independent reward ablations, sufficient reproduction details, and quantitative deployment evaluation.
• Writing Quality: 3/5. The main argument is clear and diagrams aid understanding, while inconsistent example constraints and corrupted cached equations increase the verification burden.
• Value: 4/5. Provides a concrete approach to language-guided 3D goal-state generation, with substantial boundaries remaining before reliable robot execution in open environments.