[ECCV 2026][3D Vision][3D compositional generation] Interact3D proposes a training-free “generate-then-compose” framework. It leverages the spatial priors of 2D/3D generative models to formulate 3D compositional generation as a structured registration problem. Through a two-stage composition pipeline—global-to-local geometric alignment to anchor the main object, followed by SDF collision-aware optimization to place the remaining objects—and a VLM-driven agent closed-loop for iterative correction, it automatically generates physically plausible and collision-free interactive 3D compositional scenes from a single image and text prompt, while releasing a dataset of 8,000+ interactive 3D pairs.

Tags: ECCV 2026 · 3D Vision · 3D compositional generation · physics-aware composition · collision avoidance · SDF optimization · VLM agent optimization

Interact3D: Compositional 3D Generation of Interactive Objects¶

Conference: ECCV 2026
arXiv: 2603.16085
Code: https://github.com/SII-Hui/Interact3D
Area: 3D Vision
Keywords: 3D compositional generation, physics-aware composition, collision avoidance, SDF optimization, VLM agent optimization

TL;DR¶

Interact3D proposes a training-free “generate-then-compose” framework. It leverages the spatial priors of 2D/3D generative models to formulate 3D compositional generation as a structured registration problem. Through a two-stage composition pipeline—global-to-local geometric alignment to anchor the main object, followed by SDF collision-aware optimization to place the remaining objects—and a VLM-driven agent closed-loop for iterative correction, it automatically generates physically plausible and collision-free interactive 3D compositional scenes from a single image and text prompt, while releasing a dataset of 8,000+ interactive 3D pairs.

Background & Motivation¶

Robotic simulation (Sim2Real) requires a large number of interactive 3D assets with realistic geometry, physical properties, and correct object-object spatial relations (OOR) to train manipulation policies. However, high-quality interactive 3D data are severely scarce: manual modeling and annotation are extremely costly and difficult to scale. Existing alternatives each suffer from critical limitations. Methods based on large-scale human videos provide rich semantic priors, but 2D videos naturally lack precise 3D geometric grounding and physical collision constraints, preventing robots from understanding complex spatial relations. 3D generative models such as TRELLIS2 and Hunyuan3D can synthesize high-quality single-object assets, but their outputs are fused meshes “baked” into a single geometry, with no independent object boundaries or physically meaningful spatial relations.

An intuitive solution is to directly use segmentation tools such as PartField to decompose generated scenes, but this creates holes in the geometry that are difficult to repair. Feed-forward network methods such as MIDI and 2BY2 are constrained by the extreme scarcity of 3D compositional data (2BY2 has only 517 pairs), leading to poor generalization and high training cost. The key challenge is therefore: how can geometrically compatible and physically plausible 3D compositional assets be reliably generated without expensive 3D annotations?

The core insight of this paper is that image-guided 3D generative models implicitly encode rich spatial priors: although the generated scenes have imperfect geometry, often with holes, their relative inter-object poses are reliable. Thus, the traditional problem of complex geometric reasoning can be reformulated as a structured 3D registration problem. The rough segmentation results of the generated scene serve as spatial guidance, high-quality independently generated single-object meshes are registered to the correct positions, and physical plausibility is ensured through collision optimization and VLM-based semantic correction. The core idea is: generate-then-compose—generate high-quality single objects separately, then “assemble” them into physically plausible interactive scenes using spatial relations provided by generative priors.

Method¶

Overall Architecture¶

The core mechanism of Interact3D is “generate-then-compose”: it does not attempt to directly generate a single fused mesh for a compositional scene, but instead generates high-quality single-object meshes separately and then registers them to the correct relative poses using the implicit spatial priors of generative models. The full pipeline starts from a user-provided 3D mesh M and text prompt, proceeds through image generation, 3D reconstruction, spatial guidance extraction, two-stage composition, and optional agent-based correction, and finally outputs a physically plausible interactive 3D scene.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["User input<br/>Mesh M + text prompt"] --> B["Data preparation<br/>Render M + Nano Banana Pro<br/>Generate I_scene and I_comp"]
    B --> C["TRELLIS2 reconstruction<br/>M_scene and M_comp"]
    C --> D["PartField segments M_scene<br/>Extract spatial guidance M' and M'_comp"]
    D --> E["Global-to-local geometric alignment<br/>OBB scale + GeoTransformer<br/>+ scale-aware ICP"]
    E --> F["SDF collision-aware composition<br/>Hard penalty + soft repulsion<br/>+ progressive beta scheduling"]
    F --> G{"Severe residual collision?"}
    G -->|Yes| H["VLM agent iterative correction<br/>Multi-view rendering→VLM analysis<br/>→edit prompt→reconstruct mesh"]
    H --> F
    G -->|No| I["Output: physically plausible<br/>interactive 3D scene"]

The detailed procedure is as follows. (1) Data preparation: the input mesh M is rendered into an image I_rendered from a canonical frontal view. Together with the user text prompt, Nano Banana Pro (Gemini) generates the compositional scene image I_scene, and the model is further prompted to remove the original object from I_scene to obtain a complementary image I_comp containing only the added component. (2) 3D reconstruction: TRELLIS2 reconstructs the scene mesh M_scene and the complementary mesh M_comp from I_scene and I_comp, respectively. (3) Spatial guidance extraction: PartField segments M_scene into M' and M'_comp. Although these segmented meshes often contain holes and cannot serve as final assets, their relative poses provide reliable spatial guidance. (4) Two-stage composition: Stage 1 performs global-to-local geometric alignment on the anchor object to determine a reliable initial pose; Stage 2 performs SDF collision-aware optimization on the remaining object, explicitly penalizing interpenetration while maintaining geometric alignment. (5) Agent correction: when collisions cannot be resolved by pure pose optimization, the VLM analyzes multi-view renderings and generates correction prompts, which drive the 2D image editing module to iteratively correct the geometry of the complementary component.

Key Designs¶

1. Global-to-local geometric alignment: robust registration under low overlap and segmentation holes

Registering the anchor object faces three challenges: occlusions in I_scene cause inconsistency between the generated mesh and the scene mesh; PartField segmentation introduces structural holes and incomplete surfaces; as a result, the two point clouds to be registered have very low overlap and many outliers. Conventional ICP is extremely sensitive to these issues and easily falls into local optima.

The paper adopts a hierarchical strategy in which global estimation provides a warm start and local optimization refines convergence. First, Oriented Bounding Box (OBB) estimates the geometric extents of the two point clouds and computes an initial uniform scaling factor s to remove scale ambiguity. Then GeoTransformer, a transformer-based 3D registration network, robustly estimates the global translation tau and rotation R under low-overlap geometry. Finally, the global estimate (s, R, tau) is used as initialization, and scale-aware ICP is run for precise alignment. These three progressive steps each play a distinct role: OBB resolves scale, GeoTransformer resolves global pose, and ICP resolves local residuals. The global initialization prevents convergence to local minima, while the local optimization focuses on minimizing residual geometric deviations.

2. SDF collision-aware composition: explicitly modeling physical plausibility with a signed distance field

After fixing the anchor object, the remaining object M_remain must be placed relative to the anchor object. Directly using a similar global-to-local alignment can provide a reasonable initial pose, but it cannot guarantee physical validity: small geometric deviations may still lead to object interpenetration or unnatural gaps.

The paper introduces an SDF-guided optimization stage. It first precomputes the signed distance field Phi_anchor(p) of the anchor mesh M_anchor, where a negative value indicates that point p penetrates the interior of the anchor object. For any point p on M_remain, the collision loss after transformation theta is defined as:

\[\mathcal{L}_{\mathrm{col}}(\boldsymbol{\theta})=\sum_{\mathbf{p}\in\mathbf{M}_{\mathrm{remain}}}\left(\left[-\Phi_{\mathrm{anchor}}(\boldsymbol{\theta}(\mathbf{p}))\right]_{+}^{2}+\lambda\cdot\left[\epsilon-\Phi_{\mathrm{anchor}}(\boldsymbol{\theta}(\mathbf{p}))\right]_{+}\right)\]

The first term is a hard penalty (squared ReLU, activated only when a point penetrates the object interior), and the second term is soft repulsion (generating a repulsive force within the safety margin epsilon; lambda is set to a very small value to ensure a smooth transition of the penalty field). The final optimization objective combines a geometric alignment fidelity term with the collision loss:

\[\min_{\boldsymbol{\theta}}\;\sum_{p\in\mathbf{M}_{\mathrm{remain}}}\|\boldsymbol{\theta}(\mathbf{p})-\mathbf{p}^{\prime}\|^{2}+\beta^{(k)}\mathcal{L}_{\mathrm{col}}(\boldsymbol{\theta})\]

The key design lies in the progressive scheduling of beta: beta is initialized to 0, which is equivalent to pure scale-aware ICP, and then linearly increases to beta_max. This allows early optimization to prioritize geometric alignment convergence and later optimization to gradually enforce physical plausibility. This “align first, avoid collisions later” scheduling strategy prevents the collision term from dominating early optimization and pushing objects away before their poses have converged.

3. VLM agent iterative correction: semantic closed-loop correction for geometric incompatibility that pose optimization cannot handle

SDF optimization can alleviate collisions only through translation, rotation, and scaling. When the complementary mesh itself is fundamentally incompatible with the anchor object, such as an incorrect flower stem direction inside a vase or a plush bear that cannot fit into a canvas bag, pure pose optimization is ineffective. The root cause is that severe occlusion in the 2D guidance image causes TRELLIS2 reconstruction to lose key spatial relations.

The paper introduces an agent correction loop operating at the semantic level. The current compositional scene is rendered from multiple views, including internal cross-section views, and sent to the VLM (Gemini 3 Pro) together with the original compositional prompt. The VLM analyzes spatial inconsistencies and generates targeted correction instructions, such as “shorten the flower stem and tilt it 15 degrees to the left.” These instructions drive Nano Banana Pro to perform 2D editing on the complementary image I_comp. The edited image is reconstructed by TRELLIS2 into a 3D mesh and then re-enters the composition pipeline. The entire process forms a closed loop: geometric alignment, SDF optimization, and VLM semantic correction alternate until there is no collision or the maximum number of iterations, set to 5, is reached. This effectively adds a “semantic outer loop” to pure geometric optimization: geometric methods handle problems solvable by pose changes, while semantic methods handle problems that require modifying object geometry.

A complete example: flowers in a vase¶

The full process can be illustrated with the example “insert flowers into a vase.” The user provides a vase mesh M and the prompt “a vase with flowers.” First, the frontal view of the vase is rendered. Nano Banana Pro generates the scene image I_scene of the vase with flowers and the complementary image I_comp containing only the flowers. TRELLIS2 reconstructs the scene mesh, which contains the fused geometry of the vase and flowers, and the flower mesh M_comp. PartField segments the scene mesh into the vase segment M' and the flower segment M'_comp. The latter contains holes due to segmentation artifacts, but it provides positional reference for the flowers relative to the vase.

Stage 1: the vase has a larger projected area in I_scene and is selected as the anchor object. OBB estimates the scale difference → GeoTransformer estimates the global pose → scale-aware ICP performs precise alignment, and the vase is fixed at the correct position.

Stage 2: the flowers serve as M_remain. OBB + GeoTransformer provides the initial pose, and SDF optimization gradually pushes away penetrations under the progressive increase of beta. However, a severe problem occurs here: the flower stems are heavily occluded by the vase in I_scene, causing the flower mesh reconstructed by TRELLIS2 to have stem directions fundamentally misaligned with the vase opening, which pure pose optimization cannot resolve.

Agent correction is triggered: multi-view renderings of the compositional scene, including a cross-section view of the vase, are produced. The VLM identifies that “the flower stem deviates from the central axis of the vase and is too long, causing it to protrude through the bottom of the vase,” and generates the correction prompt “shorten the stem to 2/3 of the vase depth and vertically align it with the center of the vase opening.” Nano Banana Pro edits I_comp accordingly, producing a new I_comp in which the stem is shorter and centered. After TRELLIS2 reconstruction and SDF composition optimization are rerun, collisions are eliminated, and convergence is achieved in the second round, yielding a physically plausible vase-with-flowers scene.

Loss & Training¶

The method is entirely training-free. All components—TRELLIS2, PartField, GeoTransformer, and Gemini—use pretrained weights. The “optimization” in the method refers to numerical optimization at inference time rather than network training.

The core optimization objective is the joint loss in Equation (3): the data term is the L2 distance between transformed points and their corresponding target points, ensuring geometric alignment; the collision term L_col is the combination of the SDF hard penalty and soft repulsion in Equation (2). Hyperparameters are set as follows: lambda = 0.003 (soft repulsion weight, a very small value used only to ensure a smooth transition at collision boundaries), beta linearly increases from 0 to beta_max = 3.0, and optimization runs for k_max = 100 steps. GeoTransformer uses official pretrained weights trained on 3DMatch and KITTI. The maximum number of VLM agent correction iterations is set to 5. Inference is performed on a single NVIDIA H200 GPU.

Key Experimental Results¶

Main Results¶

The method is compared with 5 baselines, grouped into three categories, on 150 test cases, and evaluated along two dimensions: semantic fidelity and physical validity.

Method	Text CLIP (Avg) ↑	Image CLIP (Avg) ↑	Human Rating (Avg) ↑	VLM Rating (Avg) ↑	R_surface (×10^-3) ↓	R_volume (×10^-3) ↓
Jigsaw	0.3004	0.7338	4.21	5.89	2.7792	16.837
2BY2	0.2798	0.6982	2.87	4.08	1.5827	9.1734
MIDI	0.3079	0.7840	5.27	5.84	22.185	116.70
PartField+RANSAC	0.3094	0.7910	6.28	6.59	2.2287	7.9280
Interact3D (Ours)	0.3314	0.8236	8.33	8.57	0.6939	3.8762

The feed-forward baselines (Jigsaw, 2BY2, MIDI) are limited by the distribution of their training data and generalize poorly. MIDI performs particularly poorly on collision metrics, with R_volume as high as 116.70. PartField+RANSAC improves the results to some extent, but RANSAC lacks global registration robustness, sometimes producing orientation flips, and does not explicitly model collisions, with R_surface being 3.2 times that of Ours. Interact3D leads across all metrics, especially the physical validity metrics: R_surface is only 0.6939×10^-3 and R_volume is 3.8762×10^-3, far lower than all baselines.

Ablation Study¶

The components of the registration pipeline are ablated on 150 test cases.

Config	Text CLIP ↑	Image CLIP ↑	Human Rating ↑	VLM Rating ↑	R_surface (×10^-3) ↓	R_volume (×10^-3) ↓
RANSAC only	0.3094	0.7910	6.28	6.59	2.2287	7.9280
ICP only	0.2985	0.7729	5.04	5.88	1.2914	9.7102
RANSAC+ICP	0.3023	0.7958	6.40	6.31	1.0481	6.7536
GeoT. only	0.3187	0.8094	7.03	7.21	0.8854	6.0039
GeoT.+ICP(SDF) w/o Agent	0.3277	0.8101	7.94	8.18	0.7458	4.9380
Full (GeoT.+ICP(SDF)+Agent)	0.3314	0.8236	8.33	8.57	0.6939	3.8762

Key findings: (1) ICP alone is even worse than RANSAC alone (Human Rating 5.04 vs 6.28), because ICP is extremely sensitive to initialization and frequently falls into local optima in low-overlap scenarios. (2) The global-to-local combination of RANSAC+ICP already improves performance, but there remains a significant gap to the GeoTransformer variants. GeoTransformer’s transformer architecture is much more robust than RANSAC when handling low-overlap geometry. (3) Removing the SDF term, comparing GeoT.+ICP(SDF) w/o Agent with RANSAC only, increases R_surface from 0.7458 to 2.2287 and R_volume from 4.9380 to 7.9280, indicating that the ablated component is precisely the collision-awareness capability. (4) Removing the agent correction loop consistently degrades all metrics, reducing Human Rating from 8.33 to 7.94, validating the effective contribution of semantic-level correction to final quality.

Key Findings¶

GeoTransformer is the core pillar of registration robustness: under low-overlap conditions caused by PartField segmentation holes and inconsistencies in generated meshes, GeoTransformer’s global registration capability substantially surpasses RANSAC and pure ICP, serving as the fundamental safeguard that prevents the pipeline from collapsing.
The SDF collision term contributes the most, but the damage is reversible: after ablating the SDF term, collision metrics degrade significantly, with R_surface increasing by 3.2 times. However, semantic metrics such as CLIP and ratings decrease only mildly, indicating that SDF mainly addresses physical validity and has limited impact on semantic fidelity.
The agent correction loop has a narrow scope but strong impact: it is actually triggered only in about 10% (16/150) of severe-collision cases, but once triggered, it can fundamentally resolve geometric incompatibilities that pure geometric methods cannot handle, such as book inversion caused by symmetry ambiguity.
The method is insensitive to hyperparameters: results remain stable when lambda, the soft repulsion weight, beta_max, and k_max vary within reasonable ranges, indicating that the progressive scheduling strategy itself is more critical than the specific parameter values.

Highlights & Insights¶

A shift in using generated scenes as spatial guidance rather than final output: the conventional approach is to generate a compositional scene and directly segment it for use. This paper takes the opposite route: it explicitly acknowledges that the segmentation results “have holes and cannot be used,” extracts only their spatial relations as registration guidance, and obtains the final assets from independently generated high-quality single objects. This division of labor between “rough guidance + high-quality single objects” can transfer to any scenario that requires compositional priors but has insufficient source data quality.
VLM as a “semantic outer loop” for geometric optimization: conventional geometric optimization handles pose, while the VLM handles object geometry. The two levels have clear and complementary roles. The VLM does not inspect coordinates but instead reads rendered images, naturally bridging the gap between 2D semantics and 3D geometry. This closed-loop pattern of “geometric optimization + VLM semantic feedback” can transfer to other 3D editing or generation tasks that require changing geometry rather than pose.
A small trick with progressive beta scheduling, but with large effect: beta linearly increases from 0 to beta_max, which is equivalent to “align fully first, then gradually push away collisions after alignment converges.” If beta is large from the beginning, the collision term pushes objects away before pose convergence and causes alignment failure. This scheduling strategy of “ensuring the main objective converges before adding constraints” is broadly applicable when multiple optimization objectives conflict.

Limitations & Future Work¶

The authors acknowledge difficulty in handling fine-grained parts such as screws and tightly coupled joints. Severe occlusion in 2D guidance images induces 3D geometric ambiguity, exposing the ceiling of relying on 2D spatial priors. A future direction is native 3D compositional generation, which reasons directly in 3D space and avoids dependence on 2D.
Objects with strong geometric symmetry cause orientation ambiguity, such as a rectangular book being flipped upside down, because GeoTransformer does not consider texture. Pure geometric matching cannot distinguish orientation for symmetric objects. A direct improvement is to incorporate texture or appearance features to assist registration.
Agent correction depends on the VLM’s ability to understand 3D space from multi-view renderings. Current VLM 3D spatial reasoning remains immature and may provide erroneous correction suggestions under complex occlusion. The experiments test only one VLM, Gemini 3 Pro, and robustness across different VLMs is not verified.
The choice of anchor object in the two-stage composition is based only on 2D projected area. This may be unreasonable when object depths differ significantly, such as a large distant object versus a small nearby object. Selection strategies based on 3D volume or SDF coverage could be considered.
Although the dataset contains 8,000+ pairs and far exceeds prior work, with 2BY2 having only 517 pairs, it covers only 9 categories of daily objects. Generalization to specialized domains such as industrial parts and medical instruments remains to be validated.

vs PartField / SAMPart3D: these methods perform 3D part segmentation, taking a fused mesh as input and outputting segmented parts. This paper shows that directly segmenting high-fidelity generated scenes leads to irreparable geometric holes, so the segmentation results are used only as spatial guidance rather than final outputs. Insight: when segmentation quality is insufficient, “guidance” is more practical than “direct use.”
vs 2BY2 / MIDI / Jigsaw: feed-forward networks predict 3D compositions. Their advantage is fast inference, while their critical limitation is poor generalization due to scarce 3D compositional training data. The training-free design of this paper completely bypasses the data bottleneck, at the cost of slower inference due to multiple rounds of generation and optimization. Insight: in data-scarce domains, using large-model priors for zero-shot generation is often more feasible than collecting data to train small models.
vs COPY-TRANSFORM-PASTE: this method also uses a VLM for 3D composition, but supervises pose optimization with CLIP similarity and lacks explicit geometric registration and collision modeling. The GeoTransformer + SDF design in this paper provides hard geometric constraints and better physical validity.
vs TRELLIS2: TRELLIS2 is the 3D reconstruction backbone used in this paper. The contribution of this paper is extending its capability from “single-object generation” to “compositional scene generation.” The core is to use registration and collision optimization as post-processing layers without modifying the generative model itself. This idea of “leaving the generative model unchanged and solving composition at the post-processing layer” can generalize to other generative backbones.

Rating¶

Novelty: ⭐⭐⭐⭐ It reformulates 3D compositional generation as registration plus collision optimization, and the VLM semantic correction loop as a closed outer loop for geometric optimization is a distinctive design. The core “generate-then-compose” idea is intuitive but had not been systematically explored before.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ 150 cases × 5 baselines × 3 baseline categories + 6 metrics, including CLIP, human rating, VLM rating, and surface/volume collision rates + a 6-row ablation table + 140 additional qualitative results + multi-part extension experiments, yielding comprehensive evaluation dimensions.
Writing Quality: ⭐⭐⭐⭐ The structure is clear, Figure 3 provides an accurate overview, and Figure 4 clearly locates the correction process. Method and experimental details are complete, and the limitations section honestly discusses failure cases.
Value: ⭐⭐⭐⭐ It addresses the real asset bottleneck in Sim2Real robot training, and the released dataset of 8,000+ pairs can directly benefit the community. The training-free design allows it to automatically benefit as the underlying generative model evolves, e.g., TRELLIS2→3→...