TabletopGen: Tabletop Scene Generation and Interactive Simulation for Robotic Manipulation¶

Conference: ECCV 2026
Paper: ECCV Official Page
Project: https://d-robotics-ai-lab.github.io/TabletopGen.project/
Area: 3D Vision
Keywords: 3D Scene Generation, Tabletop Environments, Interactive Simulation, Robotic Manipulation, Sim-to-Real Transfer

TL;DR¶

TabletopGen is a training-free, automated 3D tabletop scene generation and interactive simulation engine that decouples layout recovery into generative instance extraction and two-phase pose/scale alignment (DRO rotation optimization and TSA top-view spatial alignment), synthesizing physically plausible, collision-free tabletop environments that enable zero-shot real-to-sim-to-real robotic policy transfer.

Background & Motivation¶

The tabletop environment serves as the fundamental "last meter" of robotic manipulation, hosting fine-grained interactions such as grasping, precise placement, tool use, and assembly. With the swift emergence of Embodied AI and Vision-Language-Action (VLA) foundation models, policy learning increasingly depends on massive, diverse, and robust interaction demonstrations. However, collecting physical demonstrations on real-world tabletops is prohibitively expensive, slow to scale, and limited in coverage across diverse edge cases. Synthesizing interactive trajectories inside physics simulators offers a scalable, low-cost alternative for training generalizable robot policies.

Nevertheless, existing 3D scene generation approaches rarely produce simulation-ready tabletop environments directly. Retrieval-based frameworks depend on fixed CAD asset libraries, causing geometric mismatches and severe diversity bottlenecks. LLM-based room-scale generators target sparse room layouts and fail to handle the high-density functional arrangements of small objects and the strict physical constraints unique to tabletops. Meanwhile, single-image 3D reconstruction and end-to-end holistic generation methods struggle with single-view occlusions, frequently generating fragmented or topologically fused meshes, while lacking tabletop boundary and support priors. These shortcomings produce severe physical implausibilities such as floating objects, interpenetration, and objects falling off tables, which cause collision failures in physics engines and prevent automated trajectory planning.

To resolve the core conflict between incomplete single-view visual observations and dense physical contact constraints, this work avoids brittle monocular depth back-projections and decouples tabletop construction into independent instance extraction and visual-physical layout alignment. Core idea: TabletopGen decouples tabletop 3D scene generation into generative instance reconstruction and a two-phase pose/scale alignment pipeline, combining a Differentiable Rotation Optimizer (DRO) that recovers instance orientations from visual features with a Top-View Spatial Alignment (TSA) module that injects top-view geometric and stacking priors via an RMA-Score anchor, yielding collision-free, physics-ready environments for robotic manipulation data synthesis.

Method¶

Overall Architecture¶

TabletopGen takes either a text prompt or a single reference image as input. When given text, an LLM expands it into a detailed scene prompt specifying semantic objects, functional layouts, and tabletop materials, which a text-to-image model synthesizes into a reference image $I_{ref}$; an image input directly acts as $I_{ref}$. The overall pipeline progresses through three decoupled stages: 1. Generative Instance Extraction & Canonicalization: Segment instances from the reference image, complete occluded boundaries via multimodal generative inpainting, reconstruct individual high-quality 3D meshes, and align their local vertical axes to world Z-up; 2. Two-Phase Pose and Scale Alignment: Decouple 6-DoF layout estimation into rotation and translation-scale optimization. A Differentiable Rotation Optimizer (DRO) estimates the yaw angle around the vertical axis using tri-modal visual loss, followed by a Top-View Spatial Alignment (TSA) mechanism that uses a synthesized orthographic top-view image, stacking priors, and an RMA-Score anchor to infer globally consistent metric scales and 3D translations; 3. Interactive Simulation & Multimodal Data Engine: Import canonical meshes and transforms into Isaac Sim, compute convex decomposition collision hulls, configure physics parameters (friction, gravity), and use automated motion planning to execute collision-free manipulation trajectories, producing rich multimodal demonstrations.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Text Prompt / Single Image"] --> B["Stage 1: Generative Instance Extraction<br/>MLLM Detection + Generative Completion + Image-to-3D + Z-up Alignment"]
    B --> C["Stage 2.1: Differentiable Rotation Optimizer (DRO)<br/>Differentiable Rendering + Soft Silhouette / Edge / DINOv2 Tri-Modal Loss"]
    C --> D["Stage 2.2: Top-View Spatial Alignment (TSA)<br/>Synthesized Top-View + Stacking Prior + RMA-Score Anchor Scaling"]
    D --> E["Stage 3: Interactive Simulation & Data Synthesis<br/>Convex Collision Hulls + Physics Parameters + Policy Trajectories & Sim-to-Real"]

Key Designs¶

1. Generative Instance Extraction: Eliminating Occlusions and Standardizing Coordinates Under monocular observation, tabletop objects routinely suffer from mutual occlusion and perspective truncation. Standard inpainting often introduces blurred textures and corrupted geometries when occlusions are severe. TabletopGen uses an MLLM for open-vocabulary detection combined with GroundedSAM-v2 to extract initial instance masks. For occluded items, a multimodal generative model redraws and completes the masked regions, guaranteeing that each instance input to the 3D generator has complete structural boundaries. High-fidelity 3D meshes $m'_i$ are then generated via an Image-to-3D model (Hunyuan3D-3.0). Because isolated generative models yield arbitrary local coordinate axes, the framework applies visual and semantic alignment to rotate the local support base of each mesh to the world Z-up direction, producing standardized assets $m_i$ ready for placement.

2. Differentiable Rotation Optimizer (DRO): Tri-Modal Gradient Optimization for Precise Yaw In tabletop manipulation, an object's rotation $r_i$ around the vertical axis does not alter horizontal scene bounds but dramatically changes its 2D projected silhouette, visible texture patterns, and feasible grasp poses. Instead of relying on noisy depth back-projection, DRO optimizes $r_i$ via differentiable rendering. Given an initial camera viewpoint estimated by the MLLM, PyTorch3D renders each mesh $m_i$ to produce textured output $I_{render}(r_i)$, a soft silhouette $\hat{S}(r_i)$, and a soft edge map $\hat{E}(r_i)$ computed via Sobel filtering. The rotation is optimized against target image $I_{instance}$ and mask $S$ through a tri-modal objective: $$\mathcal{L}_{rot}(r_i) = \lambda_s \mathcal{L}_{sil}(r_i) + \lambda_e \mathcal{L}_{edge}(r_i) + \lambda_a \mathcal{L}_{app}(r_i)$$ where soft IoU loss $\mathcal{L}_{sil}(r_i) = 1 - \frac{\sum (\hat{S}(r_i) \cdot S)}{\sum (S + \hat{S}(r_i) - \hat{S}(r_i) \cdot S)}$ penalizes silhouette mismatch; edge loss $\mathcal{L}_{edge}(r_i)$ measures one-sided Chamfer distance to pre-computed Canny edge distance field $D_S$; and appearance loss $\mathcal{L}_{app}(r_i)$ computes the $L_2$ feature discrepancy under pre-trained DINOv2 embeddings $\Phi$. Optimization begins with a coarse grid search across $[0^\circ, 360^\circ)$ at $5^\circ$ increments, selects the 8 best candidates, and refines them via Adam over 140 iterations to find optimal $\hat{r}_i$.

3. Top-View Spatial Alignment (TSA): Orthographic Geometric Priors and RMA-Score Metric Anchoring Monocular perspective views inherently suffer from depth-scale ambiguity, where tiny depth errors lead to massive metric displacement. TSA overcomes this by synthesizing an orthographic top-view image $I_{top}$ of the tabletop, from which 2D bounding boxes are extracted within table boundaries, while an MLLM determines vertical stacking and support relations to eliminate floating objects and vertical penetration. To convert pixel coordinates to physical metric dimensions, the framework queries commonsense object dimensions, combines them with the estimated yaw $r_i$, and computes the physical aspect ratio $r_{phys}$ on the horizontal plane. To prevent noisy small objects from corrupting the global scale factor, TSA introduces the Ratio-Matched and Area-Weighted Score (RMA-Score): $$\varepsilon_{ratio} = |\log r_{phys} - \log r_{img}|, \quad \text{RMA-Score}(i) = \frac{A_{px}(i)}{1 + (\varepsilon_{ratio}/\tau)^2}$$ where $A_{px}(i)$ is the top-view bounding box pixel area and $\tau = 0.25$ is the error tolerance. The object with the highest RMA-Score (typically a large, dimensionally reliable object such as a cutting board, tray, or laptop) serves as the scaling anchor to establish the global metric conversion ratio $\alpha$ (meters/pixel). This determines horizontal translations $(t_{i,x}, t_{i,y})$ and 3D scales $s_i$, while vertical translation $t_{i,z}$ is fixed by stacking support planes, yielding a collision-free 3D layout.

A Worked Example¶

Consider the prompt "A hobby desk with some model cars and tools": 1. Asset Generation: The LLM expands the prompt into an organized craft bench layout; a reference image is synthesized with cars, a magnifying glass, and a toolbox. GroundedSAM-v2 segments the objects, and generative inpainting reconstructs the partially hidden chassis of the car. Hunyuan3D-3.0 produces separate meshes, which are reoriented to Z-up. 2. Pose and Scale Optimization: DRO evaluates DINOv2 and soft-IoU losses under PyTorch3D, optimizing the orientation of the car and magnifying glass to match the reference view within 140 Adam steps. A top-view image $I_{top}$ is synthesized. RMA-Score evaluates candidate anchors: the toolbox exhibits large pixel area and near-zero aspect ratio error ($\varepsilon_{ratio} = 0.03$), securing the highest score and fixing metric ratio $\alpha$. All tools receive metric scales from $\alpha$, and the magnifying glass is assigned a non-interpenetrating vertical height on top of the toolbox according to the MLLM stacking graph. 3. Simulation and Synthesis: Isaac Sim computes convex decomposition collision meshes for all assets. The LLM generates the task "Put the magnifying glass into the box", and an algorithmic motion planner generates collision-free pick-and-place trajectories, logging synchronized RGB frames, joint angles, and gripper states.

Key Experimental Results¶

Main Results¶

The authors evaluated TabletopGen against leading single-image 3D scene reconstruction methods on a benchmark of 78 diverse tabletop configurations (spanning square, circular, and triangular tables across office, dining, workbench, and crafting domains). Metrics evaluate visual and perceptual similarity (LPIPS, DINOv2, CLIP), multi-dimensional GPT-4o ratings (Visual Fidelity VF, Image Alignment IA, Physical Plausibility PP, Overall Ranking OR), and physical collision rates (object-pair collision Col_O and scene collision Col_S).

Method	LPIPS ↓	DINOv2 ↑	CLIP ↑	GPT VF ↑	GPT IA ↑	GPT PP ↑	GPT Avg. ↑	GPT OR ↓	Col_O (%) ↓	Col_S (%) ↓
ACDC (Retrieval)	0.5124	0.3775	0.6696	2.38	1.90	2.57	2.28	3.55	8.23	67.95
Gen3DSR (Single-view)	0.4891	0.5602	0.8636	2.92	4.17	3.32	3.47	3.02	16.88	85.90
MIDI (Diffusion)	0.4559	0.7070	0.8867	4.22	4.48	4.30	4.33	2.32	17.39	98.72
TabletopGen (Ours)	0.4483	0.8383	0.9077	6.06	6.30	6.22	6.19	1.08	0.42	7.69

In text-to-scene benchmarks across 30 prompt configurations, TabletopGen outperformed MesaTask and Holodeck-table by substantial margins (GPT average score 6.27 vs 4.12 and 4.08; Col_S 6.67% vs 63.33%). In a 128-participant double-blind User Study, TabletopGen achieved an 83.13% overall human preference rate, vastly outperforming MIDI (11.61%), Gen3DSR (2.88%), and ACDC (2.38%).

Ablation Study¶

The authors conducted component ablations on the pose and scale alignment pipeline, replacing DRO and TSA with naive direct MLLM parameter estimation from the reference image:

Config	LPIPS ↓	DINOv2 ↑	CLIP ↑	Col_O (%) ↓	Col_S (%) ↓	Note
Ours (Full Model)	0.4483	0.8383	0.9077	0.42	7.69	Full model: optimal perceptual alignment and near-zero collision
w/o DRO	0.4523	0.8261	0.9012	1.27	16.67	Yaw misalignment causes subtle rotations and collision increase
w/o TSA	0.4799	0.8041	0.8954	5.50	61.54	Lacks top-view priors; scene collision rate surges to 61.54%
w/o both	0.4811	0.7897	0.8922	5.41	62.82	Severe layout distortion, scale drift, and heavy interpenetration

Key Findings¶

TSA top-view geometry is the primary barrier against physical collisions: Removing TSA causes scene collisions Col_S to spike from 7.69% to 61.54% (an eight-fold increase). Orthographic top-view projection enforces clear 2D support boundaries that monocular depth estimation fundamentally cannot guarantee.
DRO is essential for fine-grained semantic visual fidelity: While omitting DRO only modestly increases collisions (16.67%), object handles, orientations, and contours degrade substantially, directly impairing DINOv2 perceptual scores and downstream robot grasp synthesis.
Zero-shot Sim-to-Real policy transfer confirms simulation fidelity: Starting from a single photograph of a real tabletop containing fruits and containers, the authors reconstructed the digital twin in simulation, synthesized 2,000 domain-randomized trajectories per task, and fine-tuned a $\pi_{0.5}$ VLA policy. Directly evaluated on a physical AgileX PiPER arm across 50 trials per task without any real data fine-tuning, the policy achieved 84% real success on mango-to-plate (98% sim), 80% on strawberry-to-mug (92% sim), and 68% on block stacking (90% sim), empirically validating the real-world utility of TabletopGen's synthetic data.

Highlights & Insights¶

Decoupled Orthogonal Optimization for 3D Layouts: Decomposing 6-DoF estimation into orientation (appearance-driven, handled via differentiable rendering) and translation-scale (contact-driven, handled via orthographic top-view geometry) elegantly bypasses high-dimensional local minima and collision traps.
RMA-Score Adaptive Metric Grounding: Rather than relying on a randomly selected reference object, RMA-Score dynamically weights aspect-ratio consistency against bounding box area, selecting the most dimensionally reliable asset as the metric anchor and stabilizing the global coordinate frame.
Modular In-scene Editing for Diverse Data Expansion: Because assets exist as independent 3D meshes with decomposed collision geometry, the framework enables zero-recomputation scene expansion through Add, Remove, Replace, and Rearrange operations, drastically lowering the cost of generating high-diversity robotics data.

Limitations & Future Work¶

Multi-Level and Complex Storage Geometries: The current top-view projection prior is optimized for single-tier planar surfaces. Multi-tier shelving units, closed drawers, or vertical cabinets introduce severe vertical occlusion, where orthographic top views lose visibility of internal shelves.
Articulated and Deformable Object Synthesis: Assets are represented as rigid meshes with convex collision hulls. Articulated objects (e.g., closing laptops, opening drawers) and soft deformable bodies (e.g., cloth, cables) are not yet automatically parameterized for dynamic interaction.
Friction and Dynamic Parameter Estimation: Physical parameters such as friction coefficients and mass distribution are assigned via semantic category defaults. Delicate contact tasks like precision insertion or friction-critical stacking still exhibit a noticeable sim-to-real gap that could benefit from tactile or video-based system identification.

vs. Retrieval-based Generation (ACDC / MesaTask): Retrieval methods are constrained by finite CAD databases, leading to style discrepancies, missing long-tail objects, and flat planar layouts. TabletopGen leverages generative instance extraction and completes entire stylized tables alongside detailed objects.
vs. Single-view End-to-End Reconstruction (Gen3DSR / MIDI): Holistic single-view methods generate fused meshes with collision rates exceeding 85% due to monocular depth ambiguity. TabletopGen enforces explicit physical stacking and orthographic top-view priors, slashing object collision rates to 0.42% and unlocking executable motion planning.

Rating¶

Novelty: ⭐⭐⭐⭐☆ [Decoupling pose recovery into DRO differentiable rendering and TSA orthographic projection effectively resolves the chronic collision bottleneck in single-view 3D scene generation.]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive 78-scene benchmark, automated GPT-4o evaluation, component ablations, 128-user study, and real-world zero-shot AgileX PiPER arm transfer experiments.]
Writing Quality: ⭐⭐⭐⭐⭐ [Rigorous, cohesive narrative with clear formulation of tri-modal rotation loss and RMA-Score metric anchoring.]
Value: ⭐⭐⭐⭐⭐ [Provides a practical, training-free, and extensible synthetic data engine for Embodied AI and VLA foundation model training.]