Reconstruction by Generation: 3D Multi-Object Scene Reconstruction from Sparse Observations

Conference: ECCV2026
Paper: ECCV Paper
Area: 3D Vision
Keywords: sparse-observation reconstruction, joint shape and pose generation, occlusion completion, pose-conditioned textures, object parts

TL;DR

RecGen jointly generates complete shapes and camera-frame poses from one or two RGB-D views and instance masks, then conditions texture generation on pose, raising strict pose accuracy on ArtVIP parts from SAM3D's 45.8% to 84.0%, although two-view reconstruction and appearance metrics do not uniformly outperform the alternatives.

Background & Motivation

Turning a real tabletop or mechanical assembly into a digital twin for robotics requires more than generating a visually similar mesh. Objects must have appropriate complete dimensions and occupy the positions and orientations implied by the observations; hidden backs and drawer side walls also need completion. Pipelines such as Any6D and SceneComplete first generate objects, then register them using depth, feature matching, or FoundationPose. If generation mistakes a visible fragment for the entire object, registration receives a candidate with incorrect dimensions or topology, which cannot simply be repaired by adjusting a rigid pose.

Shape and pose uncertainty are coupled: a small visible surface may belong either to a small object or to a much larger, heavily occluded one. Even when a cylindrical shape is aligned, its label may face the wrong direction. SAM3D already jointly predicts geometry and layout, but this paper targets richer practical inputs, including noisy sensor depth, part-level masks, and a second camera view with complementary information. Training only on unoccluded objects or removing the RGB background deprives the model of useful evidence about occlusion.

RecGen therefore conditions generation on local scene context, depth geometry, and target masks, learns complete object and part shapes from synthetic occluded scenes, and explicitly informs appearance generation of the predicted orientation. Core Idea: treat which shape to generate and where to place it as one probabilistic generation problem, then pass that generated pose to the texture stage so that completion, localization, and appearance orientation remain consistent with the same observations.

Method

Overall Architecture

Inputs comprise RGB, depth, camera intrinsics, and segmentation masks identifying target objects or parts from one or two views. Outputs are a textured mesh for each target and a corresponding similarity transform for each view. The pose belongs to \(\mathrm{Sim}(3)\), including isotropic scale as well as rotation and translation, rather than only a six-degree-of-freedom rigid transform.

The system first constructs "Multimodal Context Conditioning," uses "Joint Structure and Pose Generation" to recover a sparse voxel structure and poses, and performs "Pose-Conditioned Asset Recovery" to generate local latents and decode geometry and appearance. A scene consists of the assets reconstructed for individual targets; the method does not jointly optimize physical contact constraints across all objects.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["One or two views<br/>RGB-D, intrinsics, masks"] --> Context["Multimodal Context Conditioning"]
    Context --> Joint["Joint Structure and Pose Generation"]
    Joint --> Asset["Pose-Conditioned Asset Recovery"]
    Context --> Asset
    Asset --> Output["Textured meshes<br/>Per-view poses and scales"]
    Train["Synthetic shape and pose ground truth<br/>Training supervision only"] -.-> Joint
    Train -.-> Asset

This is a two-stage rectified flow generation system, not a fixed complete mesh followed by an independent registration module. The generators are trained separately, and the second stage explicitly receives the first stage's output. "Joint" primarily refers to generating shape structure and pose together in the first stage, not to end-to-end simultaneous optimization of the entire system.

Key Designs

1. Multimodal Context Conditioning: preserve occlusion cues while restricting depth to the target

DINOv2 encodes RGB, while a learnable convolution encodes the target mask and adds it to the image features instead of erasing all RGB pixels outside the mask. The model can therefore identify the reconstruction target while seeing nearby occluders. Training dynamically crops around the target with 20% to 100% padding: overly tight crops lose context, whereas a full image can dilute the useful signal from a small object. The mask identifies the target; it does not instruct the model to generate only the visible surface fragment.

Depth is treated differently. Camera intrinsics back-project it into a per-pixel 3D pointmap, and points outside the target mask are removed so that distant background depth does not distort scale estimation. Coordinate-wise medians estimate the target center, while the difference between the 5th and 95th percentiles of point-distance norms from that center estimates scale. The centered and scaled pointmap is encoded and added to image features. RGB background supplies occlusion context, whereas the pointmap grounds the target's geometry. With two views, per-view conditions are concatenated along the sequence dimension and receive learnable view identifiers; output pose tokens have corresponding identifiers. Randomly dropping the second view and its pose during training preserves single-view compatibility. The paper explicitly implements one or two views, which does not establish support for an arbitrary number of cameras.

2. Joint Structure and Pose Generation: denoise scale, orientation, and complete shape together

The complete target structure is represented as a \(64\times64\times64\) binary occupancy grid, compressed by a 3D convolutional VAE into a \(16\times16\times16\times8\) continuous feature grid. Directly modeling high-resolution binary voxels would be expensive, while the compressed continuous space is suitable for flow matching. The first stage places serialized structure features and an additional pose token in the same transformer, predicts velocity fields for both at every timestep, and uses Euler integration to evolve noise into structure and similarity transforms. Decoding the structure back to occupancy and extracting active voxels provides spatial support for the next stage; the pose maps object-centered coordinates into the normalized camera frame.

This coupling lets the network learn which complete shape is consistent with a particular size and placement, instead of forcing a registration algorithm to accept an incorrectly completed shape. The first stage represents rotation with a continuous 6D encoding and recovers the rotation matrix through Gram-Schmidt orthogonalization, avoiding difficulties caused by discontinuous parameterizations. All pose components undergo z-score normalization using training-set statistics, reducing loss imbalance between rotation, translation, and scale. Generative modeling can express multiple solutions arising from occlusion and symmetry, but the main text does not report candidate counts, candidate selection, or uncertainty calibration experiments. Multi-hypothesis capability should therefore not be interpreted as validated confidence estimation.

3. Pose-Conditioned Asset Recovery: place textures correctly even when geometry is indistinguishable

The second stage generates local structured latents at active voxels. Sparse convolutions first pack \(2^3\) spatial neighborhoods to shorten the transformer sequence, followed by a time-modulated sparse transformer and an upsampling head with skip connections to restore detail. As in the first stage, adaptive layer normalization injects time, and cross-attention injects multimodal observations. The crucial extra condition is the pose predicted in the first stage, encoded by a linear layer and concatenated with observation features. Rotation uses a 9D representation here, alongside translation and scale.

Pose conditioning matters particularly for labeled cylindrical containers: two meshes can share the same cylindrical outline while placing the label differently relative to the camera. Geometry alone cannot resolve that ambiguity for appearance generation. The generated local latents feed both a mesh decoder, which extracts geometry using FlexiCubes, and a Gaussian Splatting decoder, which produces colored 3D Gaussians. Rendering this Gaussian appearance from multiple viewpoints yields images that are baked into mesh textures. The Gaussians provide a bakeable appearance representation rather than replacing the final mesh geometry.

A Worked Example

Consider a labeled cylindrical container partly hidden by a foreground object, with only its upper half visible, accompanied by RGB-D images and target masks from two cameras. This is an explanatory example of the data flow, not an additional experimental sample reported by the paper.

First, the crop retains the occluder's boundary so the model can infer that the target may extend downward. The pointmap retains only the container's visible depth, and view identifiers distinguish the observations. The first stage generates a complete cylindrical structure and a rotation, translation, and scale for each view instead of treating the visible upper half as the entire container.

The second stage uses the generated orientation to determine label placement in canonical object coordinates, then decodes and bakes the texture. The same pipeline still runs with one view, but unseen backs and label orientations may admit multiple solutions. A second view adds constraints without guaranteeing that every error decreases.

Loss & Training

Both generators use conditional flow matching objectives. The first stage supervises structure latents and normalized poses jointly; the text explicitly specifies a pose-term weight of \(\alpha=0.01\). The second stage learns local latent generation. The cached joint conditional distribution, pose-normalization equation, and total-loss equation contain extraction damage, so this note records only the mechanisms and weight confirmed by the prose rather than reconstructing exact equations. The stated centering and scaling procedure also does not by itself guarantee that normalized pointmaps strictly lie inside the unit cube.

Training assets come from three object datasets, Objaverse-XL, ABO, and HSSD, and three part datasets, PhysXNet, PartNext, and PartNet-Mobility. Object scenes introduce occlusion by randomly placing other assets nearby, whereas part scenes mainly exploit self-occlusion within a single object. The authors report 198K assets, 198K scenes, and 3.2M RGB images with accompanying depth, masks, and poses, while also stating that each scene has 20 rendered views. These aggregate counts do not directly validate one another through multiplication, so the note preserves the reported accounting.

To accommodate real sensor noise, training uses stereo depth estimated by FoundationStereo rather than only ideal rendered depth. Appearance training excludes PartNet-Mobility and PhysXNet because of their lower-quality textures; the shape and appearance training sets are therefore not identical.

The model initializes from TRELLIS-image-large with approximately 1.2B parameters. Training uses AdamW with learning rate \(10^{-4}\), batch size 512, and 55K iterations, taking approximately 48 hours on 64 H100 GPUs. Classifier-free guidance uses a conditioning drop rate of 0.1; inference uses guidance scale 3.0 and 50 denoising steps. Ablation models instead use batch size 64 and 150K iterations, so their full-model results should not be treated as the main model's results.

Key Experimental Results

Main Results

Object evaluation covers HB, LM-O, HOPE, and ReOcS. Part evaluation uses an expanded ArtVIP benchmark: the original 6 scenes are extended to 12, with 284 selected parts and 924 rendered RGB-D images. The table below extracts verifiable HB, ReOcS, and ArtVIP rows from the paper's Table 1 without recomputing aggregate means that include HOPE.

\(\mathrm{CD}_{\mathrm{norm}}\) is Chamfer Distance after ICP alignment to ground truth, normalized by ground-truth diameter, primarily measuring shape error. ADD-SB measures bidirectional surface distances between posed predicted and ground-truth meshes, so it remains sensitive to shape error and is not a pure rotation metric. [email protected] is the fraction of samples within a distance threshold of 5% of the ground-truth object diameter.

Dataset	Method	CDnorm, lower is better	ADD-SB, lower is better	[email protected], higher is better	[email protected], higher is better
HB	SAM3D	0.033	0.062	54.6%	34.6%
HB	RecGen 1-view	0.032	0.049	73.8%	51.5%
HB	RecGen 2-view	0.029	0.048	74.2%	50.9%
ReOcS	SAM3D	0.026	0.057	43.6%	25.8%
ReOcS	RecGen 1-view	0.019	0.032	89.5%	60.8%
ReOcS	RecGen 2-view	0.018	0.032	91.1%	62.4%
ArtVIP	SAM3D	0.056	0.073	45.8%	22.6%
ArtVIP	RecGen 1-view	0.026	0.034	84.0%	24.4%
ArtVIP	RecGen 2-view	0.024	0.032	86.4%	24.8%

The proposed Diameter Relative Error, DRE, checks whether complete size is recovered under occlusion:

\[ e_d=\frac{|d_{\mathrm{pred}}-d_{\mathrm{gt}}|}{d_{\mathrm{gt}}}. \]

[email protected] is the fraction of samples satisfying \(e_d<0.05\). Unlike Chamfer Distance, it explicitly tests complete object scale, which need not be sufficiently accurate merely because surfaces look similar.

The paper's Table 2 further separates rendering quality before and after ICP. The following extract retains only SAM3D and single-view RecGen. The symmetric subset comes from HOPE and HB and should not be conflated with the evaluation scope of the full-set mean.

Evaluation scope	Method	Before ICP LPIPS	Before ICP SSIM	Before ICP PSNR	After ICP LPIPS	After ICP SSIM	After ICP PSNR
LM-O + HB + HOPE	SAM3D	0.219	0.821	15.72	0.161	0.841	17.42
LM-O + HB + HOPE	RecGen 1-view	0.199	0.825	15.85	0.170	0.834	16.54
Symmetric object subset	SAM3D	0.201	0.815	16.02	0.156	0.828	17.21
Symmetric object subset	RecGen 1-view	0.170	0.816	15.63	0.142	0.827	16.12

Lower LPIPS and higher SSIM and PSNR are better. RecGen leads on all three aggregate pre-ICP metrics but trails SAM3D on all three aggregate post-ICP metrics. Better LPIPS on symmetric objects also does not imply better PSNR. These results support benefits in scene alignment and texture orientation, not a claim of universally superior appearance quality.

Ablation Study

The following extract from the paper's Table 3 reports only means, avoiding mixing the original mean / median entries. Object evaluation covers HB, LM-O, and ReOcS, while part evaluation uses ArtVIP. All rows use the smaller training configuration described above, and lower is better for every metric.

Config	Object CDnorm	Object ADD-SB	Part CDnorm	Part ADD-SB
Full model	0.042	0.062	0.033	0.043
Without stereo depth noise	0.048	0.078	0.030	0.039
Without pose normalization	0.042	0.074	0.038	0.056
Without part training data	0.040	0.060	0.073	0.086
Without pretraining	0.044	0.067	0.044	0.056

Key Findings

• Part reconstruction does not emerge automatically from a general object prior: removing part data increases ArtVIP CDnorm from 0.033 to 0.073 and ADD-SB from 0.043 to 0.086, while object metrics improve slightly.
• Noise training benefits depend on the test domain: removing stereo depth noise increases real-object ADD-SB from 0.062 to 0.078, but decreases it from 0.043 to 0.039 on noise-free synthetic ArtVIP. The effect is not uniformly beneficial across all settings.
• In the paper's Table 4, dynamic cropping achieves HB ADD-SB of 0.052 versus 0.132 for full-image input. However, mask overlay achieves CDnorm of 0.031 versus dynamic cropping's 0.035, so the crop strategy is not optimal for every metric.
• Simulation replay of 58 real teleoperated trajectories across 3 tasks succeeds at 59%, versus SAM3D's 38%, a gain of 21 percentage points. Across 60 DROID scenes and 200 objects, Chamfer Distance to the input pointmap is 0.015 versus 0.021; this measures visible-geometry consistency, not ground-truth error of complete hidden surfaces.

Highlights & Insights

• RGB and depth backgrounds need not be processed identically. Keeping the former provides occlusion context, while filtering the latter prevents scale contamination, giving this conditioning design a task-specific rationale.
• Symmetry affects not only pose evaluation but also texture generation. Conditioning appearance on the generated pose supplies a consistent texture-coordinate reference for geometrically indistinguishable orientations.
• Shape metrics and posed-rendering metrics must be examined separately. The before/after ICP comparison shows that some visual gains arise from correct asset placement rather than a uniformly stronger texture decoder.

Limitations & Future Work

• Inputs assume depth, camera intrinsics, and object or part masks. This is not a complete system for automatically discovering every object from unannotated RGB, and segmentation-error sensitivity is not systematically evaluated.
• Two views do not always outperform one: LM-O CDnorm changes from 0.050 to 0.056 and ADD-SB from 0.068 to 0.075; HB [email protected] also decreases from 51.5% to 50.9%.
• Part geometry and poses do not specify joint axes, contacts, mass, or friction. Replay success remains 59%, insufficient to establish generally physically valid interactive digital twins.
• The authors report approximately 80% fewer training meshes than SAM3D, but still rely on a large pretrained TRELLIS model and 64 H100 GPUs. Fewer assets do not establish lower from-scratch training compute. The main text does not report explicit per-scene inference latency.
• Some aggregate statements conflict with individual results: for example, changing mean object CDnorm from 0.033 to 0.034 is described as an improvement despite lower being better. This note follows individual table entries and does not extrapolate the abstract's overall percentages to every setting.

Related Work & Insights

• vs Any6D / SceneComplete: These methods generate or complete shapes before matching and registration, whereas RecGen couples structure and pose within generation. The useful principle is to infer uncertain shape and scale together instead of prematurely fixing an incorrect shape.
• vs SAM3D: Both jointly reconstruct shape and layout. RecGen emphasizes pose-conditioned appearance, part data, realistic depth noise, and two-view conditioning. Its strengths at strict pose thresholds and on parts do not negate SAM3D's advantage on some post-ICP appearance metrics.
• vs TRELLIS: RecGen reuses structured 3D latents and pretrained weights while extending object generation to scene-asset reconstruction grounded in geometric observations. Physical-consistency constraints could help select generated candidates, but that is a future direction, not a mechanism validated here.

Rating

• Novelty: 4/5. Joint generation, pose-conditioned appearance, and multimodal training form a coherent design for practical reconstruction, while retaining a strong pretrained backbone.
• Experimental Thoroughness: 4/5. Evaluation spans objects, parts, appearance, ablations, and robotic replay, but lacks detailed speed, uncertainty, and input-mask robustness studies.
• Writing Quality: 3/5. The mechanisms are clearly described, but some aggregate claims are imprecise and the cached equations show substantial extraction damage.
• Value: 4/5. A useful direction for real-to-sim asset construction under occlusion, with physical usability still requiring additional constraints and validation.