DreamPartGen: Semantically Grounded Part-Level 3D Generation via Collaborative Latent Denoising

Conference: ECCV2026
Paper: ECCV Official Page
Area: 3D Vision
Keywords: part-level 3D generation, relational semantics, collaborative denoising, geometry-appearance disentanglement, text-conditioned generation

TL;DR

DreamPartGen couples part geometry and appearance with language-derived relational semantics through intra-part and inter-part synchronized denoising, reducing CD on PartRel3D from PartCrafter's 0.371 to 0.081 while producing editable compositions, although low part overlap alone does not establish physically correct assembly.

Background & Motivation

Text-to-3D generation can produce complete-looking objects, but resembling an airplane is different from generating distinct wings, a propeller, and landing gear that connect correctly. Monolithic object latents can entangle geometry, materials, and semantics, making it difficult to edit one component without changing the others. Part-based methods such as PartGen and PartCrafter provide finer generation units, yet individually plausible components can still drift apart or exhibit damaged contact regions when assembled.

The issue is therefore not just whether an object has been split into meshes, but whether the model continuously represents the roles and relations of those parts. A metallic blade specifies local material and shape; a propeller attached to a fuselage constrains assembly; symmetric wings involve multiple components. Supplying text conditioning to a denoiser does not necessarily maintain these different kinds of state explicitly. Bounding boxes and local geometry alone also do not fully express functional relations such as support and articulation.

DreamPartGen consequently turns language into structured representations within generation: local semantics undergo noising and recovery alongside parts, while global relations remain available as a non-noised planner state. Core Idea: route part attributes and inter-part assembly relations through local and global synchronization, respectively, so that geometry, appearance, and assembly are coordinated at every denoising step.

Method

Overall Architecture

The input is an object description, optionally accompanied by explicit part-relation triplets; the output is a collection of independently addressable part meshes and their assembled object. Duplex Part Latents preserve each part's geometry, appearance, and slot identity. Relational Semantic Latents separate local attributes from global relations, and Two-Level Synchronized Co-Denoising coordinates the representations.

During training, clean geometry and appearance representations come from part meshes and multi-view renderings. At inference time, these ground-truth parts are unavailable, and the two part streams are initialized from Gaussian noise. Text still supplies semantic conditions: local semantics undergo the same forward noising process used in training and are recovered jointly with the part representations before the 3D VAE decodes part meshes.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    INPUT["Text description<br/>Optional relation triplets"] --> DPL["Duplex Part Latents"]
    INPUT --> RSL["Relational Semantic Latents"]
    TRAIN["Training: meshes and views<br/>Inference: Gaussian noise"] --> DPL
    DPL --> SYNC["Two-Level Synchronized<br/>Co-Denoising"]
    RSL -->|Local semantics noised; global relations not noised| SYNC
    SYNC -->|Next denoising step| SYNC
    SYNC --> OUT["Decode part meshes<br/>Assemble and render appearance"]

Duplex Part Latents and Relational Semantic Latents are prepared in parallel; the diagram does not mean relations are first extracted from generated parts. Training relations come from canonicalized descriptions. Inference can use prompt-only conditioning and does not require an external vision-language model (VLM).

Key Designs

1. Duplex Part Latents: preserve distinct representations for geometry, appearance, and part identity

Duplex Part Latents (DPLs) maintain geometry tokens, appearance tokens, and a learnable identifier for each semantic part instead of assigning one mixed token sequence to the whole object. The geometry branch encodes sampled surface points and normals with a 3D VAE. The appearance branch encodes multi-view part renderings with a pretrained image VAE. The former captures local shape and spatial structure, whereas the latter captures color, texture, and shading. This allows an incorrect wing shape and an incorrect wing color to be handled in separate streams before synchronization reconciles them.

The identifier provides persistent slots across denoising timesteps, keeping each part trackable and reducing the risk of slot swapping. This representation supports updating a target part or transferring similar functional components across objects, but an identifier alone does not guarantee correct assembly. The paper also claims robustness to input part ordering. The supplied main text ablates identifier removal but does not include a dedicated order-perturbation experiment, so the architectural motivation should be distinguished from a separately tested property.

2. Relational Semantic Latents: maintain local attributes separately from global assembly relations

Relational Semantic Latents (RSLs) are obtained through a frozen text encoder and a learned projection, with two groups serving different purposes. Global relational tokens \(S_{\mathrm{glb}}\) encode canonicalized part-part-predicate triplets, such as a wing attached to a fuselage, two wings being symmetric, or chair legs supporting a seat. These tokens represent edges in a relation graph rather than a single summary of the entire object, providing explicit assembly guidance during inter-part message passing.

Local semantic tokens \(S_{\mathrm{loc},t}\) describe details such as metallic blades or wooden handles and are noised and denoised together with the geometry and appearance streams. The essential distinction is that local semantics are diffusion states, whereas global relations are not diffusion variables. Although the main text sometimes calls global relations fixed conditions, Equation (4) and its explanation explicitly allow deterministic updates from pooled current part features. Here, fixed should mean continuously available relational conditioning without stochastic noising, not numerically unchanged values at every timestep.

3. Two-Level Synchronized Co-Denoising: align each part internally before coordinating it with other parts

At each denoising step, intra-part synchronization first exchanges information between geometry and appearance through bidirectional cross-attention. Both streams then attend to noised local semantic tokens, with learnable fusion coefficients controlling residual updates. A pink-wing description can therefore inform the ongoing joint recovery of appearance and geometry instead of affecting only rendered color. The relevant comparison is not simply text versus no text: the conditioning-only baseline retains timestep-wise text cross-attention but removes persistent semantic states.

Inter-part synchronization then exchanges context across geometry streams and across appearance streams, while both read global relational tokens. Pooled features from the current parts also update the global planner, creating a loop in which relations guide parts and emerging part evidence refines the plan. This helps explain how individually plausible wings or wheels can be coordinated with the overall object during recovery. Attention remains a learned mechanism, however, rather than a hard contact, collision, or mechanics solver.

A Worked Example

Consider the toy airplane description used in the paper: pink wings, a blue propeller, and landing wheels. The following walkthrough illustrates the reported mechanism rather than adding a quantitative experiment.

First, trackable slots are assigned to semantic parts such as wings, the propeller, the fuselage, and wheels, and geometry and appearance noise are initialized for each. Colors and part attributes enter the local semantic stream. When explicit relations are provided, triplets such as the propeller being attached to the fuselage and the wings being symmetric are encoded into the global relation state.

At each iteration, intra-part synchronization coordinates the emerging wing shape with its pink appearance. Inter-part synchronization uses relational information to constrain how wings relate to the fuselage and how wheels fit the object. Parts are not all generated independently and assembled only at the end; their recovery repeatedly incorporates coordination before final mesh decoding.

For a targeted edit, the paper freezes other part latents and the global relational context, applies partial DDIM inversion and localized re-denoising to the selected part, and performs a brief synchronization step to restore coherence. This is a downstream use of the representation, not evidence that every possible edit has been quantitatively validated.

Loss & Training

Training has two phases. First, the 3D and 2D part denoisers are optimized under relational semantic conditioning to predict injected noise, with separate mean-squared errors averaged across parts. The second phase jointly fine-tunes both denoisers and the synchronization modules, weighting their diffusion losses with a timestep schedule based on the signal-to-noise ratio (SNR).

The method describes local semantics as jointly denoised, but Equations (5) and (6) explicitly list only the 3D and 2D noise-prediction losses. Equations are also misaligned in the supplied text extraction. It would therefore be unjustified to invent an independently specified semantic loss or reconstruct the exact SNR-weight expression from damaged formatting. Full reproduction still requires implementation details and the relevant appendix.

Supervision comes from PartRel3D, which extends PartVerse with canonicalized spatial and functional relations. It contains approximately 11K part-labeled objects across 175 categories, over 90K parts, and 300K relations; Figure 3 lists 120K functional and 180K spatial relations. An object has 8.2 parts and 27 relations on average. This supplies supervision for attachment, support, vertical placement, contact, and symmetry, rather than only teaching what individual components look like.

At inference time, explicit triplets may come from the user or an external parser. Without them, the model uses prompt-only conditioning instead of requiring a VLM to supply missing relations. Geometry latents are decoded into meshes, while 2D latents support appearance rendering when needed. For mini-scenes, the paper treats objects as macro-parts, generates them independently, and then performs a short scene-level relational synchronization. This should be distinguished from iterative synchronization inside a single object.

Key Experimental Results

Main Results

CD and EMD measure geometric error, with lower values preferred. IoU here is not segmentation accuracy: it is the average pairwise intersection-over-union of generated parts voxelized in a shared \(64\times64\times64\) grid. Lower values indicate less overlap between parts, but do not independently establish complete connections or physical validity.

The following excerpt from Table 1 uses PartCrafter as a consistent reference instead of merging gains over different baselines and datasets into one number. Values retain the scales reported in the paper.

Dataset	Method	CD (lower better)	EMD (lower better)	Pairwise Part IoU (lower better)
Objaverse	PartCrafter	0.278	1.107	0.453
Objaverse	DreamPartGen	0.141	0.810	0.359
ShapeNet	PartCrafter	0.451	1.252	0.499
ShapeNet	DreamPartGen	0.222	0.967	0.503
ABO	PartCrafter	0.266	0.905	0.505
ABO	DreamPartGen	0.101	0.531	0.404
PartRel3D	PartCrafter	0.371	1.474	0.700
PartRel3D	DreamPartGen	0.081	0.412	0.304

On PartRel3D, CD is approximately 78.2% lower than PartCrafter's result. This is not the reduction relative to the strongest previous method in that column, because HoloPart achieves CD 0.355. On ShapeNet, IoU increases slightly from 0.499 to 0.503. Consequently, the prose claim of the best IoU on every benchmark does not fully agree with Table 1; this note follows the individual table entries.

Table 2 additionally evaluates text-shape alignment on PartVerse, with half the test descriptions referring to parts and half to complete objects. ULIP-T is the inner product of normalized text and shape embeddings, with higher values preferred:

\[ \operatorname{ULIP\!\text{-}\!T}(T,S)=\langle \widehat{E}_T,\widehat{E}_S\rangle. \]

Part-level ULIP-T is 0.153 for DreamPartGen, 0.113 for HoloPart, and 0.109 for PartCrafter. The gain over HoloPart, the stronger of these two references, is approximately 35.4%. This supports improved semantic alignment but does not replace direct measurement of whether individual relations are satisfied.

Ablation Study

The following results come from Table 3, evaluated on a fixed random PartRel3D subset rather than the full evaluation in Table 1. The full model's CD of 0.145 must therefore not be directly subtracted from the earlier 0.081.

Config	CD (lower better)	EMD (lower better)	IoU (lower better)	ULIP-T (higher better)
Full model	0.145	0.771	0.212	0.158
Without global relational tokens	0.292	2.892	0.587	0.084
Without local semantic tokens	0.781	5.764	0.652	0.089
Without part identifiers	0.277	1.709	0.438	0.091

Key Findings

• Removing local semantic tokens raises CD from 0.145 to 0.781, the largest geometric degradation in this ablation, supporting the importance of persistent semantic states for local recovery.
• Removing global relational tokens raises IoU from 0.212 to 0.587 and lowers ULIP-T from 0.158 to 0.084. Relational planning affects both overlap and alignment, although this remains a component-removal experiment within a jointly designed system.
• Table 4 reports prompt-only part-generation latency of 109 seconds versus 112 seconds for PartCrafter. These are close, so an across-the-board order-of-magnitude speedup would be misleading. Object, part, and mini-scene timings are grouped separately and should not be ranked across rows.

Highlights & Insights

• Separating language states by temporal behavior is useful: local attributes undergo stochastic diffusion, while global structure remains a non-noised planner state. This permits iterative local refinement without placing all assembly guidance inside a noisy state.
• Part identifiers make editability concrete through addressable latents. Freezing non-target slots before localized re-denoising provides a more explicit mechanism than asking a whole-object prompt to leave everything else unchanged.
• The dataset contribution matches the model design: PartRel3D triplets supervise global planning tokens directly. A transferable principle is to maintain explicit relations between editable entities during generation rather than merely lengthening natural-language descriptions.

Limitations & Future Work

• The authors identify densely entangled structures, such as bushes, as future work. These objects are difficult to divide into stable semantic slots, potentially weakening the representation's advantages.
• Low part overlap and high ULIP-T do not prove correct support, contact, articulation, or motion validity. Future evaluation should measure relation satisfaction, contact error, and collisions after joint movement, rather than relying only on static renderings.
• The main text refers to appendices for relation canonicalization, training details, rare parts, and held-out relations, but the supplied cache ends with the references and does not contain Appendices A through D. Those extended experiments are not treated here as individually verified, and unspecified token counts or hyperparameters are not supplied.
• The description of global conditions as fixed must be read alongside their deterministic updates, and the exact local-semantic denoising objective is incompletely specified in the main text. The discrepancy between ShapeNet IoU in Table 1 and the summary claim further motivates table-by-table verification during reproduction.

Related Work & Insights

• Compared with CoPart: CoPart already uses mutual guidance between 2D and 3D part latents. DreamPartGen's emphasis is therefore not the first combination of these modalities, but persistent relational semantic states within local and global synchronization.
• Compared with PartCrafter: Both use structured part latents, while DreamPartGen additionally represents relational predicates and local language attributes explicitly. Geometric errors decrease substantially, but the ShapeNet overlap metric is not uniformly better.
• Compared with Trellis and CLAY: These whole-object generation baselines provide geometric-quality references, whereas DreamPartGen emphasizes trackable parts and relational control. Missing part-IoU entries in Table 1 do not establish that they are inferior for every generation use case.

Rating

• Novelty: 4/5. Dual-stream part representations have precedents; the main advance is coordination between persistent local semantics and global relational planning.
• Experimental Thoroughness: 4/5. Four geometric benchmarks, text alignment, and component ablations provide useful coverage, but direct relation-validity evaluation remains limited and the appendices are absent from the supplied cache.
• Writing Quality: 3/5. The overall design is understandable, but global-conditioning terminology, local-semantic training details, and the IoU summary need greater precision.
• Value: 4/5. The approach is relevant to editable 3D assets and compositional generation, while real interaction and physical validity require further validation.