title: >- [Paper Note] PWM-ArtGen: Part World Model for Articulated Object Generation description: >- [ECCV 2026][3D Vision][Articulated Object Generation] Proposes a unified Part World Model (PWM-ArtGen) that couples visual dynamics and kinematic parameters via decoupled diffusion timesteps, enabling scalable co-training on 19.7k unannotated images to achieve state-of-the-art single-image 3D articulated object generation. tags: - ECCV 2026 - 3D Vision - Articulated Object Generation - Visual Dynamics - World Model date: 2026-09-19 content_hash: 03abed68f97f12f5

PWM-ArtGen: Part World Model for Articulated Object Generation¶

Conference: ECCV 2026
Paper: ECCV 2026
Code: https://github.com/Wentap123/PWM-ArtGen
Area: 3D Vision
Keywords: articulated object generation, visual dynamics, kinematic parameters, part world model, co-training

TL;DR¶

PWM-ArtGen proposes the first part world model for single-image articulated 3D object generation, coupling action-induced visual dynamics and kinematic parameters through decoupled diffusion timesteps, visual dynamics regularization, and co-training on 19.7k unannotated photorealistic images to achieve robust zero-shot articulated generation.

Background & Motivation¶

Articulated objects such as cabinets, refrigerators, and appliances constitute fundamental building blocks for interactive virtual environments, robotics simulation, and embodied AI. However, creating high-quality interactive 3D assets requires laborious expert modeling, which severely limits scalability. Automatically generating articulated assets from a single static image has emerged as an attractive alternative, yet existing approaches suffer from fundamental limitations. Direct feedforward regression methods attempt to infer joint types, axes, and motion limits directly from a single resting snapshot. Because a static observation inherently lacks dynamic motion evidence, these methods degrade drastically when faced with structural ambiguity, complex textures, or occlusions. Conversely, optimization-based and multi-stage approaches attempt to synthesize multi-view or video observations from the single image before recovering kinematic parameters, which is prone to severe compounding errors across stages and heavy computational overhead.

A further roadblock is the scarcity and quality gap in training data. Widely used synthetic benchmarks such as PartNet-Mobility (PM) lack photorealism in materials, lighting, and backgrounds, causing poor generalization to in-the-wild physical scenes. Conversely, realistic datasets like the Articulated Containers Dataset (ACD) are too small to support large-scale supervised training. Annotating 3D kinematic parameters (joint axes, rotation angles, translation limits) in real-world environments is prohibitively expensive. Consequently, developing paradigms that can leverage abundant, unannotated visual data to bolster 3D kinematic estimation is essential for scalable physical asset synthesis.

To resolve these tensions, the authors argue that visual dynamics and physical kinematics are fundamentally interdependent: observed visual motion provides direct geometric constraints on joint types and orientations, while kinematic parameters govern the underlying deformation field. The core idea is to frame articulated object generation as a part-level world model (PWM-ArtGen) that jointly learns the coupled distribution of visual dynamics and kinematic parameters using decoupled diffusion timesteps, incorporates a Visual Dynamics Regularizer (VDR) to anchor semantic trajectories, and leverages co-training on 19.7k photorealistic unannotated image pairs (PM-R) to bridge the synthetic-to-real domain gap.

Method¶

Overall Architecture¶

PWM-ArtGen transforms a single input image $o$ into an interactive 3D articulated object mesh with verified kinematic relationships. The pipeline first uses an off-the-shelf module (combining SAM and GPT-4o) to extract candidate part masks and an articulation connectivity graph. For each target part, local structural guidance is constructed and passed alongside the global static image into a unified Diffusion Transformer (DiT). The DiT jointly predicts the post-action dynamic visual observation $o'$ and kinematic parameters $a$. All part predictions are subsequently aligned using a shared base bounding box anchor and refined via CAD part retrieval to ensure collision-free geometry.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Static Image o"] --> B["Part Mask & Graph Extraction<br/>SAM segmentation & GPT-4o connectivity"]
    B --> C["Hierarchical Part-Level Guidance<br/>Cropped observation & normalized 2D bboxes"]
    C --> D["Joint Diffusion with Decoupled Timesteps<br/>DiT joint noise prediction for dynamics & kinematics"]
    D --> E["Visual Dynamics Regularizer VDR<br/>Patch-level cosine alignment with frozen DINOv2"]
    D -.->|Inject unannotated PM-R pairs| F["Synthetic-Real Mixed Co-Training<br/>Weighted round-robin sampling & action-free noise"]
    E --> G["Global Base Alignment & Retrieval<br/>Shared base bbox assembly & part regularization"]
    F --> G
    G --> H["Output Articulated 3D Object Mesh"]

Key Designs¶

1. Joint Diffusion with Decoupled Timesteps: Unifying Dynamics Modeling and Unannotated Data Ingestion To eliminate accumulated errors from cascaded video-to-kinematics pipelines while bypassing the ambiguity of static snapshot regression, PWM-ArtGen models the joint conditional distribution $p(o', a \mid o, p^{\text{ref}})$ with a unified noise-prediction network. Crucially, the model assigns independent diffusion timesteps $t_a \sim \mathcal{U}(0, T)$ and $t_{o'} \sim \mathcal{U}(0, T)$ to the kinematic action branch and the visual dynamics branch, respectively. The post-action image $o'$ is mapped into latent space using a frozen SDXL VAE and tokenized into $N$ visual patch tokens, while kinematic attributes are projected into 20 parameter tokens via a trainable MLP. Because timesteps are independent, action-free samples can be seamlessly integrated during co-training by fixing $t_a = T$ (pure Gaussian noise). This enables the visual branch to absorb rich physical motion patterns from unannotated images without requiring manual kinematic annotations, directly enhancing parameter estimation.

2. Hierarchical Part-Level Structural Guidance: Removing Part Count Limits via Shared Base Anchoring Prior frameworks define rigid, object-level slot architectures that enforce an arbitrary maximum part count, limiting scalability to intricate multi-part assemblies. In contrast, PWM-ArtGen operates on individual articulated parts. Each part guidance is defined as $p^{\text{ref}} = \{o^{\text{part}}, \text{2Dbbox}^{\text{part}}, \text{2Dbbox}^{\text{base}}\}$, where the cropped part $o^{\text{part}}$ and full image $o$ are encoded by DINOv2 to yield local and global visual embeddings $f_{o^{\text{part}}}$ and $f_o$. The normalized 2D bounding boxes are concatenated to form the reference embedding $f^{\text{ref}}$, which conditions the DiT blocks via adaptive layer normalization (AdaLN). Furthermore, the kinematic attribute vector is defined as $a = \{t, r, l, d, b, b_{\text{base}}\}$, predicting the joint type $t$ (revolute or prismatic), motion range $r$, joint axis location $l \in \mathbb{R}^3$, direction $d \in \mathbb{R}^3$, part 3D bounding box $b \in \mathbb{R}^6$, and a shared base bounding box $b_{\text{base}} \in \mathbb{R}^6$. Because every part estimates the exact same global base box $b_{\text{base}}$, subsequent assembly aligns all parts relative to this shared physical frame, completely eliminating part-count constraints.

3. Visual Dynamics Regularizer (VDR): Semantic Manifold Trajectory Alignment Generative image diffusion in the latent space can occasionally introduce high-frequency texture distortions or physically ungrounded deformations that mislead the coupled kinematic tokens. Inspired by self-supervised feature alignment in REPA, the authors introduce a Visual Dynamics Regularizer (VDR). During the joint denoising process, intermediate hidden states $h_{t_{o'}}$ from the first $l$ Transformer layers of the visual branch are aligned via cosine similarity with patch-level semantic features $y_{o'}^{[n]}$ extracted by a frozen DINOv2 encoder from clean ground-truth post-action images: $$ \mathcal{L}{\text{VDR}}(\theta) = -\mathbb{E} \right) \right] $$ This regularizer anchors intermediate denoising trajectories to the semantic manifold of clean real images, ensuring structural part boundaries and rigid-body consistency during simulated interaction.}, t_{o'}, t_a} \left[ \frac{1}{N} \sum_{n=1}^{N} \operatorname{sim}\left( y_{o'}^{[n]}, h_{t_{o'}}^{[n]

4. Synthetic-Real Mixed Co-Training on PM-R: Bridging the Sim-to-Real Domain Gap To overcome the severe domain shift between synthetic CAD renderings and real physical environments, the authors constructed PartNet-Mobility-Reality (PM-R), a high-fidelity dataset of 19.7k unannotated part-level image pairs. By executing a curated prompt library with an off-the-shelf image-to-image editing model (Qwen-Image-Edit-2509), synthetic PartNet-Mobility renderings are augmented with photorealistic materials, realistic ambient illumination, and natural indoor backgrounds. To maintain balanced batch representations across workers in Distributed Data Parallel (DDP) training, an epoch-level length-proportional interleaving sampler shuffles both subsets independently and generates global index streams using weighted round-robin scheduling, ensuring stable domain transfer into the unified representation.

Loss & Training¶

The overall training objective combines the co-training diffusion denoising loss and the visual dynamics regularizer: $$ \mathcal{L} = \mathcal{L}{\epsilon_a, \epsilon}} + \lambda \mathcal{L{\text{VDR}} $$ where the joint diffusion loss $\mathcal{L}_{\epsilon_a, \epsilon_{o'}}$ aggregates fully annotated pairs $\mathcal{D}_{\text{paired}}$ and action-free pairs $\mathcal{D}_{\text{free}}$: $$ \begin{aligned} \mathcal{L}}}(\theta) = & \mathbb{E{\substack{(o, p^{\text{ref}}, a, o') \sim \mathcal{D}|^2 \right] \ & + \gamma \mathbb{E}}} \ t_a, t_{o'} \sim \mathcal{U}(0, T), \epsilon_a, \epsilon_{o'} \sim \mathcal{N}(0, 1)}} \left[ w_a |\epsilon_a^\theta - \epsilon_a|^2 + w_{o'} |\epsilon_{o'}^\theta - \epsilon_{o'{\substack{(o, p^{\text{ref}}, \emptyset, o') \sim \mathcal{D}|^2 \right] \end{aligned} $$ Training uses weights }} \ t_a = T, t_{o'} \sim \mathcal{U}(0, T), \epsilon_a, \epsilon_{o'} \sim \mathcal{N}(0, 1)}} \left[ w_a |\epsilon_a^\theta - \epsilon_a|^2 + w_{o'} |\epsilon_{o'}^\theta - \epsilon_{o'$w_a = w_{o'} = 1.0$ and action-free balance weight $\gamma = 1.0$. The DiT architecture has 12 layers, 12 attention heads, hidden dimension 768, and $2 \times 2$ patch tokens. Optimization runs on 4 NVIDIA RTX 4090 GPUs for 200k steps with global batch size 128 using AdamW (learning rate $1 \times 10^{-4}$, weight decay $1 \times 10^{-6}$). VDR is applied during the initial 80k steps with $\lambda = 0.5$ using a frozen DINOv2-Reg/14 encoder. Inference uses 10-step DDIM sampling.

Key Experimental Results¶

Main Results¶

Evaluation was conducted on both zero-shot out-of-distribution real containers (ACD test set: 135 objects, 455 parts, 910 samples) and unseen synthetic objects (PartNet-Mobility test set: 77 objects, 155 parts, 310 samples). Metrics include generalized bounding-box IoU (dgIoU), part center distance (dcDist), Chamfer Distance (dCD), Average Overlapping Ratio (AOR) for collisions, and articulation graph accuracy (Acc), measured in both Resting State (RS) and Articulated State (AS).

Quantitative comparison on the realistic ACD test set (Table 1 of the paper):

Method	RS-dgIoU ↓	AS-dgIoU ↓	RS-dcDist ↓	AS-dcDist ↓	RS-dCD ↓	AS-dCD ↓	AOR ↓	Acc% ↑
URDFormer-GTbbox	1.1986	1.2012	0.2292	0.2931	0.4343	0.4965	0.1209	48.53
NAP-ICA-GTgraph	1.0585	1.0631	0.1987	0.2810	0.1376	0.2417	0.0193	36.67
SINGAPO-GTgraph	0.9729	0.9767	0.1589	0.1968	0.1177	0.1752	0.0112	100.00
Ours-GTgraph	0.5693	0.5790	0.0687	0.1144	0.0569	0.0813	0.0031	100.00
URDFormer (w/o GT)	1.2288	1.2309	0.2914	0.4285	0.7198	0.8995	0.2840	4.48
NAP-ICA (w/o GT)	1.0233	1.0286	0.1691	0.2331	0.1110	0.1887	0.0133	16.67
SINGAPO (w/o GT)	0.9775	0.9810	0.1574	0.2016	0.1100	0.1744	0.0103	39.34
Articulate-Anything	0.7574	0.7649	0.1938	0.2684	0.1826	0.2861	0.0039	40.00
Ours (w/o GT)	0.6822	0.6914	0.1480	0.1876	0.1033	0.1385	0.0906	39.34

Quantitative evaluation of the visual dynamics branch on the PartNet-Mobility test set (Table 3 of the paper):

Method	PSNR ↑	SSIM ↑	LPIPS ↓	Parameters	Inference Time
DragAPart	24.803	0.943	0.079	1.43 B	6.9 s
Puppet-Master	24.440	0.941	0.084	1.68 B	11.6 s
Ours	24.328	0.906	0.093	0.38 B	0.7 s

Ablation Study¶

Ablation analysis on key designs for kinematic estimation on the ACD dataset with GT graph (Table 4 of the paper):

Settings (IB / VDR / Co-T)	RS-dgIoU ↓	AS-dgIoU ↓	RS-dcDist ↓	AS-dcDist ↓	RS-dCD ↓	AS-dCD ↓	AOR ↓
Action branch only (w/o IB)	0.6567	0.6684	0.0801	0.1877	0.0796	0.1520	0.0252
✓ IB (Joint Diffusion)	0.5948	0.6045	0.0739	0.1202	0.0681	0.0915	0.0033
✓ IB + ✓ VDR	0.5885	0.5986	0.0699	0.1172	0.0587	0.0840	0.0038
✓ IB + ✓ Co-T	0.5786	0.5878	0.0696	0.1174	0.0619	0.0854	0.0020
✓ IB + ✓ VDR + ✓ Co-T (Full model)	0.5693	0.5790	0.0687	0.1144	0.0569	0.0813	0.0031

Key Findings¶

Joint Visual Dynamics Modeling is Crucial: Removing the image branch (IB) degrades kinematic estimation dramatically; articulated center distance error AS-dcDist surges from 0.1202 to 0.1877 (a 56% degradation). Visual dynamics provide essential physical grounding for joint axes and boundaries.
Co-Training Unlocks Real-World Generalization: Incorporating the 19.7k unannotated PM-R image pairs via co-training reduces the Chamfer Distance on the realistic ACD dataset from 0.0681 to 0.0569 (RS) and from 0.0915 to 0.0813 (AS), directly proving that unannotated photorealistic images bridge the sim-to-real gap.
Superior Inference Efficiency: PWM-ArtGen uses only 0.38B parameters (one-fourth of DragAPart's 1.43B) and completes inference in 0.7s (nearly $10\times$ to $16\times$ faster than DragAPart and Puppet-Master), while producing visual dynamics competitive with heavy video diffusion backbones.
Mask Quality Sensitivity: Without ground-truth masks, performance drops slightly on synthetic PartNet-Mobility due to rendering artifacts that degrade SAM/GPT-4o segmentation, whereas results remain robust on higher-quality ACD objects.

Highlights & Insights¶

World Model Applied to 3D Kinematics: Re-imagines the world model paradigm beyond control or flat video generation, formulating it as a physical generative prior for 3D articulated asset creation.
Decoupled Diffusion Timesteps for Free-Data Co-Training: Elegantly harmonizes supervised kinematic prediction and unsupervised visual learning by fixing $t_a=T$ on action-free data, transforming raw internet/edited images into physical learning signals.
Lightweight Interactive Generation: Demonstrates that a compact 12-layer DiT with local DINOv2 anchoring can match or outperform multi-billion parameter foundation video models at a fraction of latency and compute.

Limitations & Future Work¶

Dependency on Upstream Mask Generators: Downstream accuracy hinges on the quality of initial masks and connectivity graphs generated by SAM and GPT-4o; errors in the initial graph cascade into subsequent assembly.
Limited Articulation Types: Currently supports single-degree-of-freedom revolute and prismatic joints, without covering complex ball-and-socket joints, screw joints, or deformable continuum structures.
Future Directions: Integrating mask prediction, kinematic tree discovery, and geometric denoising into a single unified end-to-end framework.

vs SINGAPO & CAGE: SINGAPO relies on direct feedforward regression from static snapshots. When evaluating on realistic ACD containers out-of-distribution, PWM-ArtGen reduces Chamfer Distance by more than 50% thanks to motion-grounded visual dynamics.
vs DragAPart & Puppet-Master: These baselines require heavy video diffusion backbones (>1.4B parameters) and several seconds per generation to produce only 2D pixel drags; PWM-ArtGen achieves 0.7s interactive inference with 0.38B parameters while producing fully articulable 3D meshes.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Pioneering introduction of a part-level world model for single-image articulated 3D object generation with decoupled diffusion timesteps.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive zero-shot benchmark comparisons on ACD, PartNet-Mobility, dynamics metrics, and ablation studies.
Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical formulation, clear conceptual motivation, and high-clarity structural figures.
Value: ⭐⭐⭐⭐⭐ Provides an efficient, scalable path toward high-fidelity interactive physical asset generation for robotics and embodied AI.