---

title: >- [Paper Note] Articulated Object Reconstruction from Rest-State Observation description: >- [ECCV 2026][3D Vision][Articulated Object Reconstruction] Rest2Art reconstructs multi-part articulated 3D objects from a single closed rest-state observation by iteratively co-refining VLM and SAM3 predictions on explicit meshes and fitting joints to video diffusion motion hypotheses. tags: - ECCV 2026 - 3D Vision - Articulated Objects - Mesh Segmentation - Video Diffusion date: 2026-09-19 content_hash: 106bfd4ba8902a54

Articulated Object Reconstruction from Rest-State Observation

Conference: ECCV 2026
Paper: ECCV Official
Area: 3D Vision
Keywords: Articulated Object Reconstruction, Rest-State Observation, Mesh Part Segmentation, Video Diffusion Model, Digital Twin

TL;DR

Rest2Art reconstructs multi-part articulated 3D objects directly from a single closed configuration by iteratively co-refining vision-language and segmentation predictions grounded on an explicit mesh, and estimating joint parameters from video diffusion motion hypotheses validated against geometric constraints.

Background & Motivation

Humans possess a remarkable ability to deduce how unfamiliar objects articulate simply from static visual appearance: upon encountering a closed cabinet, we readily anticipate how doors swing or drawers slide before any physical interaction takes place. In contrast, existing articulated object reconstruction techniques almost uniformly rely on observable motion cues across multiple distinct articulation configurations. Most methods require paired captures of open and closed states, pre-specified joint counts, or real-time interaction videos to resolve kinematic structures. This motion dependency severely hinders scalable real-world application, as online product catalogs, 3D repositories, and indoor scene scans (e.g., ScanNet and Replica) almost universally capture furniture in static, fully closed rest states.

Inferring articulation mechanisms from closed configurations is an inherently ill-posed inverse problem. Rest-state furniture typically features uniform color textures, flush boundary seams, and repetitive box-like geometries that offer minimal discriminative signal for low-level visual or geometric feature extractors. While foundation models—such as vision-language models and video diffusion models—possess rich world priors and can mentally simulate the opening of a closed drawer, naively chaining their outputs introduces severe failure modes. Vision-language models often miscount hidden parts or propose incomplete hierarchies, segmentation models miss parts across viewpoint shifts, and video diffusion models frequently hallucinate non-existent components or produce non-rigid distortions.

The core insight of this paper is that an explicit surface mesh serves as an ideal spatial anchor for cross-model verification and evidence fusion. Surface connectivity enables rigorous occlusion reasoning, smooth geodesic boundary propagation, and confidence-weighted aggregation that neutralizes multi-view noise. Core idea: Ground noisy foundation model predictions onto an explicit surface mesh, driving an iterative co-refinement loop between vision-language hierarchies and segmentation masks, while treating video diffusion models merely as kinematic hypothesis generators whose trajectories are solved via mesh-constrained rigid-body optimization.

Method

Overall Architecture

Rest2Art takes an initial rest-state 3D mesh (obtained from multi-view 2DGS, single-image 3D generation, or 3D scans) and outputs a collection of volumetric part meshes \(\{M_k\}_{k=1}^K\) alongside their joint kinematic parameters \(\{\Psi_k\}_{k=1}^{K-1}\) (supporting both revolute and prismatic joints), automatically discovering the part count \(K\). The pipeline comprises three primary stages: mesh-grounded part segmentation via iterative co-refinement, articulation estimation via video diffusion hypotheses, and volumetric completion for physical simulation.

In the first stage, rendered multi-view imagery is processed through an iterative mutual-correction loop between a vision-language model (VLM) and SAM3. Validated 2D masks are lifted onto the mesh surface using confidence-weighted evidence accumulation and refined through geodesic graph propagation. In the second stage, transparent part masks are overlaid on the most informative viewpoint to prompt a video diffusion model to generate dynamic articulation hypotheses. 2D trajectories are tracked, unprojected to 3D via rendered metric depth, and fitted to rigid-body joint models under camera projection, where physical penetration penalties and boundary alignments resolve directional ambiguities. Finally, surface shells are thickened and augmented with parametric interior trays to produce simulation-ready solids.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Rest-State Mesh Input<br/>2DGS / Single-image 3D / Scan"] --> B["Hierarchy & Mask Co-Refinement<br/>VLM Tree ↔ SAM3 Masks Dispute Loop"]
    B --> C["Confidence-Weighted Evidence Lifting<br/>Multi-view ray aggregation & face voting"]
    C --> D["Graph-Based Label Propagation<br/>Geodesic shortest-path & reprojection check"]
    D --> E["Articulation Video Synthesis & Tracking<br/>Wan2.2 physical prior & CoTracker3"]
    E --> F["Mesh-Constrained Joint Fitting<br/>Reprojection optimization & penetration penalty"]
    F --> G["Volumetric Mesh Completion<br/>Normal-guided thickening & interior tray"]

Key Designs

1. Hierarchy and Part Mask Co-Refinement: Turning Cross-Model Disagreements into Mutual Corrections

In closed configurations, VLMs frequently miscount parts due to subtle boundaries, while segmentation models lack high-level semantic understanding to cluster functional components. Rather than accepting raw predictions, the framework establishes a bi-directional verification loop between a VLM and SAM3. The VLM proposes an initial part hierarchy tree \(T\), extracting semantic prompts \(S(T) = \{s_1, \dots, s_C\}\) and expected counts \(n(s, T)\). SAM3 generates multi-view masks \(M_{v,s}\). When detected mask counts match the tree expectation (\(|M_{v,s}| = n(s, T)\)), masks are cached as validated evidence. When excess detections occur (\(|M_{v,s}| > n(s, T)\)), it signals an incomplete tree; the excess masks are overlaid onto the image in distinct vivid colors to enhance edge visibility and fed back into the VLM to update \(T\). Missing detections trigger multi-view evidence aggregation before any pruning decision is committed. Constrained to two iterations, this dialogue resolves ambiguities without manual intervention.

2. Confidence-Weighted Evidence Lifting and Graph-Based Refinement: Topological Mesh Aggregation

Directly unprojecting 2D masks onto 3D surfaces leads to severe boundary artifacts, occlusion bleed, and isolated noise. The method ranks views per part \(p\) by combined detection confidence and mask area, selecting the top-\(K\) viewpoints \(V_p\). Using z-buffered rasterization with back-face culling, the face support set maps pixels to visible mesh faces. Evidence for assigning face \(f\) to part \(p\) is accumulated as:

\[E(f, p) = \sum_{v \in V_p} w_p^v \cdot c_p^v \cdot \mathbb{I}[f \in \text{supp}(M_p^v)]\]

where \(w_p^v\) is the normalized view weight and \(c_p^v\) is the detection confidence. Each face initially receives the label \(\ell(f) = \arg\max_p E(f, p)\). To eliminate boundary jaggedness and unassigned regions, connected components are computed on the mesh face graph; the dominant component per label acts as a seed, while fragmented faces form an unassigned pool \(U\). Unassigned vertices are propagated via multi-source shortest paths on the vertex adjacency graph:

\[\ell(u) = \arg\min_p d_p(u), \quad u \in U\]

ensuring that part labels expand along physical surface geometry rather than bridging across free-space gaps. Propagated labels are finally re-verified against 2D masks via back-projection.

3. Video Diffusion Motion Hypotheses and Penetration-Aware Joint Fitting: Grounding Dynamic Priors

Because static observations lack motion cues, the method overlays translucent part masks onto the optimal view to condition Wan2.2 (enhanced with the VBVR physics-reasoning LoRA). A VLM-generated prompt instructs the model to hallucinate opening motions. Crucially, the video is treated solely as a qualitative hypothesis source. CoTracker3 extracts 2D point tracks from the synthesized sequence, which are unprojected into 3D using rest-pose mesh depths. Motion types (revolute vs. prismatic) are categorized using the coefficient of variation (CV) of point displacement magnitudes and track curvature.

Joint parameters—\(\Psi^r = \{\hat{\omega}, q, \theta\}\) for revolute axes and \(\Psi^p = \{\hat{v}, d\}\) for prismatic axes—are fitted by minimizing 2D reprojection errors between rigid-body transformations and observed tracks. Monocular projection ambiguities are eliminated by simulating rotation on the 3D mesh: candidate axes that cause the moving child part to penetrate the stationary parent body are penalized, and rotation vectors \(\hat{\omega}\) are regularized toward the principal directions of adjacent part boundary seams. Joint parameters are independently fitted from two distinct random-seed videos and merged when their motion classifications agree.

Loss & Training

The framework is entirely zero-shot and training-free, operating without fine-tuning underlying models. At test time, kinematic joint parameters are resolved by minimizing robustified trajectory reprojection errors alongside geometric non-penetration penalties:

\[\min_{\Psi} \sum_{t=1}^T \sum_{j=1}^{N_{tr}} \rho\left(\| \pi(K_{cam}, \mathbf{p}_t^j(\Psi)) - \mathbf{x}_t^j \|_2\right) + \lambda_{pen} \mathcal{L}_{penetration}(M_{child}, M_{parent}, \Psi)\]

where \(\mathbf{x}_t^j\) is the 2D tracked location of point \(j\) at frame \(t\), \(\pi\) is the perspective camera projection, \(\rho(\cdot)\) is a robust estimator down-weighting tracking outliers, and \(\mathbf{p}_t^j(\Psi)\) denotes the 3D point position transformed under joint parameters \(\Psi\).

Key Experimental Results

Main Results

Evaluation is conducted on the Articulated Containers Dataset (ACD, comprising HSSD and ABO subsets) and the real-world indoor scanning dataset MultiScan. Metrics include rotation/translation axis angular error (Ang., degrees), axis positional error (Pos., decimeters), and joint type classification accuracy (Type Acc., %).

Table 1: Joint Estimation Comparison on the ACD Benchmark (Paper Table 2)

Method	Task Type	Input State	ACD-HSSD Ang. (°)↓	Pos. (dm)↓	Type (%)↑	ACD-ABO Ang. (°)↓	Pos. (dm)↓	Type (%)↑
URDFormer	Gen.	1 (rest)	14.59	6.52	16.23	13.64	6.11	19.12
Singapo	Gen.	1 (rest)	1.97	3.21	59.42	0.13	2.63	41.47
Singapo + ⟳ (Co-Refine)	Gen.	1 (rest)	2.24	2.85	61.21	0.14	2.33	67.94
ArtGS	Recon.	2 states	53.12	1.37	57.71	46.29	0.96	60.76
REArtGS	Recon.	2 states	58.02	6.95	33.30	59.20	6.77	35.61
REArtGS++	Recon.	2 states	41.82	2.87	59.38	45.92	1.23	81.28
Articulation in Motion	Recon.	video	30.29	N/A	25.09	19.74	0.81	41.90
Articulate AnyMesh	Recon.	1 (rest)	11.35	1.31	67.12	25.65	0.08	38.94
Ours (Rest2Art)	Recon.	1 (rest)	11.35	0.85	74.49	4.78	0.20	73.38

Table 2: Real-World Joint Estimation on MultiScan Dataset (Paper Table 3)

Method	Setting	Scene Coverage Cov. (%)↑	Ang. Error (°)↓	Pos. Error (dm)↓	Type Acc. (%)↑
REArtGS	2-state recon	31.91	51.93	29.60	64.40
REArtGS++	2-state recon	36.17	62.72	6.85	48.74
Articulate AnyMesh	1-state rest	61.70	30.51	3.45	12.63
Ours (Rest2Art)	1-state rest	48.94	17.23	0.26	65.78

Ablation Study

Table 3: Ablation of Co-Refinement Model Combinations on ACD (Paper Table 4)

Architecture Setup	Co-Refinement (⟳)	Tree Acc. T Acc. (%)↑	Part Count Acc. # Parts Acc. (%)↑
GPT-5.2 (VLM alone)	–	62.3	59.4
Qwen3-VL (VLM alone)	–	52.2	47.8
GPT-5.2 + SAM3 (No Co-Refine)	✗	62.3	59.4
Qwen3-VL + SAM3 (No Co-Refine)	✗	52.2	47.8
Qwen3-VL + Grounded-SAM2 (No Co-Refine)	✗	53.6	46.4
GPT-5.2 ⟳ SAM3 (Ours)	✓	72.5 (+10.2)	59.4 (+0.0)
Qwen3-VL ⟳ SAM3 (Ours)	✓	72.5 (+20.3)	63.8 (+16.0)
Qwen3-VL ⟳ Grounded-SAM2	✓	60.9 (+7.3)	47.8 (+1.4)

Table 4: Impact of Mesh Reconstruction Backends on Downstream Tasks (Paper Table 5)

Backend Algorithm	Segmentation IoU ↑	Segmentation mAP ↑	Axis Dir Error (°)↓	Axis Pos Error (dm)↓
3DGS	0.30	0.83	1.69	0.154
SVRaster	0.83	0.96	1.86	0.146
2DGS (Ours Primary)	0.97	0.99	3.67	0.154

Key Findings

• Mutual Correction Dynamics: Unidirectional chaining of foundation models locks in upstream hallucinations. The bidirectional co-refinement loop improves Qwen3-VL's hierarchy accuracy by +20.3% and part count accuracy by +16.0%, showing that visual mask mismatches provide actionable feedback for language grounding.
• Physical Conditioning in Diffusion: Vanilla Wan2.2 generates severe visual hallucinations and non-rigid warping (23.1% hallucination rate). Incorporating VBVR LoRA and transparent mask overlays slashes hallucinations to 11.5% and boosts physically plausible articulation generation to 80.8% on ACD-HSSD.
• Vulnerability of Multi-State Baselines: Two-state reconstruction methods (ArtGS, REArtGS++) fail when fed AI-edited opened images, because slight non-rigid synthesis artifacts trigger false-positive motion on static bases. Grounding joint optimization on a fixed metric mesh provides superior noise resistance.

Highlights & Insights

• Foundation Models as Hypothesis Proposers, Not Direct Regressors: Rather than relying on black-box neural networks to predict exact continuous 3D axis coordinates, the framework uses generative models merely to provide qualitative hypotheses (opening trajectories), delegating quantitative precision to mesh-constrained metric optimization.
• Geodesic Topology as a Noise Filter: Projecting 2D segmentations onto explicit 3D mesh graphs replaces noisy Euclidean heuristics with surface-constrained shortest paths, ensuring part boundaries adhere strictly to functional mechanical seams.

Limitations & Future Work

• Restricted Kinematic Complexity: The formulation currently models single-degree-of-freedom revolute and prismatic joints, without explicit parameterization for multi-linkage chains, bi-fold doors, or compliant flexible joints.
• Heuristic Unobserved Interior Geometry: Interior volumes (such as hollow drawer cavities) are synthesized using parametric templates and normal offsetting, which cannot recover complex unobserved interior dividers or rail mechanisms.

Related Work & Insights

• vs. ArtGS / REArtGS / REArtGS++: These baselines strictly demand physical multi-state observations (open and closed captures); Rest2Art operates entirely from a single static rest-state observation without requiring physical interaction.
• vs. Articulate AnyMesh: While sharing the single-mesh rest-state setting, Articulate AnyMesh chains pretrained models sequentially, compounding errors; Rest2Art introduces bidirectional dispute loops and explicit non-penetration geometry verification.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Pioneers rest-state multi-part articulated reconstruction by fusing video diffusion hypotheses with mesh-grounded geometric fitting]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Comprehensive cross-benchmark evaluation on synthetic ACD and real-world MultiScan, with detailed ablations across models, backends, and video prompts]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear structural formulation, rigorous methodology, restrained mathematical notation, and compelling illustrations]
• Value: ⭐⭐⭐⭐⭐ [Crucial breakthrough for scaling interactive digital twin generation from ubiquitous static 3D models, eCommerce imagery, and indoor scans]

Related Papers

• [ECCV 2026] StructureGS: Structure-aware Gaussian Splatting for Articulated Object Reconstruction
• [CVPR 2026] DICArt: Advancing Category-level Articulated Object Pose Estimation in Discrete State-Spaces
• [CVPR 2026] SCAPO: Self-Supervised Category-Level Articulated Pose Estimation from a Single 3D Observation
• [ICCV 2025] MeshMamba: State Space Models for Articulated 3D Mesh Generation and Reconstruction
• [CVPR 2026] LAM: Language Articulated Object Modelers