MonoArt: Progressive Structural Reasoning for Monocular Articulated 3D Reconstruction

Conference: ECCV2026
Paper: ECCV paper page
PDF: Full paper
Project: MonoArt
Area: 3D Vision
Keywords: monocular articulated reconstruction, part-aware representations, dual-query decoding, joint estimation, kinematic trees

TL;DR

MonoArt organizes single-image articulated reconstruction into geometry generation, part representation, dual-query motion decoding, and kinematic estimation, achieving an F-Score of 0.670 and joint type accuracy of 67.47% on the 46-category PartNet-Mobility setting, although its axis direction error is not the best.

Background & Motivation

Generating a three-dimensional model that resembles a cabinet photograph is not equivalent to knowing how its door opens. An interactive asset must also distinguish the door from the body, locate the rotation axis, determine motion limits, and connect the parts into the correct hierarchy. These quantities constrain one another: an incorrect door segmentation affects joint localization, while a displaced axis makes an initially correct surface move incorrectly when opened. Multi-state or multi-view methods can use observed motion to identify these relationships, but a single RGB image provides no such direct evidence. Retrieval and assembly compensate for hidden structure with asset libraries, yet retrieved parts may mismatch the instance in geometry or texture.

Another route obtains motion cues from video generation, vision-language models, or predefined motion directions. The paper argues that these external cues increase computation and system dependencies, or restrict recognizable motion to handcrafted priors. Directly regressing every joint parameter from image features is also difficult because appearance, part identity, and spatial location remain entangled. MonoArt instead turns the image into a spatially addressable three-dimensional representation before learning part relationships within that representation. Here, structural reasoning means structured representation refinement inside a neural network, not a large language model producing a textual chain-of-thought.

The system first uses frozen TRELLIS to recover a mesh, then trains dedicated part and motion modules with ground-truth three-dimensional part annotations. It treats identifying a coherent part as a prerequisite for predicting its motion, rather than forcing both tasks into the same image features. This does not remove the need for learned priors or all language-derived features: the decoder still uses frozen CLIP text prototypes. Core Idea: establish geometry-aligned part representations first, then separately refine part semantics and spatial anchors so that joint regression depends on explicit structure rather than directly guessing motion from appearance.

Method

Overall Architecture

The input is a single RGB image; the output is a three-dimensional object with part masks, joint parameters, and a kinematic tree. TRELLIS Geometry Generation supplies a mesh and sparse voxel latents, which Part-Aware Semantic Reasoning transforms into structural representations of surface points. Dual-Query Motion Decoding maintains both three-dimensional positions and part content, and Kinematic Estimation converts these embeddings into usable joints and parent-child relationships. These are consecutive modules within one architecture, not an external inference workflow calling a video generator, retrieval library, or VLM agent.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Single RGB image"] --> Geometry["TRELLIS Geometry Generation"]
    Geometry --> Semantic["Part-Aware Semantic Reasoning"]
    Semantic --> Motion["Dual-Query Motion Decoding"]
    Geometry -->|Geometry features| Motion
    Motion --> Kinematics["Kinematic Estimation"]
    Semantic -->|Part point features| Kinematics
    Kinematics --> Output["Mesh, parts, joints,<br/>and kinematic tree"]

Solid arrows indicate inference data flow; three-dimensional part annotations, object categories, and matching-derived supervision are used only during training. Frozen CLIP text prototypes are incorporated inside Dual-Query Motion Decoding to improve part semantic consistency, not to replace geometric reasoning. The paper's canonical geometry is a shared reference shape for downstream processing; it should not automatically be interpreted as placing every object in the same closed joint state.

Key Designs

1. TRELLIS Geometry Generation: give motion prediction a spatially traceable representation

Frozen TRELLIS generates structured sparse voxel latents from the input image, then its mesh decoder produces an explicit mesh. Active voxels store features tied to spatial locations, preserving more spatial structure than a single global image vector. MonoArt uses a voxel grid resolution of \(64\) and samples \(100{,}000\) points on the generated mesh surface. Each surface point is mapped to continuous voxel coordinates and receives geometry-aligned features through trilinear interpolation over eight neighboring voxels. The resulting \(F_{\text{geo}}\) ties subsequent segmentation to the surface: a query identifying a part can also identify its constituent points. This step borrows the generator's prior over hidden shapes, but does not eliminate single-image occlusion ambiguity.

Freezing the backbone means that the main learning task is to interpret existing geometry, rather than retrain a large single-image three-dimensional generator. It also exposes an entry point for error propagation: if the generated mesh omits a handle, downstream queries cannot reliably segment a surface that does not exist. Geometry generation is therefore not the main novel module, but it provides a foundation and one constraint on the system's eventual performance.

2. Part-Aware Semantic Reasoning: turn local geometry into part-discriminative point representations

Local geometric features alone may not identify distant surfaces as belonging to one door, or separate adjacent components that can move independently. The model first projects point features onto the orthogonal XY, YZ, and ZX planes, accumulating information onto two-dimensional grids through bilinear interpolation. Each plane has resolution \(128\times128\), with an initial feature dimension of \(8\). The planes are flattened into tokens, processed by a multi-layer self-attention Transformer for cross-plane and global interaction, and reshaped back into triplane feature maps. Each original surface point then queries its positions on the three planes; the sampled features are concatenated and passed through an MLP to produce a \(448\)-dimensional part point embedding \(H\). A point consequently retains a local positional basis while incorporating structural context from the whole object.

During training, three-dimensional part annotations provide triplet supervision that separates embeddings of the same and different parts. Rather than merely asking each point to output a category label, this supervision shapes the feature space that downstream queries will match against. That distinction matters because the final part masks are produced from affinities between content queries and these point embeddings. In Table 2, replacing triplet supervision with cross-entropy reduces type accuracy from 67.47% to 57.74%, supporting the importance of how part representations are supervised. This remains an ablation result for the reported training setup, not evidence that triplet supervision universally outperforms cross-entropy in three-dimensional segmentation.

3. Dual-Query Motion Decoding: maintain part content and spatial anchors separately

A part query must identify both what a component is and where it lies, and these questions benefit from different representations. MonoArt maintains a \(448\)-dimensional content query and a three-dimensional position query for each candidate, using \(100\) query pairs in total. Initial queries are derived from global pooling and concatenation of \(H\) and \(F_{\text{geo}}\); the initialization branch also predicts the object category. This introduces instance context earlier than random initialization, but \(100\) is a candidate upper bound rather than a required number of parts per object.

Six refinement blocks then alternate inter-query self-attention with cross-attention to point features \(H\). Self-attention allows candidate parts to interact, while cross-attention lets them revisit spatially grounded structural evidence. Both position and content branches use residual updates, correcting spatial anchors and semantic hypotheses rather than predicting them afresh at every layer. Content queries produce part category logits, which retrieve frozen CLIP text embedding prototypes for fusion into the content branch. The accurate characterization is therefore independence from an external VLM reasoning workflow, not complete absence of language-pretrained features.

Each final content query also predicts confidence to handle redundant candidates. During training, predicted masks undergo Hungarian matching to ground-truth parts; matched queries receive confidence supervision proportional to mask overlap, while unmatched queries have a zero target. At inference time, queries below a confidence threshold are discarded, allowing the retained part count to depend on the input. The available main text does not specify this threshold, so it should not be replaced with an unsupported default constant.

4. Kinematic Estimation: convert part hypotheses into joints and a valid hierarchy

Final content queries compute affinities with all point features, producing soft surface-point assignments for each query and thus part masks. The position query is interpreted here as the part centroid; it is not itself the joint pivot, since a door's center generally does not lie on its hinge. The regressor combines position, content, and part point features, then uses lightweight MLP heads to predict joint type, unit axis direction, pivot, and motion limits. Types include fixed, revolute, prismatic, and continuous, while limits are parameterized by a center and a symmetric span. To anchor the pivot to an already localized part, the model predicts an offset from the centroid instead of an absolute pivot directly (page 13):

\[ m_o = Q_p^L + \Delta_o. \]

This lets the door center and hinge location differ while reducing the burden of searching for a pivot throughout unconstrained three-dimensional space. Table 4 separately removes the centroid residual and \(H\), showing that both spatial anchoring and part evidence contribute to parameter estimation. The kinematic tree predictor reads part category distributions and scores ordered part pairs using a learnable category compatibility matrix. It normalizes over candidate parents, selects parents, and enforces single-root and cycle-free constraints during construction; a learnable root may also be included among the candidates. These constraints ensure structural validity, not verified collision behavior, mass distributions, friction, or dynamics. The main text does not fully specify how constraint conflicts are resolved, so no particular graph optimization implementation is assumed here.

A Worked Example

Consider a photograph of a cabinet with its door partially open as an explanatory example: the goal is an asset whose door can open, not a textual reasoning trace. TRELLIS first completes the cabinet and door mesh, and surface sampling converts these shapes into geometric point features. Part-Aware Semantic Reasoning makes points on the same door easier to group in feature space and distinguish from the body. After \(100\) query pairs pass through six refinement layers, confidence filtering retains valid part hypotheses and removes redundant candidates. The door's content query produces its mask, its position query locates the centroid, and the estimator predicts the centroid-to-hinge offset, rotation axis, and motion limits. The tree predictor connects the door to an appropriate parent, allowing joint-based motion instead of treating the entire cabinet as one rotating rigid body. This example explains data flow; the paper does not provide a retained query count or opening angle for this particular cabinet, and none is invented here.

Loss & Training

Training has four phases rather than jointly optimizing every module from initialization (page 9). First, triplet supervision warms up Part-Aware Semantic Reasoning so that point representations become part-discriminative. Second, that module is frozen while object category supervision trains the dual-query initialization branch. Third, Part-Aware Semantic Reasoning, Dual-Query Motion Decoding, and the articulation parameter regressor are jointly optimized. Fourth, preceding modules are frozen and only the kinematic tree predictor is trained to learn parent-child relationships. The TRELLIS backbone and CLIP text embeddings remain frozen; three-dimensional part annotations and matching supervision are not test inputs. The paper refers additional loss settings and training hyperparameters to an appendix, but the supplied full text ends after the references and does not include that appendix. Consequently, this note does not invent a complete total loss, loss weights, learning rate, training duration, or training hardware budget.

Key Experimental Results

Main Results

The benchmark is PartNet-Mobility, with approximately 2K objects annotated with parts and joints, evaluated separately on 7 categories and all 46 categories. Geometric evaluation uniformly samples six states along predicted motion ranges, generates meshes, aligns them to ground truth, and averages across states. CD and F-Score use \(100{,}000\) surface points per mesh, with an F-Score threshold of \(0.05\). Appearance metrics use ten random viewpoints per state, giving sixty rendered images; CLIP uses ViT-L/14@336px. Kinematic evaluation first establishes a bipartite matching between predicted and ground-truth parts, then measures type correctness, axis direction error, and pivot distance error. The cached axis and pivot error equations are damaged by extraction; the original metric names and units are retained without silently replacing the pivot metric with a Euclidean point-distance formula.

The following results come from Table 1, page 9; CD is reported at the original table's \(\times10^{-2}\) scale, and pivot error uses normalized object coordinates.

Setting	Method	CD ↓	F-Score ↑	PSNR ↑	CLIP ↑	Type accuracy % ↑	Axis error rad ↓	Pivot error ↓
7 categories	URDFormer	4.73	0.275	12.43	0.845	35.22	1.324	0.404
7 categories	SINGAPO	1.26	0.572	15.22	0.870	77.12	0.493	0.201
7 categories	MonoArt	0.77	0.728	17.55	0.926	88.26	0.209	0.085
46 categories	ArtAny	2.07	0.514	16.44	0.866	43.32	0.440	0.347
46 categories	PhysXGen	3.06	0.501	16.38	0.859	46.82	0.941	0.208
46 categories	PhysXAny	1.88	0.531	17.07	0.880	63.35	0.289	0.173
46 categories	MonoArt	1.25	0.670	18.55	0.907	67.47	0.423	0.108

ArtAny and PhysXAny denote Articulate-Anything and PhysX-Anything, respectively; ground-truth test shapes were excluded from retrieval baselines' databases. The 7-category and 46-category rows are not rankings at equal difficulty; in the 46-category setting, MonoArt's axis error of 0.423 exceeds PhysXAny's 0.289.

Ablation Study

The following results come from Table 2, page 12; all configurations use the 46-category PartNet-Mobility setting, with the same units as the main table.

Part-Aware Semantic Reasoning	Part supervision	CD ↓	F-Score ↑	PSNR ↑	Type accuracy % ↑	Axis error rad ↓	Pivot error ↓
Disabled	None	1.74	0.626	17.96	24.72	0.549	0.237
Enabled	None	1.63	0.643	17.74	41.60	0.922	0.323
Enabled	Cross-entropy	1.49	0.648	17.71	57.74	1.029	0.302
Enabled	Triplet	1.25	0.670	18.55	67.47	0.423	0.108

The following subset comes from Table 3, page 12, also using 46 categories; every listed configuration applies residual updates to both position and content branches.

Query initialization	Refinement layers	F-Score ↑	Type accuracy % ↑	Axis error rad ↓	Pivot error ↓
Random	6	0.622	44.06	0.472	0.329
Object context	0	0.652	62.65	0.640	0.186
Object context	6	0.670	67.47	0.423	0.108
Object context	9	0.659	66.81	0.475	0.161

Key Findings

• Without part supervision, adding the reasoner does not improve every motion metric: axis error rises from 0.549 to 0.922, showing that structural capacity alone does not guarantee useful motion representations.
• Six layers outperform nine, so more refinement is not monotonically beneficial; the main text does not establish the specific cause of degradation.
• Page 14 reports a single A6000, excluding I/O and averaging 100 runs: MonoArt takes 20.5 seconds, of which TRELLIS uses 18.2 seconds; SINGAPO takes 19.6 seconds, so MonoArt is not absolutely the fastest.
• The twenty-participant study on page 11 separately rates geometry and motion quality: MonoArt scores 4.63 / 4.37 versus PhysX-Anything's 3.34 / 3.12; these are subjective ratings, not physical joint measurements.

Highlights & Insights

• The representation space matches the output mechanism: triplet-trained part point embeddings are used directly for query matching rather than serving only as hidden features for an auxiliary classifier.
• Spatial anchors serve two roles, supporting iterative localization and providing a reference for residual pivot regression, so part localization and joint localization share geometric evidence.
• A transferable reader insight is to learn stable entity representations before predicting relationships between entities; this does not imply that copying the reported parameters solves other physical tasks.

Limitations & Future Work

• The authors note that uniform sampling assigns too few points to tiny components such as buttons on large objects, causing weak features, over-smoothing, and difficult motion estimation.
• They also acknowledge that learned part-whole priors may not cover novel topologies or rare motion patterns; axes and limits can be wrong even when segmentation remains reasonable.
• In terms of evidence, the robotics application demonstrates Franka manipulation in IsaacSim, while the scene application uses existing instance masks and 6D poses; neither establishes a systematic real-robot success rate.
• Reader-proposed extensions include adaptive sampling for small parts and structural uncertainty for novel topologies; neither proposal is evaluated in the paper.
• Damaged cached equations and the missing appendix constrain precise reproduction; this note does not guess missing loss settings or graph-constraint implementations.

Related Work & Insights

• TRELLIS, original reference [60]: supplies structured three-dimensional latents and the generation backbone; MonoArt focuses on converting them into part and kinematic outputs rather than replacing its generation capabilities.
• SINGAPO, original reference [31]: targets single-image controlled generation of articulated parts and is compared as a retrieval-based baseline; runtime is similar, but geometry and joint metrics differ substantially on 7 categories.
• Articulate-Anything and PhysX-Anything, original references [23] and [3]: use vision-language priors to model structure; MonoArt instead emphasizes representation learning with three-dimensional supervision while retaining CLIP semantic prototypes.
• PARIS, original reference [32]: aligns reconstructions across articulation states to estimate parts and motion; MonoArt reduces test input requirements at the cost of greater reliance on learned structural priors.
• These numbers refer to the linked paper's own bibliography and do not imply independent verification of every cited work's experiments or implementation.

Rating

• Novelty: 4/5. The contribution combines part structural supervision, dual-query refinement, and joint anchoring rather than introducing a new foundational three-dimensional generator.
• Experimental Thoroughness: 4/5. Two category settings, module ablations, and an in-the-wild user study are included, but quantitative evidence for real robotics and novel topologies is limited.
• Writing Quality: 4/5. The architecture is clear, but broad speed claims need qualification against the numbers, and the available cache cannot recover all equations.
• Value: 4/5. The method is practically relevant to generating operable three-dimensional assets from one image, subject to the geometry backbone and training topology distribution.