Skip to content

Articulat3D: Reconstructing Articulated Digital Twins From Monocular Videos with Geometric and Motion Constraints

Conference: ECCV 2026
arXiv: 2603.11606
Code: Not open-sourced
Area: 3D Vision
Keywords: Articulated Object Reconstruction, Digital Twins, Monocular Video, Kinematic Constraints, 3D Gaussian Splatting

TL;DR

Articulat3D proposes a two-stage optimization framework to reconstruct interactive digital twins of articulated objects from casual monocular videos. The first stage performs structured, part-level motion decomposition on 3D point trajectories using low-dimensional motion bases. The second stage constrains the motion to satisfy physical rigid-body rules using explicit kinematic primitives (axes, pivots, and frame-wise scalars of revolute/prismatic joints). It achieves state-of-the-art (SOTA) accuracy on both synthetic and real-world data without requiring static pre-scans.

Background & Motivation

Digital reconstruction of articulated objects (such as doors, drawers, laptops) is a core problem in computer vision and robotics. Reconstructing interactive digital twins with part geometry, appearance, and joint parameters can be directly applied to AR, robotics simulation, and interactive scene understanding. Among all sensing modalities, monocular video has the greatest potential for scalability—it has extremely low acquisition costs, can be shot casually, and is highly abundant on the internet.

However, existing methods face fundamental scalability bottlenecks. The first category of methods uses feedforward networks to directly predict joint parameters from images. While fast, they rely on external model databases or require fine-tuning on specific datasets, leading to weak generalization. The second and more mainstream category explicitly estimates joint parameters from multi-view, multi-state images. Though offering strong geometric constraints, they require controlled capture environments and precise camera intrinsic/extrinsic parameters, rendering them unsuitable for real-world open scenes. The third category transitions to monocular videos and seems more scalable, but actually still implicitly relies on a static pre-scan of the object (e.g., the camera rotating around the object at the beginning of the video), making them unable to handle in-the-wild video scenarios that exhibit motion from the very beginning.

A more fundamental, yet widely overlooked issue, is that almost all existing methods optimize motion parameters (such as dual quaternions) per discrete state or independently per frame. This frame-by-frame optimization inherently disconnects the temporal sequence—the model fits the appearance at specific timestamps rather than learning the continuous evolution of the motion itself. In reality, articulated motion (e.g., a door opening) follows continuous physical trajectories. Treating each frame as an isolated optimization objective inevitably loses temporal consistency, resulting in a lack of physical plausibility during transition phases.

The key challenge is thus made explicit: monocular videos are cheap but extremely under-constrained, whereas multi-view setups are precise but unscalable; frame-by-frame optimization achieves visual realism but loses physical consistency, while kinematic constraints guarantee physical plausibility but are highly sensitive to initialization. This paper poses the question: Is it possible to reconstruct interactive digital twins solely from casual monocular videos, by enabling the model to understand the geometric constraints that govern articulated motion?

The key insight is to model motion as a continuous, constrained trajectory rather than a set of independent frame-by-frame state snapshots. The core idea is a two-stage optimization: first using motion bases to discover rigid motion groups from low-dimensional structures (solving the initialization problem), and then constraining the motion to rigid transformations of explicit kinematic primitives (solving the physical plausibility problem).

Method

Overall Architecture

The core problem addressed by Articulat3D is to reconstruct interactive digital twins containing part-level geometry, appearance, and joint parameters, given only a casually captured monocular video \(X = \{x_i\}_{i=1}^N\). The pipeline consists of preprocessing and a two-stage optimization. The inputs are raw video frames, and the outputs are 3D Gaussians plus explicit joint parameters (axes, pivots, and frame-wise motion parameters), which can be directly imported into physics simulation environments for interactive operations.

In the preprocessing stage, SAM3 is used to segment the target object, Qwen-Image is used to inpaint hand-occluded areas, and TAPIP3D is applied to extract sparse 3D point trajectories and classify them by motion type (static/revolute/prismatic). In the two-stage optimization, the first stage (motion bases initialization) softly decomposes noisy 3D trajectories into rigid motion groups using a shared SE(3) basis and sparse part assignments. The second stage (kinematic constraint refinement) replaces the loose SE(3) basis with a strict parameterization of kinematic primitives and corrects part boundaries using differentiable re-assignment via Straight-Through Estimators (STE). The entire optimization jointly refines canonical Gaussian parameters, latent assignment vectors, and kinematic parameters.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Monocular Video Frames"] --> B["Preprocessing<br/>SAM3 Segmentation + Qwen Inpainting<br/>TAPIP3D Trajectory Extraction & Classification"]
    B --> C["Motion Bases Modeling<br/>Shared SE(3) Bases · Sparse Assignment<br/>Soft Decomposition into Rigid Motion Groups"]
    C --> D["Kinematic Primitive Parameterization<br/>Axis + Pivot + Frame-wise Scalar<br/>Rodrigues/Translation Rigid Transformations"]
    D --> E["Joint Refinement<br/>STE Differentiable Hard Assignment<br/>Part Boundary Correction"]
    E --> F["Output: Interactive Digital Twin<br/>3D Gaussians + Joint Parameters"]

Key Designs

1. Motion Bases Modeling: Translating Under-constrained Point-wise Tracking into Structured Part-level Motion Decomposition

Direct per-Gaussian trajectory optimization of articulated motion in monocular videos is severely ill-posed—sparse TAPIP3D trajectories are noisy and incomplete, and directly optimizing frame-by-frame displacement for each Gaussian fails to converge. The core insight of this design is that articulated motion naturally resides in a low-dimensional subspace, and the object's motion can be expressed as a linear combination of a few shared SE(3) bases.

Specifically, starting from TAPIP3D trajectories, spatio-temporal K-Means clustering is first performed to obtain \(B\) part-level motion groups (including one identity basis corresponding to the static part). A weighted Procrustes alignment is solved for each group's dynamic trajectory to obtain the initial frame-wise SE(3) basis \(\mathbf{T}_{0\rightarrow t}^{(b)}\). Each 3D Gaussian is assigned a set of learnable coefficients \(\{\mathbf{w}^{(b)}\}_{b=1}^B\). After softmax normalization (\(\sum_b \mathbf{w}^{(b)} = 1\)), its aggregated transformation is a weighted combination of each basis:

\[\mathbf{T}_{0\rightarrow t} = \text{Normalize}\left(\sum_{b=1}^{B} \mathbf{w}^{(b)}\mathbf{T}_{0\rightarrow t}^{(b)}\right)\]

where Normalize guarantees the result is a valid member of SE(3) via SVD orthogonalization. The Gaussian is transformed from the canonical state \((\boldsymbol{\mu}_0, \mathbf{R}_0)\) to its pose at time \(t\):

\[\boldsymbol{\mu}_t = \mathbf{R}_{0\rightarrow t}\boldsymbol{\mu}_0 + \mathbf{t}_{0\rightarrow t}, \quad \mathbf{R}_t = \mathbf{R}_{0\rightarrow t}\mathbf{R}_0\]

The key difference between this design and Shape of Motion is that the latter allows each Gaussian to combine all bases using arbitrary soft mixing weights (leading to redundant and poorly interpretable motion representations). In contrast, Articulat3D guides the assignment towards near-discrete through strong prior initialization and sparse regularization, making the motion representation part-aware from the very beginning. Furthermore, this method decouples the physical articulated motion interval from the static camera panning frames, applying temporal regularization only during physically meaningful motion segments. Consequently, the shared bases force Gaussians with similar semantic properties to move along a consistent low-dimensional trajectory, maintaining spatio-temporal coherence even in regions with sparse tracking data.

2. Kinematic Primitive Parameterization: Constraining Motion with Explicit Joint Axes and Pivots, Seamlessly Transitioning from "Soft Blending" to "Rigid Physics"

Although the motion bases in the first stage are flexible, they lack physical rigor—weighted mixing of SE(3) bases does not guarantee rotation around a fixed axis or translation along a fixed direction. This design replaces the unconstrained SE(3) basis with strict kinematic primitives, forcing each component to follow the rigid-body motion laws of revolute or prismatic joints.

For each object part \(k\), three learnable parameters are defined: a normalized joint axis \(\mathbf{a}_k \in \mathbb{S}^2\), a pivot point \(\mathbf{c}_k \in \mathbb{R}^3\), and a frame-wise motion scalar \(q_k(t)\) (revolute angle or translating displacement). For revolute joints, the rotation transformation is computed using Rodrigues' formula:

\[\mathbf{R}_k(t) = \exp(\theta_k(t)[\mathbf{a}_k]_\times), \quad \mathbf{t}_k(t) = (\mathbf{I} - \mathbf{R}_k(t))\mathbf{c}_k\]

where \(\theta_k(t) = q_k(t)\) is the rotation angle, and \([\mathbf{a}_k]_\times\) is the skew-symmetric matrix of the axis vector. For prismatic joints, the rotation component is the identity matrix, and the motion is pure translation along the axis:

\[\mathbf{R}_k(t) = \mathbf{I}, \quad \mathbf{t}_k(t) = d_k(t)\mathbf{a}_k\]

where \(d_k(t) = q_k(t)\) is the translation displacement.

The bridging from motion bases to kinematic primitives is completed through "robust joint initialization": first, a hard assignment is performed based on the maximum coefficient to determine the assigned part of each Gaussian. Then, a trimmed mean (discarding 20% outliers) is calculated on all Gaussian positions belonging to that part to synthesize a stable center trajectory \(\mathbf{c}_k(t)\). Finally, PCA is performed on the center trajectory—if there is only one significant principal component, it is classified as a prismatic joint (linear motion); if there are two significant principal components, it is classified as a revolute joint (planar arc motion, with the normal vector of the third principal component initializing the joint axis). This purely geometric joint-type determination is simple and effective, requiring no extra classifier training.

3. Joint Refinement and STE Re-assignment: Differentiably Correcting Initial Part Boundaries While Preserving Strict Rigid Transformations

Initial hard assignments based on motion basis coefficients are often unreliable near kinematic boundaries—the motion of boundary Gaussians is influenced by neighboring parts, leading to prone-to-error initial assignments. If these assignments are permanently locked, the errors will be amplified in the kinematic refinement stage. The challenge of this design lies in: strict rigid-body transformation requires each Gaussian to belong to one and only one joint (hard assignment), but correcting assignments requires gradients (soft assignment).

The solution introduces a learnable latent vector \(\mathbf{z}_i \in \mathbb{R}^K\) for each Gaussian, yielding soft assignment probabilities \(p_{i,k}\) via softmax with temperature \(\tau\). In the forward pass, a strict rigid transformation using the most probable joint is applied: \(\mathbf{T}_i = \mathbf{T}_{k^*}\) (\(k^* = \arg\max_k p_{i,k}\)), ensuring physical consistency in the output. In the backward pass, a Straight-Through Estimator (STE) is used to approximate the transformation as a soft blend \(\sum_j p_{i,j}\mathbf{T}_j\) to propagate gradients:

\[\frac{\partial \mathcal{L}}{\partial z_{i,k}} \approx p_{i,k}\sum_{j=1}^{K}\left\langle\frac{\partial \mathcal{L}}{\partial \mathbf{T}_i}, \mathbf{T}_j\right\rangle(\delta_{jk} - p_{i,j})\]

where \(\delta_{jk}\) is the Kronecker delta, and \(\langle\cdot,\cdot\rangle\) is the Frobenius inner product. This "forward-hard, backward-soft" strategy allows the model to maintain strict rigid motion while automatically correcting initial misallocations based on kinematic consistency (Figure 3 illustrates the boundary correction from chaotic to physically plausible), serving as the key technical link between the first-stage soft decomposition and the second-stage rigid constraints.

Loss & Training

The total loss consists of three terms:

\[\mathcal{L} = \mathcal{L}_{\text{render}} + \lambda_{\text{acc}}\mathcal{L}_{\text{acc}} + \lambda_{z}\mathcal{L}_{z}\]

The rendering loss \(\mathcal{L}_{\text{render}}\) combines photometric error and geometric supervision: \(\mathcal{L}_{\text{render}} = (1 - \lambda_{\text{SSIM}})\mathcal{L}_1 + \lambda_{\text{SSIM}}\mathcal{L}_{\text{D-SSIM}} + \lambda_D\mathcal{L}_D\). Depth supervision uses a robust scale-invariant loss \(\mathcal{L}_D = \log(1 + \|\boldsymbol{D} - \bar{\boldsymbol{D}}\|_1)\) between the monocular depth estimation \(\bar{\boldsymbol{D}}\) from MegaSaM and the rendered depth \(\boldsymbol{D}\).

The acceleration loss \(\mathcal{L}_{\text{acc}}\) directly acts on the second-order finite difference of the learnable joint scalars \(q_k(t)\): \(\mathcal{L}_{\text{acc}} = \sum_{t,k} \|q_k(t+1) - 2q_k(t) + q_k(t-1)\|^2\), rendering high-frequency oscillations and encouraging uniform or uniformly accelerated motion. This effectively suppresses jitter in monocular tracking priors.

The depth stabilization loss \(\mathcal{L}_z\) targets the "breathing effect" commonly observed in monocular reconstruction—where the optimizer compensates for color reprojection errors by wrongly adjusting the depth of Gaussians along the camera's optical axis. This loss penalizes Z-axis displacements that cannot be explained by the kinematic model: \(\mathcal{L}_z = \sum_{i=1}^N \sum_{t=1}^T \omega_i \cdot \|\Delta\mathbf{x}_{i,t}^{(z)}\|^2\), where \(\omega_i\) is the confidence strength weight of the first-stage motion prior, forcing the model to explain observations via rotation and lateral translation rather than non-physical pulsations along the depth direction.

Key Experimental Results

Main Results

Fully evaluated on three datasets: Video2Articulation-S (73 synthetic videos, 11 categories of single-moving-part objects), Articulat3D-Sim (newly introduced here, 20 videos, 17 categories, 2-7 moving parts, synchronized articulation + large camera motion), and Articulat3D-Real (newly introduced here, 12 iPhone-recorded real videos, 9 categories, diverse lighting and backgrounds). Evaluation dimensions include joint estimation (axis error, position error), reconstruction accuracy (Chamfer Distance overall/moving part/static part), 3D tracking (EPE), and view synthesis (PSNR/SSIM/LPIPS).

Method Axis Error ↓ Position Error ↓ CD-w ↓ CD-m ↓ CD-s ↓ EPE ↓ PSNR ↑
Articulat3D-Sim Dataset
RSRD 55.75 91.52 69.16 55.41 10.05 3.14 28.19
Articulate Anything 44.65 129.00 16.10 17.74 16.36 2.02 N/A
iTACO 48.50 38.00 3.13 1.63 3.72 0.14 N/A
Shape of Motion N/A N/A N/A N/A N/A 0.08 33.37
Articulat3D 0.53 0.65 0.76 0.84 0.85 0.04 37.80
Video2Articulation-S Dataset
iTACO 16.05 13.00 1.00 13.00 6.00 N/A N/A
Articulat3D 1.60 1.83 0.82 1.12 1.82 N/A 35.91
Articulat3D-Real Dataset
RSRD N/A N/A N/A N/A N/A N/A 16.12
Shape of Motion N/A N/A N/A N/A N/A N/A 24.13
Articulat3D N/A N/A N/A N/A N/A N/A 26.73

Articulat3D consistently leads across all metrics on all datasets. On the most challenging Articulat3D-Sim (multi-part scenes), the joint axis error is only 0.53 degrees (compared to iTACO's 48.50 degrees, which fails entirely on multi-part objects), and the position error is 0.65 cm (compared to RSRD's 91.52 cm, where template mismatch causes structural collapse). In real scenes, the PSNR reaches 26.73, significantly outperforming RSRD (16.12), which relies on static pre-scans. Notably, Articulat3D requires no static scans or controlled camera poses, enabling reconstruction directly from the first frame of the video.

Ablation Study

The following ablations were all conducted on the Articulat3D-Sim dataset:

Configuration Axis Error ↓ Position Error ↓ CD-w ↓ CD-m ↓ EPE ↓ PSNR ↑ Description
Full Articulat3D 0.53 0.65 0.76 0.84 0.04 37.80 Full model
w/o motion prior (No 3D trajectory guidance) 43.42 31.26 18.78 92.45 28.29 20.78 No motion prior, model collapses
w/o prior init (Skip motion initialization) 1.36 3.42 1.13 1.67 0.06 30.89 Directly optimizing kinematic parameters gets stuck in local minima
w/o kinem. const. (No kinematic constraints) 0.86 4.98 5.19 7.71 0.09 35.45 No rigid constraints leads to geometric drift
w/ track noise (Add noise to trajectory) 0.56 0.67 0.78 0.88 0.05 37.62 Robust to tracking noise
w/ less data (Only 1/3 of the frames) 0.61 0.68 0.83 0.92 0.07 34.60 Kinematic accuracy is nearly unchanged with sparse data

Key Findings

  • Motion prior is the foundation and cannot be omitted: Removing the 3D trajectory guidance leads to complete model collapse (position error jumps from 0.65 to 31.26, and CD-m leaps from 0.84 to 92.45), indicating that photometric constraints alone cannot disentangle geometry and kinematics, making the motion prior crucial for resolving this ambiguity.
  • The two stages each perform distinct and indispensable roles: Skipping the motion initialization and directly optimizing kinematic parameters deteriorates the axis error to 1.36 (a 2.5x degradation) because the optimization space for kinematics is highly non-convex; a poor initialization directly leads to erroneous local minima. Removing joint-kinematic constraints causes CD-m to jump to 7.71 (nearly a 10x degradation). Though rendering quality decreases slightly (PSNR 35.45 vs 37.80), the geometry drifts severely—kinematic constraints are the critical bottleneck for guaranteeing physical plausibility.
  • Kinematic primitives are naturally noise-resistant: Injecting Gaussian noise into the 3D trajectories leaves almost all metrics unchanged (axis error only shifts from 0.53 to 0.56) because explicit axis-angle parameterization forms a rigid kinematic bottleneck that only permits physically valid motions, automatically filtering out noise that does not comply with rigid body kinematics.
  • Kinematic accuracy is maintained with only 1/3 of the frames: Axis error only increases from 0.53 to 0.61 degrees, and position error from 0.65 to 0.68 cm, proving that the motion bases can still effectively capture the low-dimensional structure of articulated motion from sparse temporal data. The drop in rendering quality (PSNR 37.80 to 34.60) is due to fewer photometric constraints available for Gaussian optimization, rather than issues with the motion model itself.

Highlights & Insights

  • The motive logic of the two-stage progressive design is highly coherent: Motion bases solve "what rigid parts compose this object" (decomposition problem), while kinematic joints solve "how these parts physically move" (constraint problem). The two stages complement each other naturally—the flexibility of the bases gives the second stage a good start, and the rigidity of the primitives provides a physical exit for the first stage. This "loose-first, tight-later" strategy is highly transferable to other ill-posed reconstruction problems requiring a transition from weak to strong constraints.
  • PCA-based joint type determination is an elegant heuristic: Without training any neural network, the joint type (revolute vs. prismatic) is determined purely geometrically from the principal components of trajectories—one component indicates translation, while two components indicate rotation (planar arc). This is extremely elegant and effective in practice. The design philosophy of "using simple geometry instead of introducing learning modules where possible" is highly commendable.
  • The forward-hard, backward-soft STE trick is very practical: In scenarios requiring discrete decisions alongside differentiable optimization, this pattern—using argmax in the forward pass to guarantee correct behavior and softmax in the backward pass to ensure gradient flow—can be replicated across many tasks (such as dynamic routing, adaptive module selection, part-level manipulation boundary optimization, etc.).
  • The depth stabilization loss is highly targeted: The "breathing effect" along the Z-axis is a pervasive artifact in monocular reconstruction, but few specify a loss to address it. Directly penalizing Z-axis displacements that cannot be explained by the kinematic model offers a low-cost solution to depth ambiguity, which can be transferred to other 3DGS-based monocular dynamic reconstruction tasks.

Limitations & Future Work

  • The authors acknowledge that texture-less surfaces cause unreliable sparse 3D trajectories, which in turn degrades initialization quality. This is a common bottleneck for TAPIP3D-like methods—sparse tracking inherently relies on texture features, and shiny or uniform objects lack trackable keypoints.
  • The current implementation assumes independent joints and does not support hierarchical kinematic chains (such as multi-stage nested robotic arms), which limits its modeling capability for complex assemblies. A future direction is introducing tree-structured kinematic models to constrain nested relationships via parent-child joint formulations.
  • Dynamics parameters (mass, friction coefficients) are not modeled. The reconstructed digital twins can only perform kinematic replay in physical simulations but cannot simulate true force-motion relationships. Future work could explore joint optimization to estimate physical parameters from video.
  • The "kinematic primitive parameterization" in the experiments requires knowing the number of parts \(K\) in advance (or obtaining it from prior clustering), but inferring the number of parts for unknown object categories remains an unsolved sub-problem.
  • Quantitative evaluation in real-world scenes heavily relies on rendering metrics, as ground-truth kinematic parameters are difficult to capture (requiring manual annotations), leaving the evaluation system somewhat incomplete.
  • vs Shape of Motion (Wang 2025): Also a 4D-GS-based dynamic reconstruction method, Shape of Motion represents motion using general free motion bases and soft blending weights without distinguishing between articulated structures and free deformations. Articulat3D introduces two key upgrades: pushing the soft blending towards sparse, near-discrete weightings (part-aware initialization) and replacing free bases with explicit kinematic primitives. The distinction is that the former treats all motions as general displacements, whereas the latter recognizes that articulated motion has a specific structure and should be governed by specialized constraints.
  • vs VideoArtGS / RSRD: These methods require a static scanning segment at the beginning of the video to initialize 3D geometry. Articulat3D works directly on videos without static segments using motion bases, significantly lowering the barrier to data collection. The core difference is that the former treats "acquiring geometry first" and "estimating motion second" as two isolated steps, while the latter jointly optimizes them conditioned on each other.
  • vs Articulate Anything (Le 2024): As a method relying on retrieval from large-scale mesh databases, its generalization capability is limited by the database coverage. Articulat3D does not rely on any external model database and generalizes well to unseen categories, which is a fundamental advantage of reconstruction methods over retrieval methods.
  • Insight: The "soft decomposition, then hard constraint" two-stage paradigm of this paper can be extended to other reconstruction-constraint problems, such as reconstructing physical parameters (elastic modulus, damping) of deformable objects from video—first obtaining an initial deformation field in a data-driven way, and then refining it with physical equation constraints.

Rating

  • Novelty: ⭐⭐⭐⭐ The combination of a progressive two-stage design and explicit parameterization of kinematic primitives is innovative; while motion bases are inspired by Shape of Motion and kinematic primitives originate from robotics, combining both and adapting them to the 3DGS framework for monocular articulated reconstruction is novel.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Three datasets covering simple synthetic, complex synthetic, and real-world scenes; four evaluation metrics; complete ablation study (with tests analyzing "what happens if a component is removed"); plus robustness to noise, data sparseness, hyperparameter sensitivity, and runtime comparisons, backed by a solid appendix.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clearly structured (a linear progression from motivation, to method, to experiments), strong articulation of core challenges, complete methodological details (including mathematical derivations and technical motivations), and rich, visually convincing figure comparisons.
  • Value: ⭐⭐⭐⭐ Lowering the barrier to creating articulated digital twins from controlled multi-view capture to casual monocular video is of significant scaling value. Its current value lies primarily at the academic advancement and framework level; industrial adoption will require addressing hierarchical kinematic chains and estimating physical parameters.