title: >- [Paper Note] MotionDreamer: Universal Skeletal Motion Generation for 3D Rigged Shapes description: >- [ECCV 2026][video_generation][Skeletal Motion Generation] A topology-agnostic diffusion framework utilizing skinning-aware semantic injection and bidirectional video-skeleton fusion to synthesize category-agnostic skeletal animation from monocular driving videos. tags: - ECCV 2026 - video_generation - 3d_vision - Skeletal Motion Generation - Diffusion Model - Motion Transfer date: 2026-09-19 content_hash: 090008b58a100fe3
MotionDreamer: Universal Skeletal Motion Generation for 3D Rigged Shapes¶
Conference: ECCV 2026
Paper: ECCV Official Link
Project Page: https://research.davytao.me/skelmo/
Cached File: ../paper_cache/ECCV2026/eccv-4359.txt
Area: Video Generation / 3D Vision
Keywords: Skeletal Animation Generation, Topology-Agnostic Diffusion Model, Skinning-Aware Semantic Injection, Bidirectional Video-Skeleton Fusion, Cross-Species Motion Transfer
TL;DR¶
Addressing the morphological restrictions of parametric templates and the computational bottleneck of per-instance optimization, MotionDreamer (termed SkelMo in the paper) introduces a curated dynamic dataset of 25,200 diverse rigged animations and a topology-agnostic diffusion framework with skinning-aware semantic injection and bidirectional video-skeleton fusion, synthesizing high-fidelity 3D skeletal motions across arbitrary unseen morphologies from monocular driving videos.
Background & Motivation¶
Synthesizing high-quality, anatomically plausible motions for arbitrary rigged 3D shapes is an essential prerequisite for scalable 4D asset production, immersive interactive entertainment, and embodied robotic manipulation. Historically, generating realistic dynamic sequences has predominantly relied on either expensive motion capture studios—which depend on specialized markers and calibrated hardware restricted almost exclusively to human subjects—or labor-intensive manual keyframing by skilled animators, a bottleneck incapable of scaling to vast open-world virtual characters. With the recent renaissance in text-to-video and image-to-video foundation models, lifting motion patterns from monocular videos has emerged as an attractive paradigm. Nevertheless, existing pipelines remain largely confined to parametric human or quadruped templates (such as SMPL and SMAL), rendering them incapable of adapting to stylized characters, multi-legged creatures, or fantastical creatures. Meanwhile, per-case optimization schemes (e.g., Puppeteer) suffer from severe computational overhead, vulnerability to local minima, and sensitivity to 2D-to-3D depth ambiguities; conversely, feed-forward vertex-trajectory 4D synthesis models (e.g., Motion 3-to-4, ActionMesh) predict unconstrained vertex displacements without skeletal structure, frequently inducing severe surface distortion, mesh adhesion, and unnatural "melting" artifacts.
The core tension stems from the fundamental kinematic discrepancy between 2D monocular appearance dynamics and heterogeneous 3D skeletal topologies: without a pre-shared anatomical template, how can a network reliably establish a semantic correspondence and motion mapping between uncalibrated 2D video pixels and arbitrary 3D joint hierarchies? Furthermore, open-source 3D repositories severely lack high-fidelity 3D assets that concurrently feature clean geometry, complete skeletal rigging, and rich, dynamic animation sequences, creating a severe data bottleneck that hinders the training of universal motion generators.
This work addresses these challenges by moving away from template fitting and optimization toward a data-driven, category-agnostic conditional diffusion paradigm. The authors curate a dynamic dataset of approximately 25,200 high-quality 3D animations and integrate surface visual semantics directly into skeletal representations via skinning weights, while introducing a bidirectional cross-attention bridge between driving videos and target skeletons. Core idea: by leveraging a newly curated foundation dataset of 25,200 cross-category animations, the paper introduces a topology-agnostic diffusion framework that maps multi-view DINOv2 surface semantics directly into skeletal joints via normalized skinning weights and coordinates dynamic cues through bidirectional video-skeleton fusion, achieving high-fidelity, distortion-free motion generation and cross-species retargeting under a global Cartesian diffusion space.
Method¶
Overall Architecture¶
Given a category-agnostic 3D mesh \(\mathcal{M} = \{\mathcal{V}, \mathcal{F}\}\), its associated rest-pose skeleton \(\mathcal{S}_0 = \{\mathcal{P}_0, \mathcal{G}\}\) with joint coordinates \(\mathcal{P}_0 \in \mathbb{R}^{J \times 3}\) and topological hierarchy \(\mathcal{G}\), along with a monocular driving video \(\mathcal{X} = \{I_l\}_{l=1}^L\), MotionDreamer aims to generate a temporally coherent sequence of posed 3D skeletons \(\{\mathcal{S}_l = (\mathcal{P}_l, \mathcal{G})\}_{l=1}^L\) mirroring the motion in \(\mathcal{X}\).
The generative pipeline is built upon a topology-agnostic diffusion transformer. At diffusion step \(t\), the driving video \(\mathcal{X}\) is encoded by a frozen DINOv2 backbone and transformed via spatial-temporal self-attention into global dynamic features; concurrently, the rest-pose skeleton \(\mathcal{S}_0\) is projected into latent embedding \(z_0\) and temporally concatenated with the noisy motion latents \(z_t\) to serve as an invariant structural prior. The denoising network comprises \(M\) transformer blocks that iteratively perform skeletal attention, temporal self-attention, bidirectional video-skeleton fusion, and skinning-aware semantic cross-attention. Finally, the denoised latent \(\hat{z}_0\) is projected back to Cartesian space and passed through a 150-step Inverse Kinematics (IK) optimization to eliminate bone stretching and preserve rigid anatomical articulation.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: 3D Mesh M, Rigged Skeleton S0, and Driving Video X"] --> B["Multi-View Rendering & DINOv2 Back-projection to Vertices"]
B --> C["Skinning-Aware Texture-Semantic Injection<br/>Normalized skinning weights aggregate vertex features to joints"]
A --> D["Spatial-Temporal Video Encoder<br/>DINOv2 + spatial-temporal self-attention for dynamic features"]
C --> E["Bidirectional Video-Skeleton Cross-Modal Interaction<br/>Skeletal query S→X and video topology query X→S"]
D --> E
E --> F["Global Cartesian Topology-Agnostic Diffusion<br/>Skeletal tree-distance attention + Rest-pose temporal concatenation"]
F --> G["Inverse Kinematics IK Refinement<br/>150 iterations projection to eliminate bone stretching"]
G --> H["Output: Posed 3D Skeletal Animation Sequence {Pl}"]
Key Designs¶
1. Skinning-Aware Texture-Semantic Injection: Bridging Surface Appearance and Joint Latents
Raw joint coordinates and topological graphs convey only abstract spatial connectivity while completely lacking semantic contextualization. Consequently, the synthesis backbone struggles to associate motion cues in the driving video with corresponding anatomical parts (e.g., distinguishing a left forelimb from a right forelimb, or locating extra appendages on fantastical creatures).
This module injects fine-grained surface semantic context into the skeletal space by rendering \(N\) multi-view images \(\{I_i^0\}_{i=1}^N\) of mesh \(\mathcal{M}\) from canonical viewpoints and extracting dense descriptors \(F_i\) using a pre-trained DINOv2 model. These descriptors are back-projected onto unoccluded surface vertices using camera projection matrix \(\pi_i\) and visibility mask \(\text{Vis}(v, i)\), followed by multi-view average pooling to obtain vertex feature \(f_v\):
To compress thousands of vertex features into \(J\) skeletal joints without hand-crafted heuristics, the method leverages the model's existing linear blend skinning (LBS) weight matrix \(W \in \mathbb{R}^{J \times |\mathcal{V}|}\). For each joint \(j\), weights across all vertices are normalized to partition unity:
The aggregated joint semantic feature matrix \(\tilde{F}_{\mathcal{S}} \in \mathbb{R}^{J \times C}\) is obtained via linear projection:
Within the diffusion transformer blocks, skeletal latents act as queries to attend to keys and values formed by \(\tilde{F}_{\mathcal{S}}\) via cross-attention. Because skinning weights directly reflect physical deformation relationships, each joint naturally acquires a localized visual profile of its driven surface region, resolving the semantic cold-start dilemma for arbitrary unseen morphologies.
2. Bidirectional Video-Skeleton Fusion: Symmetrical Motion-Topology Coordination
When transferring 2D video dynamics to a 3D skeleton, naive unidirectional conditioning causes the skeleton to indiscriminately absorb 2D perspective artifacts or fail to constrain motion magnitude according to the target body's mechanical limits.
This design implements a dual-path symmetrical cross-attention mechanism. The driving video features are first processed by spatial self-attention (to encode per-frame instantaneous pose semantics) and temporal self-attention (to capture inter-frame dynamic trajectories), producing a refined motion representation \(F_{\mathcal{X}}^{\text{global}}\). Subsequently, two parallel interaction paths are executed: The first path enables the skeletal joints to query visual motion cues from the video frames (\(S \rightarrow X\)):
The second path allows video representations to query the target skeletal hierarchy (\(X \rightarrow S\)):
Integrating both channels into a unified spatio-temporal embedding ensures that the driving motion is neither weakly broadcast nor overly distorted, but dynamically filtered and tailored to respect the structural topology of target skeleton \(\mathcal{S}\).
3. Global Cartesian Coordinate Diffusion & Skeletal Attention: Error Containment and Kinematic Preservation
Conventional animation models predominantly rely on relative rotational representations (e.g., Euler angles, quaternions, or axis-angles). However, relative rotations suffer from non-linear hierarchical error compounding during 2D-to-3D lifting, where minute errors at root or proximal joints cause catastrophic end-effector displacement.
MotionDreamer adopts global Cartesian coordinates \(\mathcal{P} \in \mathbb{R}^{L \times J \times 3}\) as its primary motion representation, establishing direct geometric correspondence with the 2D video projection and significantly stabilizing diffusion denoising. To prevent anatomical distortions and bone scaling during generation, the initial rest-pose skeleton \(\mathcal{S}_0\) is projected to \(z_0\) and concatenated with the noisy sample \(z_t\) along the temporal axis as an invariant geometric anchor. The transformer layers integrate Skeletal Attention, modulating self-attention weights via a tree-based distance matrix derived from topology \(\mathcal{G}\) to reinforce biologically coupled joints. In the post-processing stage, 150 iterations of an Inverse Kinematics (IK) solver project the generated Cartesian joints back onto the rigid bone constraints of \(\mathcal{S}_0\), eliminating bone stretching and joint dislocations.
Loss & Training¶
The framework is trained using the simplified conditional diffusion objective to predict the added Gaussian noise on 3D joint trajectories:
where \(z_t\) denotes noisy joint positions at diffusion step \(t\), and \(C\) represents the multi-modal conditioning (driving video and mesh semantics). The network is trained from scratch for 80,000 steps with a batch size of 16 using the AdamW optimizer. The initial learning rate is \(1 \times 10^{-4}\) with a step decay multiplying by 0.99 every 10,000 steps. Training is distributed across 4 NVIDIA RTX 5880-48G GPUs.
Key Experimental Results¶
Main Results¶
The quantitative benchmark evaluates 100 randomly sampled models from the curated dataset, rendered from a canonical front view at a distance of 2 meters (\(\le 40\) frames). MotionDreamer is compared against optimization-based skeleton fitting (Puppeteer) and state-of-the-art video-to-4D vertex trajectory generators (Motion 3-to-4, ActionMesh), with the latter converted to skeletal motions via standard post-processing fitting. Metrics include Mean Per-Joint Position Error (MPJPE) and Chamfer Distance (CD) for surface fidelity.
| Method | MPJPE (↓) | CD (↓) | Note |
|---|---|---|---|
| Motion 3-to-4 [4] (CVPR 2026) | 0.230 | 0.475 | Vertex trajectories suffer from severe mesh adhesion and stretching |
| ActionMesh [22] (CVPR 2026) | 0.197 | 0.418 | Lacks skeletal rigidity; exhibits surface melting artifacts |
| Puppeteer [24] (NeurIPS 2026) | 0.161 | 0.155 | Flow- and depth-guided fitting; slow and prone to local optima |
| SkelMo (Ours) | 0.054 | 0.086 | Outperforms all baselines: -66.5% MPJPE vs best baseline; -44.5% CD |
Ablation Study¶
The ablation evaluates the impact of each core technical contribution: "w/o Inject." removes the skinning-based DINOv2 semantic injection; "w/o Bidir." ablates the bidirectional cross-modal fusion; "Rel. Repr." substitutes global Cartesian coordinates with relative joint rotations.
| Config | MPJPE (↓) | CD (↓) | Note |
|---|---|---|---|
| Full Model | 0.054 | 0.086 | Full architecture achieving optimal precision |
| w/o Inject. | 0.234 | 0.235 | Joint collapse due to lack of semantic anatomical priors (+333% MPJPE) |
| w/o Bidir. | 0.118 | 0.147 | Captures gross motion but misses fine-grained limb orientations |
| Rel. Repr. | 0.285 | 0.340 | Compounding rotational errors cause floating and limb length variation |
Furthermore, robustness under imperfect auto-rigged skeletons (generated via RigAnything) demonstrates consistent performance: SkelMo MSE increases only slightly from \(0.048 \times 10^{-1}\) to \(0.082 \times 10^{-1}\), and Vertex-CD from \(0.107 \times 10^{-1}\) to \(0.146 \times 10^{-1}\), continuing to drastically outperform baseline models under ground-truth rigs (e.g., Puppeteer achieves \(0.229 \times 10^{-1}\) MSE).
Key Findings¶
- Semantic grounding prevents structural collapse: Omitting texture-semantic injection results in severe joint clustering (e.g., Spider-Man's limbs collapsing inward), as the network lacks exterior boundary awareness and limb identity priors.
- Global Cartesian formulation stabilizes diffusion: Relative rotational modeling accumulates cascading errors during 2D-to-3D projection, inducing severe ground penetration and floating artifacts. Global Cartesian coordinates provide absolute spatial reference anchors that ensure stable ground contact.
- Broad cross-species generalization: The model successfully retargets human driving motions onto quadruped skeletons and complex creatures while naturally adapting to differences in bone proportions and morphological layouts.
Highlights & Insights¶
- Elegant exploitation of skinning weights for dimension reduction: Using normalized Linear Blend Skinning weights to linearly pool dense vertex features onto discrete joints bypasses complicated correspondence learning, conferring rich semantics to skeletal joints at negligible compute cost.
- Symmetrical cross-modal filtering: The bidirectional interaction not only transfers video motion to the skeleton but uses the target skeleton's structural topology to filter out physically impossible 2D video artifacts.
- High-quality dynamic dataset foundation: Sourcing refined skeletons from Articulation-XL / Objaverse, retargeting animation clips with AutoRig-Pro, and calibrating forward headings via Qwen spatial reasoning establishes a benchmark with ~25,200 dynamic 3D sequences (>3.5 million frames).
Limitations & Future Work¶
- Lack of multimodal semantic conditioning: The current system operates solely on monocular video drivers; incorporating fine-grained textual prompts or audio cues represents an essential step toward universal motion generation.
- Absence of environmental interaction and physics awareness: Generated animations are evaluated purely kinematically, without explicit simulation of terrain contact dynamics, external collisions, or object manipulation.
- Monocular depth ambiguity: In scenarios featuring severe self-occlusion or extreme out-of-plane camera motions, depth-axis joint estimation can exhibit minor jitter.
Related Work & Insights¶
- vs Puppeteer: Puppeteer relies on expensive test-time per-case optimization using optical flow and monocular depth, which is slow and prone to getting trapped in local minima; SkelMo executes fast feed-forward diffusion generation with superior cross-category generalization.
- vs Motion 3-to-4 / ActionMesh: Free-form vertex-trajectory methods lack underlying skeletal constraints, frequently leading to mesh self-intersection, tearing, and geometric melting; SkelMo maintains strict rigid articulation and clean surfaces via skeletal generation paired with IK projection.
- vs SMPL / SMAL Parametric Models: Template-based methods are rigidly bound to specific human or animal anatomies; SkelMo is entirely topology-agnostic, supporting humans, quadrupeds, bipeds, and fantastical creatures in a unified model.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Elegant skinning-weight-guided semantic injection and bidirectional video-skeleton diffusion architecture for universal rigged shapes.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across 25,200 animated assets, comprehensive ablations, auto-rigging robustness analysis, and cross-species in-the-wild demonstrations.
- Writing Quality: ⭐⭐⭐⭐⭐ Mathematically well-formulated, logically coherent, and supported by extensive visual and quantitative comparisons.
- Value: ⭐⭐⭐⭐⭐ Provides a foundational data asset and a state-of-the-art methodology for 4D asset production, game development, and embodied robotics.