MorphGS: Morphology-Adaptive Articulated Motion Transfer from Videos

Conference: ECCV 2026
Paper: ECCV 2026 Poster
Code: https://xodus777.github.io/MorphGS/
Area: 3D Vision
Keywords: motion retargeting, 3D Gaussian splatting, differentiable rendering, skeleton-driven deformation, monocular motion transfer

TL;DR

MorphGS reformulates monocular articulated motion transfer as a target-driven analysis-by-synthesis optimization problem, coupling rig-anchored morphology parameterization with differentiable 3D Gaussian rasterization to recover motion without intermediate source reconstruction or parametric shape templates.

Background & Motivation

Transferring articulated 3D motion from monocular in-the-wild videos to rigged 3D character meshes is a longstanding goal in computer vision and computer animation. Prevailing techniques predominantly follow a reconstruct-then-retarget paradigm: they first reconstruct an intermediate 3D mesh or skeletal sequence from the monocular RGB footage, and then retarget the resulting trajectory onto the desired target character. Consequently, these approaches rely heavily on category-specific parametric shape templates such as SMPL for humans or SMAL for quadrupeds, severely restricting their utility when animating arbitrary non-standard characters such as birds, marine animals, or stylized creatures. Furthermore, depth and occlusion ambiguities in monocular video inevitably corrupt intermediate 3D reconstructions, and these errors are cascadingly amplified during retargeting when the source subject and target character exhibit substantial morphological discrepancies.

The core tension underlying this problem is that articulated motion in a 2D video is observed through image-space projections, yet prior works attempt to disentangle motion in the source geometric space where cross-morphology mapping is ill-posed. Conversely, directly optimizing arbitrary target mesh deformation alongside pose from monocular inputs introduces severe shape-pose ambiguity, where unconstrained surface vertices shift arbitrarily to compensate for joint rotation errors, resulting in catastrophic structural drift and loss of character identity.

To resolve this limitation, MorphGS approaches video motion retargeting from an inverse-graphics analysis-by-synthesis perspective. Rather than building an explicit 3D replica of the source, the framework places the rigged target character into the differentiable rendering loop, iteratively refining target morphology and pose to explain the source video observations directly in image space. Core idea: cast articulated motion transfer as target-driven analysis-by-synthesis, decoupling character identity from time-varying pose via a rig-anchored, time-invariant morphology parameterization (bone lengths, global scale, and rest-pose offsets), while anchoring optimization with dense 2D-3D semantic correspondences and diffusion-synthesized novel views.

Method

Overall Architecture

Given a source monocular RGB video \(\{I_t \in \mathbb{R}^{H \times W \times 3}\}_{t=1}^T\) and a rigged target 3D mesh \(\mathcal{M}^{\mathrm{tgt}}\), MorphGS optimizes a temporally coherent joint pose trajectory \(\{P_t(\phi)\}_{t=1}^T\) that animates the target character while preserving its canonical structural identity. MorphGS achieves this by avoiding explicit source 3D reconstruction and instead synthesizing the posed target directly via differentiable rendering. At each time step \(t\), the posed shape \(\tilde{\mathcal{M}}_t(\phi, m)\) is generated through two factorized stages: first, canonical morphology adaptation \(\tilde{\mathcal{M}}(m) = \mathcal{W}(\mathcal{M}^{\mathrm{tgt}}; m)\) adjusts bone lengths, global scale, and local geometry to bridge the anatomical gap between source and target; second, rig-driven posing \(\mathcal{S}(\tilde{\mathcal{M}}(m), P_t(\phi))\) articulates the adjusted mesh via forward kinematics and Linear Blend Skinning (LBS) based on the pose predicted by a temporal MLP. The articulated mesh is represented using 3D Gaussian primitives and projected via differentiable rasterization to match the source frame.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Target Mesh Mtgt + Source Video It"] --> B["Rig-Anchored Morphology Parameterization<br/>Bone lengths ℓb + Scale s + Offsets ok"]
    B --> C["Pose-Conditioned Temporal MLP<br/>Predicts joint rotations & root translation"]
    C --> D["Articulated 3D Gaussian Skinning & Rasterization<br/>LBS-driven differentiable splatting"]
    D --> E["Multi-Source Image-Space Supervision<br/>Rendering losses + dense keypoints + novel views"]
    E --> F["Output: Retargeted Pose Trajectory Pt(ϕ)<br/>Preserves target geometry and rig topology"]

Key Designs

1. Rig-Anchored Morphology Parameterization: Disentangling Identity from Articulation

Jointly optimizing shape and pose from monocular video under unconstrained vertex representations is inherently ill-posed, as free vertices inevitably absorb articulation errors. MorphGS prevents shape-pose entanglement by strictly confining time-invariant morphological variation to structured parameters coupled to the target rig, defined as \(m = \big( s_{\mathrm{global}}, \{\ell_b\}_{b \in \mathcal{B}}, \{\mathbf{o}_k\}_{k=1}^K \big)\). In the target's kinematic tree, bone directions \(\mathbf{d}_b\) and parent-child hierarchies are held fixed, allowing only the scalar bone lengths \(\ell_b \in \mathbb{R}^+\) to vary along kinematic paths from the root joint: $\(\mathbf{j}_{\mathrm{rest}}(j; \boldsymbol{\ell}) = \mathbf{j}_{\mathrm{rest}}(j_{\mathrm{root}}) + \sum_{b \in \mathcal{P}(j_{\mathrm{root}}, j)} \ell_b \mathbf{d}_b\)$ Each mesh vertex is then grounded to a skeleton anchor \(\mathbf{p}_k = \sum_{j \in \mathcal{J}} w_{kj} \mathbf{j}_{\mathrm{rest}}(j; \boldsymbol{\ell})\) defined by the convex combination of rest joints weighted by LBS skinning weights \(w_{kj}\). Local surface variations (such as limb thickness or silhouette details) are modeled as rest-space offsets \(\mathbf{o}_k \in \mathbb{R}^3\) relative to \(\mathbf{p}_k\), followed by an isotropic global scale \(s_{\mathrm{global}}\) to resolve monocular depth-scale ambiguity: \(\bar{\mathbf{v}}_k = s_{\mathrm{global}} (\mathbf{p}_k + \mathbf{o}_k)\). Under weak-perspective projection, enforcing strict time-invariance on \(\ell_b\) and \(\mathbf{o}_k\) restricts all temporal dynamics to joint rotations, rendering morphology and pose mathematically identifiable up to a global scale.

2. Pose-Conditioned Temporal MLP and Articulated 3D Gaussian Rendering

To guarantee smooth motion without per-frame jitter or convergence to local minima, MorphGS parameterizes the continuous pose trajectory using a compact coordinate MLP \(f_\phi\) conditioned on sinusoidal time embeddings: \(P_t(\phi) = f_\phi(\mathrm{emb}(t))\), which outputs axis-angle relative joint rotations and root translations. For efficient gradient propagation from pixel-space objectives back to joint rotations, the canonical mesh vertices are initialized as \(K\) 3D Gaussian primitives that inherit their skinning weights \(\{w_{kj}\}\) directly from the target rig. Forward kinematics converts local rotations into global transformation matrices \(\mathbf{T}_j^t \in \mathrm{SE}(3)\), driving the canonical Gaussian centers via Linear Blend Skinning: $\(\mathbf{v}_k^t = \sum_{j \in \mathcal{J}} w_{kj} \mathbf{T}_j^t \bar{\mathbf{v}}_k\)$ The transformed Gaussian primitives are then rasterized via depth-sorted alpha compositing to generate rendered frames \(\hat{I}_t = \mathcal{R}(\tilde{\mathcal{M}}_t(\phi, m))\), establishing an end-to-end differentiable pipeline from image pixels to skeletal rotations.

3. Dense 2D-3D Semantic Correspondences and Synthesized-View Guidance

Relying solely on photometric and silhouette losses leaves optimization vulnerable to bad local minima when transferring across characters with severe morphological differences, and monocular observations provide no constraint for self-occluded limbs. MorphGS incorporates two complementary structural constraints: First, a geometry-aware feature extractor maps source video pixels \(\mathbf{u}\) to rendered multi-view target mesh features, computing pooled cosine similarity scores to establish dense 2D-3D keypoint correspondences \((\mathbf{u}, \tilde{\mathbf{x}}_{\mathbf{u}, t})\). A smooth-\(\ell_1\) reprojection loss \(\mathcal{L}_{\mathrm{keyp}}\) is enforced on high-confidence pairs, anchoring distinct anatomical parts even in early optimization phases. Second, a video diffusion model synthesizes spatio-temporally consistent pseudo-views of the source subject, expanding rendering supervision across multiple viewpoints (\(\mathcal{L}_{\mathrm{mv}}\)) and resolving monocular depth and occlusion ambiguities.

Loss & Training

The overall per-frame optimization objective integrates rendering terms, structural keypoint anchors, and temporal regularization: $\(\mathcal{L}_t = \mathcal{L}_{\mathrm{rgb}} + \mathcal{L}_{\mathrm{mask}} + \lambda_{\mathrm{keyp}} \mathcal{L}_{\mathrm{keyp}} + \lambda_{\mathrm{mv}} \mathcal{L}_{\mathrm{mv}} + \mathcal{L}_{\mathrm{reg}}\)$ Here, \(\mathcal{L}_{\mathrm{rgb}}\) ensures photometric consistency within the foreground mask \(\Omega_t\), \(\mathcal{L}_{\mathrm{mask}}\) aligns alpha silhouettes, \(\mathcal{L}_{\mathrm{keyp}}\) enforces 2D-3D keypoint alignment, \(\mathcal{L}_{\mathrm{mv}}\) penalizes rendering discrepancy across synthesized views, and \(\mathcal{L}_{\mathrm{reg}}\) penalizes angular velocity and acceleration jerkiness across adjacent frames.

Optimization follows a three-stage progressive schedule over 5,000 iterations: - Stage 1 (0 to 500 iters): Optimizes only global alignment variables (root translation, root orientation, and global scale \(s_{\mathrm{global}}\)) to anchor initial spatial scale and bounding volume; - Stage 2 (500 to 1,500 iters): Unfreezes bone lengths \(\{\ell_b\}\) and local joint rotations via the pose MLP \(\phi\), adapting skeletal proportions to capture macro articulated motion; - Stage 3 (1,500 to 5,000 iters): Jointly optimizes all parameters, including rest-space offsets \(\{\mathbf{o}_k\}\) and Gaussian splatting attributes (color, scale, opacity, rotation), capturing fine surface details while refining the pose trajectory. Optimization executes on a single NVIDIA RTX 4090 GPU in approximately 5 minutes for a 10K-vertex target mesh. After convergence, the adapted morphology parameters are discarded, and only the pure pose trajectory \(\{P_t(\phi)\}\) is applied to the original target mesh.

Key Experimental Results

Main Results

Quantitative evaluations are conducted across synthetic benchmarks (Mixamo with 12 humanoid pairs, DT4D-Quadrupeds with 20 quadruped pairs, and DT4D-Others with 5 non-standard creature categories, totaling 1,505 evaluated mesh sequences) as well as real-world monocular video clips from the DAVIS dataset. Metrics include Point-wise Mesh Distance (PMD, \(\times 10^3\) ↓) for 3D geometric error and Fréchet Video Motion Distance (FVMD, \(\times 10^{-3}\) ↓) for perceptual motion plausibility.

Dataset / Split	Metric	MorphGS (Ours)	Best Baseline	Baseline Method	Gain / Error Reduction
Mixamo-Humanoids	PMD (×10³) ↓	1.91	2.96	SPT+	-35.5% geometric error
Mixamo-Humanoids	FVMD (×10⁻³) ↓	8.82	12.56	SPT+	Superior motion consistency
DT4D-Quadrupeds	PMD (×10³) ↓	1.33	3.28	NPR+	-59.5% geometric error
DT4D-Quadrupeds	FVMD (×10⁻³) ↓	8.81	15.24	Pinocchio+	Significant perceptual fidelity gain
DT4D-Others (Non-standard)	PMD (×10³) ↓	1.95	7.16	Transfer4D	-72.8% error (prior SOTA degrades)
DT4D-Others (Non-standard)	FVMD (×10⁻³) ↓	12.83	14.23	Pinocchio+	Robust transfer on exotic topologies

On real-world monocular videos from DAVIS (FVMD \(\times 10^{-3}\) ↓): - Human scenes average: MorphGS achieves 12.74 (outperforming Pinocchio+ at 15.39 and SPT+ at 15.17); - Animal scenes average: MorphGS achieves 12.49 (outperforming Pinocchio+ at 15.83 and NPR+ at 19.00), notably achieving 6.43 on the challenging "cows" sequence compared to 12.58 for Pinocchio+.

Ablation Study

The ablation study on Mixamo across 12 motion scenes systematically assesses the role of individual morphology components and supervision losses:

Ablation Category	Configuration	PMD (×10³) ↓	FVMD (×10⁻³) ↓	Key Insight / Mechanism
Morphology Parameterization	Naïve model (per-vertex free offsets)	8.45	17.54	Severe unconstrained distortion; Gaussians drift off rig
Morphology Parameterization	Fixed morphology (\(m\) disabled)	3.72	15.30	Pose angles overcompensate for anatomical shape gaps
Morphology Parameterization	+ Bone lengths \(\ell_b\)	2.87	12.45	Corrects limb proportions and skeletal scaling
Morphology Parameterization	+ Bone lengths \(\ell_b\), scale \(s_{\mathrm{global}}\)	1.99	9.55	Resolves monocular depth-scale ambiguity
Morphology Parameterization	Full model (+ \(\ell_b, s_{\mathrm{global}}, \mathbf{o}_k\))	1.91	8.82	Rest-pose offsets capture fine silhouette geometry
Supervision Losses	\(\mathcal{L}_{\mathrm{render}}\) only	4.52	13.57	Photometric and silhouette cues lack part correspondence
Supervision Losses	+ Dense 2D-3D keypoints \(\mathcal{L}_{\mathrm{keyp}}\)	2.96	11.93	Explicit semantic anchors provide strong part guidance
Supervision Losses	Full model (+ \(\mathcal{L}_{\mathrm{keyp}}, \mathcal{L}_{\mathrm{mv}}\))	1.91	8.82	Synthesized multi-view eliminates self-occlusion ambiguity

Key Findings

• Skeleton-anchored parameterization prevents geometric collapse: Optimizing vertices as free variables without rig anchoring (Naïve model) degrades PMD drastically from 1.91 to 8.45 (+342% error), confirming that rigid kinematic anchoring is essential to prevent shape variables from absorbing pose errors.
• Superior generalization to non-standard creature topologies: On the DT4D-Others benchmark, baselines completely degrade (PMD 7.16 to 7.59), whereas MorphGS sustains an accurate PMD of 1.95, proving its template-free flexibility on non-human, non-quadruped subjects like whales and birds.
• Multi-view synthesis effectively resolves monocular occlusion: Incorporating diffusion-synthesized novel views (\(\mathcal{L}_{\mathrm{mv}}\)) drops PMD from 2.96 to 1.91, proving that multi-view consistency effectively constrains occluded body parts during unilateral turns and complex motion.

Highlights & Insights

• Inverting the retargeting paradigm: By formulating motion transfer as target-driven analysis-by-synthesis, MorphGS bypasses the error-prone intermediate 3D source reconstruction step entirely, eliminating error propagation under morphological mismatch.
• Identifiable shape-pose factorization: By coupling bone lengths and local rest-space offsets to the target rig and holding them strictly time-invariant, the optimization mathematically ensures that temporal video dynamics are absorbed solely by joint rotations.
• Practical 5-minute single-GPU optimization: Leveraging 3D Gaussian Splatting and staged optimization, MorphGS completes full articulated sequence transfer in approximately 5 minutes on a standard RTX 4090 GPU.

Limitations & Future Work

• Requirement for compatible kinematic topology: The framework assumes that source and target share a broadly compatible kinematic structure (e.g., matching limb branching and joint hierarchy), limiting direct transfer between drastically different topologies (such as quadrupeds to octopuses).
• Absence of explicit physical contact dynamics: Because optimization relies on image-space cues without ground contact or mass constraints, high-dynamic behaviors may exhibit slight foot-sliding or inter-penetration.
• Future directions: Integrating physics-based priors to enforce ground contact and dynamic balance, as well as extending morphology parameterization to model non-rigid soft-tissue inertia.

Related Work & Insights

• vs SPT+ / NPR+ (Reconstruct-then-Retarget Pipelines): Prior methods reconstruct SMPL/SMAL source meshes from video and then solve cross-mesh deformation transfer. MorphGS operates directly in target parameter space via differentiable rendering, eliminating the need for category-specific parametric templates.
• vs Transfer4D (RGB-D Skeleton Transfer): Transfer4D relies on metric depth inputs and struggles on non-standard creature categories. MorphGS operates on standard monocular RGB video and demonstrates superior robustness across diverse animal morphologies.
• vs 4D Gaussian Reconstruction (GaussianAvatar / Riggs): While dynamic Gaussian methods reconstruct subject-specific avatars from video, they do not transfer motion across morphologically disparate characters. MorphGS bridges this gap by decoupling target morphology from observed motion.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Target-driven analysis-by-synthesis for motion retargeting with rig-coupled morphology parameterization]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Rigorous benchmarking on humanoids, quadrupeds, non-standard creatures, and real DAVIS videos with extensive ablations]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical formulation, thorough motivation, and elegant visual presentation]
• Value: ⭐⭐⭐⭐⭐ [Provides a practical, template-free solution for cross-species 3D character animation from monocular video]