EMOSH: Expressive Motion and Shape Disentanglement for Human Animation

Conference: ECCV 2026
arXiv: 2606.28026
Code: To be confirmed
Area: Video Generation
Keywords: Human Animation, Motion-Shape Disentanglement, 3D Parametric Human Model, Diffusion Models, Confidence-Aware Tracking

TL;DR

EMOSH utilizes the Expressive Human Model (EHM), a highly expressive 3D parametric human model, as control signals. By explicitly disentangling shape and pose parameters, it fundamentally eliminates the body shape leakage issue. Combined with a coarse-to-fine hybrid motion injection and a spatially aligned conditioning mechanism, it achieves high-fidelity human animation generation while preserving fine-grained control over expressions and gestures.

Background & Motivation

Controllable human animation is a core technology for applications such as digital humans and film/television content creation. Current mainstream methods employ 2D skeleton poses as control signals for diffusion models (e.g., Animate Anyone, HyperMotion, Wan-Animate, etc.), which can preserve the appearance information of the reference character well through mechanisms like ReferenceNet. However, these methods inherently suffer from the "motion-shape coupling" problem—2D skeletons implicitly encode the body shape of the driving character, which cannot be thoroughly eliminated even through pose retargeting. This leads to the reference character's shape being "contaminated" by the driving character in the generated videos. On the other hand, some works (e.g., Champ, RealisDance) attempt to use depth maps or normal maps rendered from 3D parametric models like SMPL as conditions, achieving shape-motion disentanglement at the geometric level. However, the expressive capacity of the SMPL model is limited to coarse-grained control at the limb level, which fails to depict facial expressions and intricate hand gestures, yielding stiff and lifeless generation results that fall far short of 2D pose-based methods in terms of fine-grained control.

This dilemma between "expressiveness" and "disentanglement capability" is the Key Challenge of current human animation: 2D pose methods offer fine control but leak body shape, whereas 3D mesh methods decouple shape cleanly but provide coarse control. A Key Insight is that if a more expressive 3D parametric model—one capable of simultaneously modeling the body, hand gestures, and facial expressions—is adopted, it is expected to achieve motion-shape disentanglement without sacrificing control precision. In addition to this, there is another deep-seated problem in existing methods that is often overlooked: self-driven data from the same video source (where space is naturally aligned) is used during training, but cross-driven tasks with different characters (where space is unaligned) are faced during inference. This domain gap leads to continuous error accumulation during autogenerating long videos, resulting in identity drift and visual artifacts.

The Core Idea of this paper is to leverage a highly expressive 3D parametric human model, EHM (integrating SMPL-X and FLAME), to explicitly disentangle body shape parameters from motion parameters, fundamentally eliminating body shape leakage. Meanwhile, a coarse-to-fine hybrid motion injection strategy is designed to compensate for the high-frequency details missing in pure mesh rendering, and a spatially aligned conditioning mechanism is introduced to bridge the training-inference domain gap, achieving high-fidelity human animation that is both fine-grained and cleanly decoupled.

Method

Overall Architecture

EMOSH is based on the DiT backbone architecture of Wan2.1-I2V, fine-tuned as a LoRA adaptation for human animation tasks. The overall pipeline is as follows: Given a driving video and a reference image, first, a confidence-aware motion tracker estimates the EHM parameters (pose \(\theta\), expression \(\psi\), shape \(\beta\), camera \(C\)) for each video frame, while the shape parameters \(\beta_r\) are extracted from the reference image. Subsequently, motion retargeting is performed on the driving mesh—preserving the pose and expression parameters of the driving video while injecting the shape parameters of the reference image to achieve explicit motion-shape disentanglement. The retargeted mesh is then rendered into hybrid condition maps via two methods: semantic coloring and sparse keypoint rendering. These maps are encoded by a VAE and injected into the noise latent space in an additive manner. For subsequent segments in long video generation, the spatially aligned latent of the first frame of the first video block is additionally inserted into the latent sequence as a visual anchor to mitigate identity drift. The entire generation process is completed in the DiT denoising network using a Flow Matching objective.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Driving Video"] --> B["Motion Tracker<br/>Jointly Optimize EHM Parameters"]
    B --> C["Motion-Shape Disentanglement<br/>Retargeting: Replace Shape"]

    D["Reference Image"] --> E["Shape Extraction<br/>β_r"]
    E --> C

    C --> F["Hybrid Condition Rendering<br/>Semantic Coloring + Keypoints"]
    F --> G["VAE Encoding<br/>Motion Latents"]
    G --> H["Additive Injection<br/>Motion Signals + Noise Latents"]

    E --> I["Reference Latent"]
    I --> H

    H --> J["DiT Denoising<br/>Wan2.1 Backbone"]
    J --> K["Output Video"]

Key Designs

1. Expressive Motion Representation and Confidence-Aware Tracking Based on EHM

Pure 2D pose methods cannot disentangle body shape, while traditional 3D models like SMPL are limited to coarse-grained control at the limb level. EMOSH introduces EHM (Expressive Human Model) as the core representation—it integrates SMPL-X and FLAME, decomposing the human body parameterization into four independent parts: body shape \(\beta_b\), head shape \(\beta_f\), expression coefficients \(\psi\), and pose parameters \(\theta\), generating 3D meshes via Linear Blend Skinning (LBS). This decomposition naturally disentangles motion and shape, and can simultaneously express the body, hands, and facial movements, raising the upper bound of mesh control expressiveness to a level comparable with 2D poses.

To robustly estimate these parameters from monocular videos, the paper designs a confidence-aware joint optimization tracker. Unlike the cascaded tracking of GUAVA (which optimizes the face first, then the body), EMOSH adopts a unified framework to simultaneously recover body, hand, facial, and camera parameters. The core difficulty lies in the unreliability of 2D keypoint estimation in complex scenarios—for example, facial keypoints shift during large-angle head rotations, or hands reappear mistakenly when occluded. The solution is to introduce a confidence-aware validity gate: for hands, it checks both keypoint confidence and the IoU of the 3D mesh projection with the image boundaries, using keypoint supervision only when the hand is strictly visible; for the face, it computes the angle between the head orientation and the camera viewpoint, zeroing out face keypoint loss weights in large-angle or back-view scenarios to prevent structural collapse due to forced alignment. The total loss for joint optimization is as follows:

\[ \mathcal{L}_{ehm} = \lambda_{kpt}\mathcal{L}_{kpt} + \lambda_{3d}\mathcal{L}_{hand}^{3D} + \lambda_{reg}\mathcal{L}_{reg} + \lambda_{smooth}\mathcal{L}_{smooth} \]

where the 2D reprojection loss computes the L1 distance between the 2D detections and the 3D vertices projected onto the 2D plane via differentiable rendering, the depth component is resolved using a 3D hand vertex alignment loss (with the Z-axis weight scaled up by 10x) to handle gesture depth ambiguity, and the smoothness term penalizes parameter jitter between frames to eliminate frame-by-frame detection noise.

2. Explicit Motion-Shape Disentanglement: Parameter-Level Retargeting

During cross-driven inference, directly using the tracked mesh of the driving video inevitably leaks the body shape of the driving character. EMOSH fundamentally solves this issue through parameter-level retargeting of the EHM: recombining the pose \(\theta^d\) and expression \(\psi^d\) of the driving character with the body shape \(\beta_b^r\) and head shape \(\beta_f^r\) of the reference character:

\[ M_{ehm}^{target} = M_{ehm}(\theta^d, \psi^d, \beta_b^r, \beta_f^r) \]

The resulting mesh is fully faithful to the driving video in terms of motion, while its shape is strictly anchored to the reference character. In practice, shape replacement alters body proportions, causing the rendered head position to shift or even move out of the frame. The solution is to compute the average Y-coordinate difference of the head between the original driving mesh and the shape-replaced mesh in the first frame, and apply a global Y-axis offset to the entire retargeted mesh, eliminating facial misalignment.

3. Coarse-to-Fine Hybrid Motion Injection: Fusing Global Geometry and Local High-Frequency Details

Pure mesh rendering primarily provides low-frequency signals—at a long distance or low resolution, the rendered maps lose micro-expression changes such as blinking or lip closure, which is the key shortcoming of past mesh-based methods compared to 2D pose-based methods. EMOSH mixes two complementary signals: first, it maps the normalized 3D coordinates of each vertex to fixed RGB values under a standard T-pose, generating a dense semantic coloring condition map via differentiable rendering to let the network spatially distinguish different body regions (this is the "coarse" level, providing global structure). Then, a set of key semantic vertices (such as eye corners and mouth corners) is selected and explicitly drawn with different colors onto the rendered map (this is the "fine" level, supplementing high-frequency details). The final condition map is encoded by the VAE, passes through a lightweight embedding layer, and is directly added to the noise latents. This design cleverly combines "low-frequency structure + high-frequency details" into a single condition map, achieving the control precision of 2D pose methods while maintaining disentanglement capability—ablation experiments show that the generation quality of fine expressions like blinking drops significantly once keypoint rendering is removed.

4. Spatially Aligned Conditioning Mechanism: Bridging the Training-Inference Domain Gap

This is an easily overlooked but deeply impactful issue. During training, the reference frame and the target frame come from the same video and are naturally spatially aligned; during inference, cross-identity data exhibits spatial misalignment. As autogeneration progresses frame-by-frame, the model's attention gradually shifts to the spatially aligned temporal latents, ignoring the original reference latents, and identity consistency deteriorates significantly over time. The key insight is that the first frame latent of the first video block is naturally spatially aligned. The proposed solution is to retain the first frame latent of the first video block after generating it, and then insert it between the original reference latent and the temporal latents in the concatenated sequence for all subsequent video blocks. This anchor latent accumulates far fewer errors than the recursively generated temporal latents, effectively redistributing the model's attention and preventing over-reliance on noisy temporal cues. During training, spatial misalignment is simulated with a 50% probability using random geometric augmentations (rotation, translation), and auxiliary reference frames are randomly injected with a 10% probability, allowing the model to adapt in advance to the inference mode with spatially aligned latents.

Loss & Training

Based on the Wan2.1-I2V-14B backbone, the model fine-tunes the DiT blocks using LoRA (rank 32) with a learning rate of \(1 \times 10^{-4}\). Training employs a Flow Matching objective function (optimal transport path) and randomly samples 77-frame video clips at \(512 \times 512\) resolution. The training dataset consists of approximately 900,000 video segments (37.8% internet human videos + 60.6% SpeakerVid + 1.6% VFHQ), trained on 32 H20 GPUs for about 100 hours and 6,500 iterations. During inference, an overlapping frame-based autoregressive strategy is adopted, where the last 5 frames of the previous video block are encoded as temporal latents to serve as the context for the subsequent block.

Key Experimental Results

Main Results

Under self-driven scenarios (same-source video/first-frame reconstruction), EMOSH comprehensively outperforms all baseline methods across three datasets:

Dataset	Metric	EMOSH	Wan-Animate	HyperMotion	Champ
EchoMimicV2	PSNR↑	23.66	22.86	22.32	17.40
EchoMimicV2	LPIPS↓	0.1428	0.1802	0.1800	0.2911
TikTok	PSNR↑	17.80	17.18	15.48	13.34
Self-collected	PSNR↑	21.18	20.72	19.48	14.85

In cross-driven scenarios, EMOSH's Identity Preservation Score (IPS, based on ArcFace facial feature cosine similarity) reaches 0.4445, which is significantly higher than Wan-Animate (0.3802) and HyperMotion (0.2872). In user study (GSB pairwise comparison), EMOSH beats Wan-Animate, HyperMotion, and UniAnimate-DiT with win rates of 89.21%, 98.42%, and 95.62%, respectively.

Ablation Study

Configuration	EchoMimicV2 PSNR↑	IPS↑	Description
Full (EMOSH)	23.66	0.4445	Full model
w/o Tracker	20.51	—	Replaced the motion tracker with off-the-shelf estimators; expression and gesture control precision dropped significantly
w/o Hybrid Motion	22.90	—	Removed 2D keypoint drawing, keeping only pure mesh rendering; high-frequency details (e.g., blinking) were lost
w/o Disentanglement	—	0.4258	Removed EHM retargeting; driving character's shape leaked
w/o SAC	—	0.4237	Removed spatially aligned conditioning; identity preservation degraded, and long-video artifacts increased

Key Findings

• The confidence-aware motion tracker contributes the most: removing it drops PSNR from 23.66 to 20.51, and its tracking speed is 1.9x faster than GUAVA (223s vs. 424s for a 436-frame video).
• The contribution of 2D keypoint drawing in hybrid motion injection is mainly manifested in high-frequency details (dropping PSNR from 23.66 to 22.90 upon ablation), while semantic coloring provides the underlying structure, with the two complementing each other.
• The advantage of SAC expands as the video length increases: the difference is minor for short clips, but the gain in identity preservation continuously widens for long videos.
• Additional FID/FVD metrics validate the comprehensive advantages of EMOSH in spatial quality and temporal consistency.

Highlights & Insights

• The choice of EHM is the core breakthrough: Instead of using 3D for the sake of 3D, it identifies a representation that simultaneously satisfies "expressive capability (body + hands + face)" and "disentanglement capability (shape vs. motion)", merging the respective strengths of the previous 2D and 3D paradigms.
• The hybrid conditioning design is simple yet effective: Drawing sparse keypoints directly on the semantically colored rendering to supplement high-frequency details is far more elegant than maintaining two independent control branches, and the additive injection preserves the simplicity of the network structure.
• The concept of the SAC anchor frame stems from a precise analysis of training-inference discrepancies: Rather than crudely adding post-processing at inference time, it targets the root cause of "attention drift" by redistributing attention using a naturally aligned anchor—this idea can be transferred to other long video/autoregressive generation tasks.
• The confidence-aware gating design of the joint optimization tracker is ingenious: Utilizing 3D information from hand "projection IoU detection" and facial "orientation angle detection" to filter out unreliable 2D keypoint supervision represents a classic paradigm of "using 3D priors to remedy 2D deficiencies."

Limitations & Future Work

• Currently, only a low-resolution (\(512 \times 512\)) version has been trained. Restricted by training data quality, high-resolution generation remains to be explored.
• The inference speed is slow, requiring several minutes to generate short videos, which incurs high computational overhead.
• The mesh conditioning cannot disentangle camera motion from human motion, leading to failure in controlling rotational trajectories other than zoom-in/out shots—since the training data is dominated by static cameras + moving subjects, the model misinterprets rendering scale changes as subject motion.
• During shape retargeting, due to differences in proportions across different people, the same pose on different body shapes can alter the relative positions of end-effectors, which might cause "clapping hands" to be misinterpreted as "crossed hands."
• Generation quality heavily relies on the precision of the upstream motion tracker, meaning any tracking errors (such as mesh self-intersection) will propagate directly to the final video.

Related Work & Insights

• vs. Champ / RealisDance: They use depth/normal maps rendered by SMPL as conditions, which achieves disentanglement but offers weak expression and gesture control. EMOSH replaces SMPL with the highly expressive EHM and adds hybrid conditioning to compensate for high-frequency details.
• vs. Wan-Animate: Both are based on the Wan2.1 backbone, but Wan-Animate still uses 2D poses + facial keypoints, failing to eliminate shape leakage. EMOSH fundamentally resolves this issue through 3D parametric disentanglement. Their complementarity lies in that Wan-Animate's weights can serve as the DiT initialization for EMOSH.
• vs. GUAVA Tracker: GUAVA employs a cascaded tracking pipeline (face first, then body), which is inefficient and prone to failure in large-angle scenarios. EMOSH's joint optimization + confidence-aware gating outperforms it in both accuracy and speed (1.9x).
• The design concept of confidence-aware gating (using 3D priors to validate the reliability of 2D detections) can be applied to other 3D reconstruction/tracking tasks that rely on keypoint supervision.

Rating

• Novelty: ⭐⭐⭐⭐ [The idea of introducing the highly expressive EHM into human animation to achieve motion-shape disentanglement is novel, and the hybrid conditioning and SAC designs are ingenious]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Three self-driven test sets + cross-driven IPS + user study + detailed ablations, covering both quantitative and qualitative aspects comprehensively]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear problem motivation, smooth methodological logic, complete figures and tables, and the Supplementary material provides extensive implementation details]
• Value: ⭐⭐⭐⭐ [Resolves a crucial disentanglement bottleneck in human animation with high practical value; however, resolution is limited and inference speed requires optimization, calling for further work to fully commercialize]

Related Papers

• [ICCV 2025] X-Dancer: Expressive Music to Human Dance Video Generation
• [CVPR 2026] PersonaLive! Expressive Portrait Image Animation for Live Streaming
• [CVPR 2025] TokenMotion: Decoupled Motion Control via Token Disentanglement for Human-centric Video Generation
• [CVPR 2026] Vanast: Virtual Try-On with Human Image Animation via Synthetic Triplet Supervision
• [AAAI 2026] MotionCharacter: Fine-Grained Motion Controllable Human Video Generation