CMDer: Controllable Mode Decomposition-Based Single Motion Synthesis with Diffusion¶
Conference: ECCV 2026
Paper: ECCV 2026 Official
Area: Image Generation
Keywords: single motion synthesis, multivariate variational mode decomposition, diffusion models, frequency-selective control, crowd animation
TL;DR¶
CMDer decomposes a single motion exemplar into multivariate intrinsic frequency modes via Multivariate Variational Mode Decomposition (MVMD) and trains a lightweight diffusion model with a Mode Enhancement Block (MEB) in mode space, enabling frequency-selective control that preserves base rhythmic locomotion while varying fine-grained details.
Background & Motivation¶
Generating diverse, high-fidelity, and controllable human or creature motions from a single motion sequence is crucial in data-scarce scenarios such as stylized game character animation and custom animal rigging. However, existing single-motion generation techniques predominantly operate in the spatial domain (e.g., joint-level constraints, kinematic trees) or the temporal domain (e.g., keyframing, temporal in-betweening, patch matching). These paradigms struggle to strike an optimal balance between coherence and diversity: strict temporal constraints easily lead to motion locking or repetitive sequence splicing, whereas unconstrained generative modeling often triggers severe skeletal distortions and jittery unnatural transitions.
The fundamental bottleneck stems from the intrinsic hierarchical nature of motion signals across frequency scales. Low-frequency components encapsulate the essential global rhythm, stride cadence, and characteristic bodily posture of a movement, whereas high-frequency components encode idiosyncratic local details, secondary dynamics, and subtle limb vibrations. Classical Fourier analysis assumes signal stationarity, making it ill-suited for non-periodic transitions typical of complex motions; wavelet transforms rely on pre-selected basis functions and fail to dynamically maintain coordinated multi-joint phase alignments; and empirical mode decomposition methods suffer from mode mixing and severe noise susceptibility.
To overcome these challenges, this paper shifts the control dimension from temporal-spatial masking to adaptive frequency-mode manipulation. By employing multivariate variational mode decomposition, motion signals can be cleanly partitioned into coordinated intrinsic frequency bands across all joints simultaneously. Core idea: adaptively decompose single multivariate motion sequences into intrinsic frequency modes via Multivariate Variational Mode Decomposition (MVMD), and train a lightweight U-Net with anisotropic Mode Enhancement Blocks (MEB) to denoise directly in the mode latent space, enabling users to preserve fundamental rhythmic styles while freely synthesizing diverse fine-grained details.
Method¶
Overall Architecture¶
The CMDer pipeline comprises three principal stages: Motion Mode Decomposition Module (MMDecomp), Motion Mode Diffusion Network (MMDiffusion), and Motion Mode Composition Module (MMComp). Given an input single motion sequence composed of 6D rotation features and global root positions, MMDecomp applies MVMD to decompose the non-stationary multichannel signal into \(K\) intrinsic mode functions (IMFs) sharing common center frequencies. Next, MMDiffusion encodes rotation and position modes through dedicated convolutional blocks and passes them into a lightweight single-layer U-Net diffusion architecture equipped with Mode Enhancement Blocks (MEB), performing denoising under user-defined mode-spatial-temporal conditioning masks. Finally, MMComp reconstructs the output motion by summing the generated mode components along the frequency axis into a unified, continuous 3D skeletal animation.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Single Motion Input<br/>6D Rotations R and Root Positions P"] --> B["1. Multivariate Variational Mode Decomposition MMDecomp<br/>Adaptively extracts K coordinated IMF modes"]
B --> C["Feature Embedding Block<br/>Separate adaptive convolutional mapping for R and P"]
C --> D["2. Mode-Enhanced Diffusion Denoising MMDiffusion<br/>Lightweight U-Net with anisotropic MEBs"]
D --> E["Feature Decoding Block<br/>GroupNorm and 1D convolutions reconstruct modes"]
E --> F["3. Mode Summation Composition MMComp<br/>Additive synthesis of continuous 3D motion"]
Key Designs¶
1. Multivariate Variational Mode Decomposition: Adaptive Multi-Joint Frequency Disentanglement To address the inability of fixed-basis transforms to capture non-stationary transitions and inter-joint coordination, the framework introduces Multivariate Variational Mode Decomposition (MVMD) to skeletal motion generation. Treating motion trajectories as multi-channel continuous signals, MVMD formulates a constrained variational problem that minimizes the sum of estimated bandwidths of analytic signals around shared center frequencies \(\omega_k\), subject to exact signal reconstruction: $$ \min_{{u_{k,c}}, {\omega_k}} \sum_{k=1}^K \sum_{c=1}^C \left| \partial_t \left[ u^+{k,c}(t) e^{-j\omega_k t} \right] \right|_2^2 \quad \text{s.t.} \quad \sum(t) = x_c(t) $$ where }^K u_{k,c\(u^+_{k,c}(t)\) denotes the analytic signal of the \(k\)-th mode for channel \(c\). Solved iteratively using ADMM in the Fourier domain, this formulation enforces shared frequency evolutions across joints, ensuring that lower-indexed modes strictly capture coordinated macroscopic rhythmic motions while higher-indexed modes cleanly isolate instantaneous tremors and subtle variations.
2. Mode Enhancement Block: Localized Anisotropic Dynamics Without Overfitting Under single-sample supervision, conventional global self-attention tends to overfit or smooth out delicate kinematic variations due to its unconstrained receptive field. To preserve localized spatial-temporal details while keeping computational complexity minimal, the authors introduce a pure convolution-based Mode Enhancement Block (MEB). Operating independently on rotation mode features \(Z_R \in \mathbb{R}^{K \times C_R \times T}\) and position mode features \(Z_P \in \mathbb{R}^{K \times C_P \times T}\), MEB utilizes depthwise strip convolutions with small kernels along orthogonal axes (\((1, t)\) temporal and \((j, 1)\) spatial kernels) combined with a Sigmoid gating mechanism: $$ Z'i = \mathrm{Sigmoid} \circ \mathrm{BN} \circ \mathrm{Conv}_4 \circ \mathrm{Conv}_1(Z_i) $$ This design strictly bounds the receptive field to local neighborhoods, enabling the model to learn fine temporal rhythms and joint dependencies within each mode without memorizing global trajectory sequences.} \circ \mathrm{Conv}_{dw2} \circ \sigma \circ \mathrm{BN} \circ \mathrm{Conv
3. Frequency-Selective Diffusion & Editing: Mode-Level Mask Conditioning Unlike standard motion diffusion models that operate directly on raw spatial coordinates, CMDer learns reverse diffusion directly in the decomposed mode domain. The network predicts the clean signal \(\hat{x}_0 = G(x_\tau, \tau, c)\) under a combined objective with rotation mean squared error \(\mathcal{L}_R\), position mean squared error \(\mathcal{L}_P\), and a variational lower-bound term \(\mathcal{L}_{\mathrm{VB}}\): $$ \mathcal{L} = \lambda_R \mathcal{L}R + \lambda_P \mathcal{L}_P + \mathcal{L} $$ During inference and editing, spatial-temporal ROI masks are expanded to mode space }\(m' \in \mathbb{R}^{K \times (J \times (Q+3)) \times T}\). Users can selectively preserve low-frequency modes (e.g., modes 0–1 with \(K=8\)) to retain the fundamental gait and body sway of an exemplar, while diffusing unconstrained high-frequency modes from random Gaussian noise to produce coordinated, non-cloned crowd animations or style transfers.
Key Experimental Results¶
Main Results¶
On the Mixamo standard benchmark comprising 20 human motion clips (60 to 900 frames) with diverse styles, models are compared across Coverage (Cov.), Global Diversity (GloDiv.), Local Diversity (LocDiv.), Inter Diversity (IntDiv.), Intra Diversity Difference (IntDiff.), Smoothness via spectral arc length (Smo.), and Harmonic Mean (HM.).
| Method | Cov. ↑ | GloDiv. ↑ | LocDiv. ↑ | IntDiv. ↑ | IntDiff. ↓ | Smo. ↑ | HM. ↑ |
|---|---|---|---|---|---|---|---|
| GANimator [16] | 96.39 | 1.12 | 1.07 | 0.08 | 0.26 | -7.20 | 0.72 |
| GenMM [18] | 99.26 | 0.34 | 0.25 | 0.12 | 0.22 | -7.32 | 0.38 |
| SinMDM [21] | 97.41 | 1.30 | 0.91 | 0.13 | 0.19 | -7.35 | 0.76 |
| CMDer (Ours) | 97.94 | 1.32 | 1.10 | 0.12 | 0.18 | -7.15 | 0.78 |
Ablation Study¶
1. Ablation on MEB and Decomposition Strategy
| Config | Cov. ↑ | GloDiv. ↑ | LocDiv. ↑ | Smo. ↑ | HM. ↑ | Note |
|---|---|---|---|---|---|---|
| MVMD + w/ MEB (Full) | 97.94 | 1.32 | 1.10 | -7.15 | 0.78 | Balanced coverage, superior smoothness and diversity |
| MVMD + w/o MEB | 56.60 | 1.77 | 1.68 | -13.54 | 0.00 | Severe mode collapse and out-of-distribution distortion |
| FFT + w/ MEB | 92.61 | 1.48 | 1.37 | -10.93 | 0.46 | Fixed basis fails to track non-stationary multi-joint motion |
2. MVMD Sensitivity Analysis Across Number of Modes \(K\)
| Modes \(K\) | Recon.(%) ↑ | Freq. Sep. (Hz) ↑ | Stability ↑ | Time (s) ↓ | Gain Ratio ↑ |
|---|---|---|---|---|---|
| 4 | 96.75 | 0.657 | 0.881 | 5.28 | 0.660 |
| 8 (Default) | 98.70 | 0.533 | 0.895 | 11.43 | 0.185 |
| 12 | 99.24 | 0.499 | 0.895 | 16.98 | 0.043 |
| 20 | 99.53 | 0.376 | 0.880 | 27.47 | 0.016 |
Key Findings¶
- MEB is critical for stabilizing single-sample learning: Removing MEB causes coverage to drop drastically from 97.94 to 56.60 and smoothness to plunge from -7.15 to -13.54. The abnormally elevated diversity scores (GloDiv 1.77, LocDiv 1.68) reflect erratic, physically unrealistic skeletal distortions, reducing HM to 0.00.
- Adaptive MVMD significantly outperforms fixed-basis FFT: Replacing MVMD with an FFT-based band decomposition leads to substantial degradation in motion smoothness (-10.93 vs. -7.15) and composite harmonic mean (0.46 vs. 0.78), proving the necessity of data-adaptive, joint-correlated frequency bases.
- Decomposition parameter \(K=8\) marks the performance sweet spot: Reconstructing with \(K=8\) achieves 98.70% signal fidelity with the highest alignment stability (0.895); higher mode counts (\(K=20\)) degrade frequency separation to 0.376 Hz due to band overlap and redundancy.
Highlights & Insights¶
- Bridging classical variational signal decomposition and diffusion: Introducing MVMD with joint-frequency constraints directly resolves the tension between multi-joint coordination and frequency band isolation, providing a principled inductive bias for low-data generative modeling.
- Frequency-selective control as a new editing axis: Enables selective preservation of base rhythmic style and stochastic generation of fine details, naturally solving long-standing dilemmas in crowd animation and music-driven motion synthesis.
- Strip depthwise convolutions over self-attention: Demonstrates that replacing heavy attention modules with anisotropic small-kernel strip convolutions effectively prevents overfitting in single-instance sequence generation.
Limitations & Future Work¶
- Foot-skating artifacts: Like many diffusion-based kinematic synthesis approaches, the absence of explicit foot-ground inverse kinematics (IK) contact constraints can occasionally cause minor foot-sliding artifacts during locomotion.
- Static mode count assumption: The number of modes \(K=8\) is fixed globally, whereas highly dynamic non-periodic stunts and simple cyclic gaits may have different optimal mode counts.
- Cross-skeleton spectral dynamics: Exploring how spectral energy distributions generalize across diverse topological morphologies (e.g., quadrupedal animals vs. bipedal characters) remains an open research direction.
Related Work & Insights¶
- vs GANimator [16]: GANimator uses a hierarchical progressive synthesis architecture that requires lengthy training per sequence and often suffers from motion freezing in long sequences; CMDer achieves higher smoothness (-7.15 vs. -7.20) and stable long-sequence continuation via full-convolutional diffusion.
- vs GenMM [18]: GenMM relies on training-free motion patch matching, yielding high coverage but severely restricted diversity (GloDiv 0.34); CMDer delivers balanced coverage (97.94) and substantially higher global diversity (1.32).
- vs SinMDM [21]: SinMDM operates strictly in the spatio-temporal domain with narrow receptive fields, frequently producing temporal acceleration discontinuities; CMDer operates in smoothed mode space, achieving higher composite harmonic mean (0.78 vs. 0.76) and more consistent physical velocities.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ First framework combining multivariate variational mode decomposition with diffusion for frequency-controllable motion synthesis.
- Experimental Thoroughness: ⭐⭐⭐⭐☆ Comprehensive quantitative benchmarking on Mixamo and Truebone, accompanied by ablations and user perceptual studies.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear mathematical formulations, intuitive architectural visualizations, and rigorous logical progression.
- Value: ⭐⭐⭐⭐☆ Highly impactful for character animation, game development, and data-efficient generative modeling.