MEGADance: Mixture-of-Experts Architecture for Genre-Aware 3D Dance Generation

Conference: NeurIPS 2025 arXiv: 2505.17543 Code: To be released (upon acceptance) Area: 3D Dance Generation / Speech & Audio Keywords: Music-driven dance generation, Mixture-of-Experts (MoE), Mamba-Transformer, Finite Scalar Quantization (FSQ), style-controllable

TL;DR

This paper proposes MEGADance, the first music-driven 3D dance generation method based on a Mixture-of-Experts (MoE) architecture. It decouples choreographic consistency into "dance universality" (Universal Expert) and "style specificity" (Specialized Expert), combined with FSQ quantization and a Mamba-Transformer hybrid backbone, achieving state-of-the-art dance quality and strong style controllability.

Background & Motivation

Background: Music-driven 3D dance generation is divided into one-stage methods (direct mapping) and two-stage methods (discrete choreographic unit quantization followed by conditional generation). Two-stage methods yield better biomechanical plausibility by leveraging real motion priors.

Limitations of Prior Work: - VQ-VAE quantization suffers from codebook collapse (only ~75% utilization) - Style information is treated merely as a weak auxiliary bias (e.g., feature addition, cross-attention), leading to music–motion asynchrony and style discontinuity - Complex rhythmic transitions may cause cross-style motion contamination (e.g., Uyghur-style movements mixed into breaking sequences)

Key Challenge: Simultaneously maintaining universal dance quality across styles and style-specific precision within each genre is inherently conflicting.

Goal: Elevate style from an auxiliary modifier to a core semantic driver.

Key Insight: Drawing on the idea of parameter separation in MoE, assign independent experts to each style.

Core Idea: Decouple dance generation by modeling universality through a Universal Expert and capturing style specificity through Specialized Experts.

Method

Overall Architecture

Two-stage pipeline: - Stage 1 (HFDQ): High-Fidelity Dance Quantization — encodes dance motion into a discrete latent space (FSQ + kinematic/dynamic constraints) - Stage 2 (GADG): Genre-Aware Dance Generation — maps music to latent representations (MoE + Mamba-Transformer backbone)

Key Designs

1. Finite Scalar Quantization (FSQ):

   • Function: Replaces the traditional VQ-VAE codebook to eliminate codebook collapse
   • Design Motivation: The argmin selection in VQ-VAE leads to asynchronous updates and low utilization (~75%)
   • Mechanism: Replaces discrete argmin with differentiable bounded rounding: \(\hat{\mathbf{z}} = f(\mathbf{z}) + \text{sg}[\text{Round}[f(\mathbf{z})] - f(\mathbf{z})]\) where \(f(\cdot) = \text{sigmoid}(\cdot)\); each channel is quantized into \(L\) integers, yielding codebook size \(k = \prod_{i=1}^d L_i\)
   • Novelty: Achieves 100% codebook utilization (vs. 75% for VQ-VAE)

2. Kinematic-Dynamic Constraints:

   • Function: Augments SMPL parameter reconstruction with joint-level and temporal constraints
   • Design Motivation: Direct SMPL parameter reconstruction treats all joints equally, ignoring the kinematic tree structure of the human body (root errors propagate globally; hand errors remain local)
   • Mechanism: 3D joints are obtained via forward kinematics; position, velocity (\(\alpha_1\)), and acceleration (\(\alpha_2\)) are jointly constrained: \(\mathcal{L}_{\text{joint}} = \|\hat{J}-J\|_1 + \alpha_1\|\hat{J}'-J'\|_1 + \alpha_2\|\hat{J}''-J''\|_1\)

3. Mixture-of-Experts (MoE) Architecture:

   • Function: Decouples dance universality from style specificity
   • Specialized Expert: Each genre (Pop, Jazz, Breaking, etc.) is assigned a dedicated expert, activated via hard routing with style labels. Isolates genre-specific motion patterns (e.g., explosive Krump vs. fluid Contemporary) and introduces style-aware control priors
   • Universal Expert: Shared across all genres; learns low-level universal patterns such as beat synchronization, periodicity, and biomechanical consistency. Prevents modal mismatch that arises when using Specialized Experts alone (e.g., applying a Popping Expert to ballet music produces static or repetitive motions)
   • Design Motivation: Decoupling shared and style-specific factors allows each expert to specialize in a distinct subspace

4. Mamba-Transformer Hybrid Backbone:

   • Function: Combines Mamba's local dependency modeling with Transformer's global cross-modal understanding
   • Transformer component: Concatenates music, upper-body, and lower-body features along the time axis; employs a sliding-window attention mechanism (training–inference aligned)
   • Mamba component: Models intra-modal local dependencies for music, upper-body, and lower-body features separately
   • Sliding-window attention: Resolves the train–inference inconsistency of standard causal attention during long-sequence autoregressive inference

Loss & Training

• HFDQ stage: \(\mathcal{L}_{FSQ} = \mathcal{L}_{\text{smpl}} + \mathcal{L}_{\text{joint}}\) (incorporating position, velocity, and acceleration)
• GADG stage: Cross-entropy loss aligning predicted motion token probabilities with target pose codes
• Inference: Autoregressive generation for short sequences (≤5.5s); sliding-window concatenation for long sequences (5.5s overlap)

Key Experimental Results

Main Results

Comparison on the FineDance dataset:

Method	FID_k↓	FID_g↓	FID_s↓	DIV_k↑	BAS↑
Bailando++	54.79	16.29	8.42	6.18	0.213
FineNet	65.15	23.81	13.22	5.84	0.219
Lodge	55.03	14.87	5.22	6.14	0.218
MEGADance	50.00	13.02	2.52	6.23	0.226

On AIST++: FID_k=25.89, FID_g=12.62, BAS=0.238, all best-in-class.

User study (30 participants, 5-point scale): DQ=4.25, DS=4.30, DC=4.23, significantly outperforming all baselines.

Style Controllability Evaluation

Method	FID_s↓	DIV_s↑	ACC↑	F1↑
FineNet	13.22	4.29	42.06	37.44
Lodge	5.22	5.50	51.86	45.23
MEGADance	2.52	5.78	75.64	70.81
GT	0	6.07	78.31	76.35

Style classification accuracy approaches ground truth (75.64% vs. 78.31%).

Ablation Study

GADG stage ablation (FineDance):

Configuration	FID_k↓	FID_g↓	FID_s↓	BAS↑
w/o Specialized Expert	53.05	19.26	7.95	0.218
w/o Universal Expert	54.50	15.52	2.91	0.223
w/o Mamba	56.29	14.51	2.67	0.221
Full	50.00	13.02	2.52	0.226

HFDQ stage ablation:

Configuration	Joint MSE↓	Joint MAE↓
FSQ → VQ-VAE	0.0220	0.0842
w/o Kinematic Loss	0.0089	0.0507
w/o Dynamic Loss	0.0073	0.0482
Full	0.0069	0.0469

Key Findings

• The Specialized Expert is critical for style fidelity (removing it degrades FID_s from 2.52 → 7.95)
• The Universal Expert primarily improves motion structure and dynamic consistency (marked improvements in FID_k and FID_g)
• FSQ raises codebook utilization from 75% (VQ-VAE) to 100%, reducing joint MSE by 68%
• Generation speed: only 0.19 seconds of computation per second of output, suitable for real-time applications
• Even under cross-modal conflicts (e.g., Chinese music + Breaking style), beat synchronization and style fidelity are maintained

Highlights & Insights

• First application of MoE to dance generation: Style decoupling through structured inductive bias outperforms shallow fusion approaches
• Rationale for hard routing: Style labels are discrete; hard routing is more appropriate than soft routing and avoids blurring of style boundaries
• Training–inference alignment: The sliding-window attention mechanism elegantly resolves inconsistencies in autoregressive long-sequence inference
• FSQ as a replacement for VQ-VAE: Simple yet effective; 100% codebook utilization is a practice worth borrowing for other sequence generation tasks

Limitations & Future Work

• Style labels must be provided manually; automatic style recognition and label-free settings remain unexplored
• Text conditioning has not been incorporated (the authors note plans for this in the conclusion)
• Experiments are primarily conducted on street dance and Chinese dance datasets; generalization to other genres (e.g., ballet, contemporary dance) requires further validation
• MoE scalability — the number of experts grows linearly with the number of style categories, which may become a bottleneck

Related Work & Insights

• The two-stage paradigm (quantization + generation) originates from the Bailando/Bailando++ series; MEGADance improves upon both stages
• The Mamba-Transformer hybrid architecture echoes the recent trend toward efficient sequence modeling
• Insight: The MoE-based style decoupling idea is transferable to other conditional generation tasks (text stylization, music generation, etc.)

Rating

• Novelty: ⭐⭐⭐⭐ First application of MoE to dance generation combined with FSQ as a VQ-VAE replacement; combinatorial innovation is significant
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation on two datasets, user study, style controllability analysis, and detailed ablations
• Writing Quality: ⭐⭐⭐⭐ Method description is clear, ablation design is well-motivated, and visualizations are intuitive
• Value: ⭐⭐⭐⭐ A systematic solution for style-controllable dance generation with strong real-time performance and practical applicability

Related Papers

• [NeurIPS 2025] MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognition
• [ICCV 2025] Align Your Rhythm: Generating Highly Aligned Dance Poses with Gating-Enhanced Rhythm-Aware Feature Representation
• [NeurIPS 2025] Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers
• [NeurIPS 2025] Unifying Symbolic Music Arrangement: Track-Aware Reconstruction and Structured Tokenization
• [AAAI 2026] TEXT: Text-Routed Sparse Mixture of Experts for Multimodal Sentiment Analysis with Explanation Enhancement and Temporal Alignment