InfiniteDance: Scalable 3D Dance Generation Towards in-the-wild Generalization¶

Conference: ECCV2026
Paper: ECCV Paper
Project: InfiniteDance
Area: Audio & Speech
Keywords: music-driven dance generation, human motion restoration, continuous embeddings, retrieval-augmented generation, cadence mixture of experts

TL;DR¶

InfiniteDance turns online dance videos into 100.69 hours of physically and foot-restored 3D motion, then trains ChoreoLLaMA with continuous embeddings, reference-dance retrieval, and cadence experts, reducing kinematic FID from Lodge's 89.52 to 30.54 on its dataset, while slow-music evaluation and long-sequence structure remain limitations of real-world generalization.

Background & Motivation¶

Music-driven dance generation must jointly handle musical structure, movement style, and body-ground contact. Beat alignment is only one requirement: movements can hit every beat yet exhibit persistent foot sliding, arbitrary arm gestures, or an expression inappropriate for slow music. Diffusion methods such as EDGE and FineDance generate short clips, while Lodge uses a coarse-to-fine structure for longer dances, but its handcrafted priors for energetic movement do not necessarily suit slower genres. Discrete-motion methods such as Bailando depend on music-motion sequence correspondence, and low-level music inputs may not sufficiently describe overall choreographic structure. These issues prevent benchmark performance from directly representing behavior under arbitrary musical inputs.

Expanding training data seems straightforward, but acquisition methods impose constraints. Professional motion capture provides accurate motion, yet equipment and performer costs limit scale; monocular online videos are abundant but lack reliable 3D contact supervision. Sliding, floating, and penetration in reconstructed motion are not incidental noise: they can become movement patterns learned by the generator. Physical simulation removes some implausible contacts, but inaccurate ground-friction estimates can turn sliding into leg jitter. What needs to scale is therefore not unfiltered video volume, but paired music and 3D motion that preserve choreographic expressiveness.

The paper treats data and generation as one system: repair video-derived contact artifacts, then let the model use existing choreographic priors to interpret unfamiliar music. Retrieval provides reference motion structure, while frequency experts prevent fast-tempo examples from dominating every generation. Here, "in-the-wild" refers both to online data sources and unfamiliar-music evaluation; it does not mean that all real-world situations have been solved. Core Idea: support training with scalable, physics-aware motion acquisition, then jointly condition an autoregressive choreographer on continuous music features, retrieved dances, and distinct motion-frequency bands.

Method¶

Overall Architecture¶

The system has an offline data-preparation path and an online music-conditioned generation path. Offline, single-person dance videos undergo whole-body motion estimation, physical imitation, and foot restoration to produce InfiniteDance data with music, genre, hand, and facial information. Online, music and an optional genre are embedded, reference motions are retrieved from the training set, and Cadence-MoE fuses the conditions before ChoreoLLaMA predicts motion codes. Codebook lookup and motion decoding recover 3D dance, and FRDM can additionally restore the feet of generated sequences. FRDM is consequently not an expert inside LLaMA, but a motion-restoration model used on both the data and output sides.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Video["Single-person dance videos"] --> Repair["Physics-Aware Acquisition<br/>and Foot Restoration"]
    Music["Music and optional genre"] --> Embed["Continuous Embedding Interface"]
    Repair -->|Paired motion and music| Embed
    Embed --> Retrieve["Training-Set Dance Retrieval"]
    Repair -->|Training reference database| Retrieve
    Retrieve --> Experts["Cadence Mixture of Experts"]
    Embed -->|Music and genre conditions| Experts
    Experts --> Generate["Autoregressive LLaMA prediction<br/>Codebook lookup and motion decoding"]
    Repair -.->|Training motion supervision| Generate
    Generate --> Output["FRDM post-processing<br/>3D dance output"]

Key Designs¶

1. Physics-Aware Acquisition and Foot Restoration: address contact artifacts before scaling data

The acquisition pipeline uses YOLOv8 to obtain single-person clips, then recovers body motion with the gravity-aware GVHMR model. SMPLest-X supplies hand and expression parameters, so the dataset is not restricted to a coarse body skeleton. Estimated motion becomes the imitation target in a physical environment, whose constraints reduce penetration, floating, and sliding. Table 7 labels this stage PHC; it substantially reduces penetration but introduces pronounced leg jitter. The authors do not simply smooth the entire body, because smoothing can remove dance details and disrupt established foot contacts.

The Foot Restoration Diffusion Model, FRDM, adjusts only the root, knees, and feet while preserving the upper body. It uses a 259-dimensional per-frame representation comprising root motion and local joint velocities, positions, and rotations for 22 body joints. This is consistent with the overall dataset providing 55 joints plus hand and facial information: restoration operates on the body subset it needs to modify. Root features contain yaw angular velocity, planar linear velocities, and root height. Positions and velocities directly describe whether a foot stays stable on the ground, while rotations preserve the geometry of the original pose.

Training samples high-quality MoCap motion, replaces only root, knee, and foot features with their noisy versions, and asks the denoising network to recover clean motion. This self-supervised corruption process does not require manually locating errors in every online video, but it still depends on external high-quality motion for training. Inference likewise retains the remaining body regions as input conditions and updates only the restoration regions at each diffusion step. Early steps use geometric guidance to keep the reconstruction consistent with the original motion; final steps use foot-contact guidance to stabilize contact. The training foot loss and staged inference guidance serve different roles, rather than merely increasing a single foot-sliding penalty.

2. Continuous Embedding Interface: expose music and motion detail to the language model

A pretrained MuQ encoder extracts music features, which are linearly projected to LLaMA's input dimension. Motion uses a three-layer residual vector-quantized variational autoencoder, RVQ-VAE, with successive codebooks representing residuals unexplained by earlier layers. Each codebook has 512 entries with 1024-dimensional quantized embeddings; the projected LLaMA input dimension is 2048. The distinction is not that discrete codes disappear, but that code indices are not simply passed through a newly learned embedding table. The model looks up the trained motion codebooks, retrieves their continuous quantized embeddings, and organizes and projects these multilayer features into motion conditions.

This preserves local motion structure learned by the tokenizer and retains continuous information from MuQ on the music side. An index is a symbolic identifier and does not directly expose the existing feature relationships between motion codes. The authors argue that reusing these representations supports fine-grained music-motion correspondence, with the input-interface ablation in Table 4 providing evidence. However, the experiments do not compare every possible discrete embedding training strategy, so they do not establish that discrete representations are intrinsically unable to encode rhythm. The prediction targets remain motion token indices, which return to continuous representations through the same codebooks before motion decoding.

3. Training-Set Dance Retrieval: provide composable motion references for unfamiliar music

MD-Retrieval follows a CLIP-like dual-encoder design, mapping music and dance into comparable feature spaces. Its music and motion encoders use efficient attention and learn cross-modal matching with InfoNCE on the InfiniteDance training set. During both ChoreoLLaMA training and inference, the system retrieves the 10 most relevant reference dances from the training split. The reference database is not a collection of test motions, an important boundary when interpreting results and leakage risks. Reference motions undergo projection and weighted aggregation to supply priors for fusion; the final dance is not produced by directly concatenating retrieved clips.

Users may specify a genre, or the system may use the genre associated with the retrieved dances. The paper does not clearly specify how conflicting reference genres are aggregated, so this should not be described as majority voting. Retrieval offers established choreographic patterns for rare music instead of requiring the generator to infer every bodily movement from music features alone. Benefits under lower-similarity retrieval and feature-space separation between generated and reference motions support the interpretation that the model uses priors rather than directly copies. Such distance analyses nevertheless cannot strictly rule out duplicate videos, closely related choreography, or subtler train-test leakage.

4. Cadence Mixture of Experts: specialize in motion frequencies rather than impose one rhythm rule

Cadence-MoE applies a real-valued fast Fourier transform along time to the aggregated reference motion and divides its spectrum into frequency bands. Values outside each band are zeroed, and an inverse transform recovers the corresponding temporal component for its specialist expert. Each expert contains linear layers, multi-head attention, and a Mamba block to model motion structure within that band. Music features and learnable genre embeddings repeated along time participate in conditioning; a linear-plus-softmax gate predicts expert weights. The standard configuration uses 2 frequency bands for relatively smooth, slow variation and faster motion changes, rather than assigning every dance genre to one of two hard categories.

The decomposition concerns the temporal frequencies of reference motion, not a direct equation between raw audio frequency and dance speed. Low-frequency body trajectories and high-frequency details can jointly form a movement, with the gate changing their influence according to the conditions. The intended effect is to reduce bias toward fast-tempo training examples without a handcrafted threshold forcing the model to dance quickly or slowly. The fused representation conditions ChoreoLLaMA, initialized from LLaMA3.2-1B, to predict motion codes autoregressively. Each generated code is looked up and projected into a continuous motion embedding for subsequent prediction, preserving the same interface during generation.

A Worked Example¶

Consider a slow music input with a user-specified genre; this illustrates the mechanism and is not an additional experiment. MuQ first extracts music features, and MD-Retrieval obtains 10 reference dances from the training set. Frequency decomposition sends these references to 2 experts operating at different temporal scales, while the gate uses music and genre to produce fused conditions. ChoreoLLaMA predicts motion codes sequentially, returning each predicted code to a codebook embedding for continued generation rather than outputting natural-language movement descriptions. After decoding, if the feet still jitter during ground contact, FRDM can repair the root and lower limbs without rewriting the upper-body performance. The example also shows why retrieval and restoration are not interchangeable: the former supplies choreographic priors, while the latter improves contact quality after generation or reconstruction.

Loss & Training¶

FRDM training uses MotoricaDance, FineDance, DD100, and InterDance, retargeted to a standard SMPL-X body shape. Its objectives include motion reconstruction, root reconstruction, displacement constraints on contacting feet, and consistency between integrated velocities and predicted positions. The legible velocity-position consistency term in the paper is:

\[ \mathcal{L}_{vp}=\left\|\operatorname{cumsum}(\hat{\mathbf{j}}^v_0)-\hat{\mathbf{j}}^p_0\right\|_2^2. \]

An epsilon-insensitive constraint also allows a small discrepancy between positions obtained through forward kinematics from rotations and directly predicted positions. The tolerance reflects their different roles: rotations preserve geometry, whereas positions and velocities more directly support contact restoration. Foot contact depends on both velocity and height thresholds; reported settings are tolerance 0.1, velocity threshold 0.001, toe-height threshold 0.05, and ankle-height threshold 0.08. These are the paper's implementation values; the relevant text does not explicitly establish units for every threshold, so no unit conversion is added. Cached Equations (3)–(7) contain visibly broken expressions, especially guidance mixtures and contact masks; only the verifiable mechanism is explained here, without reconstructing exact algebra.

RVQ-VAE is trained for 24 hours on one A100; ChoreoLLaMA uses batch size 8 and learning rate \(3\times10^{-4}\). Inference uses temperature 0.85, top-k sampling with \(k=30\), and top-p sampling with \(p=0.8\); sampling k is distinct from retrieving 10 reference motions. InfiniteDance uses an 85% / 5% / 10% training, validation, and test split with consistent genre distributions. The general setup in Section 5.1 combines public datasets using their original splits, but Table 3 explicitly trains both compared models on InfiniteDance; its cross-dataset results should follow that table-specific protocol.

Key Experimental Results¶

Main Results¶

The following selection from Table 2, page 12, compares generated motion on InfiniteDance; lower FID is better, FSR measures sliding during foot contact, and BAS measures music-motion beat alignment. FIDk and FIDg use kinematic and geometric features, respectively; Divk and Divg measure diversity in the corresponding feature spaces, so higher diversity does not automatically mean better dancing.

Method	FIDk ↓	FIDg ↓	FSR ↓	Divk ↑	Divg ↑	BAS ↑
Bailando	117.38	82.37	15.56%	5.46	5.28	0.2137
EDGE	96.07	63.53	14.15%	4.36	4.97	0.2321
FineDance	94.39	62.29	13.53%	4.42	4.85	0.2318
Lodge	89.52	60.38	6.72%	3.93	5.00	0.2329
ChoreoLLaMA	30.54	16.31	5.33%	6.23	5.11	0.2342

The authors explicitly retrain EDGE, Lodge, and Bailando, while acknowledging that their default configurations may not suit the new dataset's scale and complexity. In the unseen-music OOD evaluation of Table 3, page 13, ChoreoLLaMA achieves FIDk 56.22 versus Lodge's 119.66, but BAS is 0.2315 versus 0.2332, so it does not lead on every metric. This OOD setting includes BPMs outside the training range and rare instruments or styles, rather than constituting a random evaluation over all online music.

Ablation Study¶

The following results come from Table 4, page 13, with components added progressively on the InfiniteDance test set; this is not a full factorial evaluation of every component combination.

Config	FIDk ↓	Divk ↑	BAS ↑
Token indices, no RAG or MoE	79.84	13.09	0.2073
Continuous embeddings, no RAG or MoE	62.87	5.49	0.2269
Continuous embeddings + RAG	38.74	6.16	0.2325
Continuous embeddings + RAG + MoE	30.54	6.23	0.2342

To separate data quality from generation quality, the next table selects the acquisition analysis from Table 7, page 15; Jitter follows the reported values because the available main text provides no reproducible definition or unit.

Acquisition or restoration method	FSR ↓	Jitter ↓	Penetration rate ↓
FineDance, professional MoCap	6.22%	12.69	0.6954%
GVHMR	28.63%	31.89	0.7864%
GVHMR + PHC	8.87%	78.60	0.0536%
GVHMR + PHC + Smooth	14.29%	15.39	0.0561%
GVHMR + PHC + FRDM	5.09%	14.33	0.0559%

Key Findings¶

Continuous embeddings reduce FIDk and improve BAS even as Divk drops from 13.09 to 5.49, indicating that high diversity can also reflect unstructured motion.
RAG reduces FIDk from 62.87 to 38.74, and adding MoE further reduces it to 30.54; the progressive ablation supports complementary contributions without isolating every interaction.
Acquisition-side FRDM lowers both sliding and jitter relative to ordinary smoothing, but Jitter 14.33 remains above professional MoCap's 12.69, so it does not surpass MoCap on every quality metric.
Section 5.4 separately reports generated-motion FSR falling from 8.89% to 5.33% after FRDM; this is a different evaluation from the acquisition pipeline's 5.09% and must not be conflated.

Highlights & Insights¶

Data restoration and generation modeling reinforce one another. Removing learnable contact errors during acquisition and residual artifacts after generation addresses more of the system than increasing generator capacity alone.
A pretrained codebook is an input representation source, not just a compression dictionary. This interface is relevant to motion tasks that predict discrete targets while retaining continuous local structure.
Frequency experts operate on reference motion rather than predefined genre rules. Organizing experts by signal timescale and letting conditioning determine their mixture is a transferable idea, although new-task experiments would still be needed.

Limitations & Future Work¶

The authors acknowledge that BAS poorly evaluates slow dances: slightly faster, easily beat-aligned motion can inflate the score without establishing superiority over human choreography.
Following music becomes less reliable under long silence or abrupt genre changes; retrieval mainly captures within-clip similarity, which also limits long-sequence structure.
Data acquisition prioritizes stable cameras and fully visible dancers, limiting demonstrated coverage of difficult cases such as heavy occlusion and rapid camera motion.
Table 1 and the contribution summary report 30 genres, while Section 3.2 reports 6 major categories and 33 fine-grained subcategories; their relationship is unexplained and remains a statistical uncertainty.
Table 5's Top-10 row differs from Table 4's RAG row in Divk and BAS despite both reporting FIDk 38.74; the tables are not merged into one configuration or silently corrected here.
The paper demonstrates results with expanded data and a larger backbone but lacks a systematic model-capacity scaling curve; scalability is not an established scaling law.

vs Lodge: Lodge organizes long dances with coarse-to-fine structure and choreographic priors; this paper uses retrieval and frequency experts for cross-music adaptation, while explicitly retaining long-range structure as a weakness.
vs Bailando: Both involve motion tokens, but this paper emphasizes continuous music and codebook embeddings plus reference-motion conditioning, rather than abandoning discrete generation.
vs GVHMR / PHC: GVHMR estimates body motion from video and PHC supplies physical constraints; FRDM repairs foot jitter after physical imitation without replacing their whole-body estimation and physical processing.
Research direction: Extend clip retrieval to phrase-level structural planning and separately evaluate slow-dance expressiveness, long-range choreographic consistency, and contact quality; this is a reader proposal, not a completed experiment in the paper.

Rating¶

Novelty: 4/5. Physical data restoration, retrieval, and frequency experts form a concrete combination, although many underlying modules have precedents.
Experimental Thoroughness: 4/5. Generation, generalization, and acquisition ablations are covered, but baseline tuning fairness, scaling analysis, and some metric definitions remain incomplete.
Writing Quality: 3/5. The overall approach is clear; genre counts and cross-table values require clarification, while damaged cached equations additionally limit exact inspection.
Value: 4/5. The work offers practical lessons for scaling music-motion pairs, controlling physical quality, and designing conditional generation interfaces.