OmniDance: Multimodal Driven Dance Video Generation with Large-scale Internet Data

Conference: ECCV 2026
Paper: ECCV 2026
Code: https://github.com/AMAP-ML/OmniDance
Area: Video Generation
Keywords: Music-driven dance, multimodal control, Diffusion Transformer, curriculum learning, classifier-free guidance

TL;DR

OmniDance uses approximately 300k Internet dance clips and coordinated music injection, curriculum training, and modality-specific sampling to extend Wan2.2-TI2V-5B into a unified text/music/jointly driven dance video model, improving overall quality and controllability on the CIPE-Dance test set without making joint conditioning best on every metric.

Background & Motivation

Making a person in a reference image dance to music requires more than expanding the motion range of a talking-head model. Dance demands coordinated arms, torso, and legs, converts beats, tempo, and musical energy into full-body movement, and must preserve identity, clothing, and background. Existing two-stage approaches first predict 2D keypoints or 3D SMPL parameters from music and then synthesize images, making their success depend on reliable intermediate poses; occlusion, complex displacement, and expensive 3D supervision constrain the motion repertoire they can learn. General video foundation models offer stronger visual priors, but their predominantly text/image-based training does not make them understand music-dance relationships simply by accepting an audio input.

Data and condition fusion are therefore two sides of the same problem: insufficient natural dance footage restricts motion learning, while poorly separated conditioning roles can let a new music branch disrupt text control and appearance quality. The authors treat text as a carrier of global semantics such as genre, choreographic intent, and key poses, and music as a carrier of localized temporal cues such as rhythm, tempo, and energy changes. This division is a design hypothesis for architecture and training, not a rigorous signal decomposition proving that text contains only low frequencies and music only high frequencies. The goal is also not to require all three inputs at all times, but to support text plus image, music plus image, and their joint combination within one generator.

The paper therefore first builds CIPE-Dance through progressive filtering and choreography-oriented descriptions, then schedules music integration across model depth, training stages, and sampling progress. The first two make music a usable motion condition; the last determines when an individual generation should rely more on semantics or rhythm. Core idea: retain the text/image foundation model as a semantic and appearance anchor, and introduce music through stronger deep-layer contributions, easy-to-hard training, and relatively stronger late-stage guidance instead of letting all conditions compete equally everywhere.

Method

Overall Architecture

OmniDance always receives a reference image, optionally accompanied by text, music, or both, and outputs a dance video that preserves the reference person's appearance. TI2V, MI2V, and MTI2V denote these three combinations, so “music-only driving” does not mean that a person image is unnecessary. The system builds on the latent video generator Wan2.2-TI2V-5B, encodes text with umT5-XXL and music with MERT, and feeds music features into the DiT through added cross-attention. The foundation model's 3D VAE handles the reference image and video latent representations, and VAE decoding produces pixel-space video; training does not first require prediction of an explicit skeleton sequence.

The diagram combines offline data construction, model adaptation, and sampling. Its arrows show dependencies, not a requirement to repeat data filtering or training at every inference call.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Internet dance videos"] --> B["Choreography-Informed Data Construction"]
    B --> C["Music-Text Progressive Specialization"]
    C --> D["Anchored Curriculum Training"]
    D --> E["Modality-Specialized Time-Dependent Guidance"]
    F["Reference image<br/>text or music or both"] --> E
    E --> G["Dance video<br/>VAE decoding and sliding-window extension"]

Key Designs

1. Choreography-Informed Data Construction: turn web videos into supervision for controllable generation

CIPE-Dance starts with public videos from approximately 500 dance creators, requires more than 50K followers per creator, and restricts publication dates to after 2018. Videos are split into 5-second clips at 16 FPS, initially yielding approximately 630k clips, and then pass visual quality, reference clarity, dance content, single-dancer, and scene stability filters. The prose on page 6 specifies IQ above 60.00 and AQ above 50.00; Figure 2 swaps these threshold labels, so this note follows the prose rather than treating the figure as another verified configuration. Because the first frame becomes the generation reference, clarity checks remove poorly visible subjects, back-facing poses, and missing facial cues, while dance verification removes talking, posing, and nearly static clips. These conditions serve generation directly: the reference must reveal identity, and the target must contain movement rather than merely dance-related visual context.

When directly deciding whether a clip contains one dancer is difficult, the pipeline instead searches for concrete negatives such as group dancing, follow-along picture-in-picture videos, and mirror reflections, then rejects abrupt scene changes and aggressive camera jitter. According to the detailed account in Section 3.1, reference, dance, and single-dancer checks use Qwen3-VL-2B, while scene stability uses Qwen3-VL-8B; not every difficult filter should be described as using the same larger model. Qwen3-VL-8B then independently describes retained clips through body dynamics, choreographic content, expressiveness, camera presentation, and overall look. Text consequently specifies moving body parts, genre vocabulary, performance intent, framing, clothing, and environment rather than merely stating that someone is dancing, giving the text branch grounded dance semantics to learn first. The final dataset contains approximately 300k clips, over 400 hours, and more than 30 dance genres; these are paper-reported scales, not an independently verified release status.

2. Music-Text Progressive Specialization: progressively amplify music residuals in deeper layers

The backbone contains 30 DiT blocks, each originally comprising self-attention, text cross-attention, and a feed-forward network; added music cross-attention follows text conditioning. Music features come from MERT, and their residual updates are not injected equally into every block but multiplied by a coefficient that increases with depth. The depth coefficient in the mechanism of Equation (3), page 9, can be isolated as:

\[ s_\ell=\left(\frac{\ell}{L}\right)^\gamma,\qquad L=30,\quad\gamma=1.5. \]

Here \(\ell\) is the block index and \(L\) is the number of blocks; each music cross-attention residual is multiplied by \(s_\ell\) before being added to the latent features. Shallow layers are therefore less disturbed by the new branch, allowing text to establish the overall action layout, while deeper layers receive stronger music contributions to refine rhythm-related motion. This is soft scaling rather than a hard partition: shallow layers still receive music, and deeper layers retain text conditioning. “Specialization” does not mean generating a text-driven video and asking a second generator to repair its rhythm; it changes conditional contributions across the depth of a single DiT.

The design exploits the backbone's coarse-to-fine representations, but its effectiveness requires an intervention-based check. Reversing the depth schedule in Table 2 reduces BAS to 0.234 from 0.287 with the normal schedule, supporting the specific choice of stronger deep-layer music rather than arbitrary layer scheduling. Replacing MERT with Wav2Vec2.0 also weakens beat alignment while improving some appearance metrics, showing that music representation quality depends on the objective and cannot be judged from attractive images alone.

3. Anchored Curriculum Training: learn dance text first, joint conditions next, and missing text last

Stage I trains only TI2V with the music branch frozen, adapting the general generator to the relationship between dance descriptions and video. It does not train a new captioner; it makes the generator respond more reliably to action and choreography descriptions, creating a semantic anchor for subsequent music learning. Stage II retains both TI2V and MTI2V, introducing music while text can already explain the motion; continued TI2V training also constrains the existing capability instead of switching all training immediately to audio. The new music residual projection is zero-initialized, and new parameters receive linear learning-rate warm-up so that an untrained music branch does not strongly alter backbone outputs at the start.

Stage III finally trains TI2V, MTI2V, and MI2V together, requiring the model to organize dance from music even when text is absent. This reduces dependence on text prompts without removing text capability, and the final model can switch between all three modes. The paper describes this phase as jointly optimizing all relevant parameters but does not provide a sufficiently detailed parameter-group list or three-task sampling ratio; specific freezing rules or mixture probabilities should not be added. The stages retain the same kind of video generation objective and change conditioning combinations and training order, rather than introducing a newly reported explicit beat-reward loss.

4. Modality-Specialized Time-Dependent Guidance: increase music's relative late-stage influence, not its absolute weight

Inference constructs predictions with neither text nor music, with text only, and with text plus music, and then combines conditional increments; the reference image remains present. The MTI2V branch of Equation (7), page 10, is restated below in shorthand, where \(v_0\) is the velocity prediction without text or music, \(v_T\) uses text only, and \(v_{TM}\) uses both:

\[ v_{\mathrm{MTI2V}}=v_0+\lambda_t(\tau)(v_T-v_0)+\lambda_m(\tau)(v_{TM}-v_T). \]

The second difference measures the effect of adding music while retaining text, rather than subtracting an unconditional prediction from a separate music-only prediction. TI2V uses only the text-versus-unconditional increment, whereas MI2V uses the music-only-versus-unconditional increment, giving absent modalities explicit sampling paths. Here \(\tau\) denotes progress along sampling and should not be confused with the direction of the training noising time. The actual settings decay text guidance from 5 to 1 and music guidance from 4 to 2; both absolute coefficients decrease, but text decreases faster, making music relatively more influential later. This explains the paper's “text early, music late” description without incorrectly turning it into an increasing music CFG schedule.

Long video generation follows a Wan-style sliding window, using the current clip's final frame as the reference image for the next window and repeatedly generating short clips. This provides sequential extension rather than a separately specified global choreography planner; passing only the final frame cannot itself guarantee freedom from long-term identity, rhythm, or action-narrative drift.

Loss & Training

The backbone uses flow matching: a linear path connects clean video latents and Gaussian noise, and the DiT learns the path's target velocity by minimizing velocity prediction error. Equations (1) and (2) are corrupted in the local text extraction, so this note retains the training interpretation supported by the prose without inventing the noising direction or full conditioning subscripts. The VAE compresses time, height, and width by \((4,16,16)\), respectively; music and text condition latent generation, and the decoder reconstructs the video. Page 11 reports 5k, 5k, and 10k steps for Stages I, II, and III, batch size 32, AdamW, and a learning rate of \(1\times10^{-5}\). Training uses \(704\times1280\) resolution and 77 frames, also described as 5 seconds at 16 FPS; the original reporting is retained rather than silently replacing 77 with exactly 80 frames. Inference uses 40 flow-matching sampling steps; full training takes approximately 70 hours on 32 H20 GPUs, with about 80GB memory per GPU. On one H20, generating a nominal 5-second clip takes approximately 4.5 minutes for TI2V/MI2V and 7.2 minutes for MTI2V, using approximately 40GB memory.

Key Experimental Results

Main Results

Evaluation uses 100 randomly held-out CIPE-Dance clips that the authors exclude from training; see pages 7 and 11–12 for the setup and Table 1 on page 11 for the main results. IQ, SC, and MS measure imaging quality, subject consistency, and motion smoothness; BAS scores alignment between musical and motion beats, with higher values preferred, but is not equivalent to overall choreographic quality. Motion FID measures feature-distribution differences from ViTPose-extracted 2D keypoints; \(\mathrm{FID}_k\) uses kinetic features and is lower-is-better, not an error against ground-truth 3D motion capture. OC measures overall text-video consistency; MI2V uses the simplified evaluation text “a person is dancing,” so its semantic constraint is not identical to that of detailed-text tasks.

Condition / Method	IQ ↑	SC ↑	MS ↑	FIDk ↓	BAS ↑	OC ↑
TI2V / Wan2.2-TI2V-5B	64.66	90.10	98.08	19.15	0.252	12.11
TI2V / OmniDance	67.74	90.16	99.24	13.55	0.259	12.53
MI2V / Hallo2	64.09	90.77	97.05	16.49	0.244	11.85
MI2V / OmniDance	65.61	92.63	98.09	17.55	0.293	12.40
MTI2V / OmniDance	67.77	94.34	99.22	13.97	0.287	13.08

TI2V improves FIDk from the backbone's 19.15 to 13.55, but MI2V's FIDk of 17.55 does not outperform Hallo2's 16.49. Joint conditioning improves subject consistency and OC without surpassing MI2V on BAS; it should be interpreted as a combined improvement across quality dimensions, not dominance over every single-modality result. Hallo2 receives full-body mask adaptation and fine-tuning, while the paper also compares WAN-S2V and Echomimic-V3 through direct inference; adaptation conditions differ, so not all gaps can be attributed to architecture.

Ablation Study

The following combines the architectural ablations in Table 2 and training/inference ablations in Table 3 on page 14, all under MTI2V; references follow table captions because the architectural discussion incorrectly points to Table 3. AQ measures aesthetic quality; “shared CFG” gives text and music the same guidance weight decaying from 5 to 3.

Source / Config	AQ ↑	SC ↑	MS ↑	BAS ↑	OC ↑
Tables 2, 3 / Full model	55.76	94.34	99.22	0.287	13.08
Table 2 / Without depth scaling MTPS	54.18	90.61	93.74	0.257	12.53
Table 2 / Reversed depth schedule MTPS (R)	52.31	89.89	95.98	0.234	12.37
Table 2 / MERT replaced by Wav2Vec2.0	56.10	95.72	98.64	0.241	12.78
Table 3 / Without Stage I	52.92	92.74	94.88	0.239	11.72
Table 3 / Without Stage II	51.38	94.61	98.11	0.293	12.71
Table 3 / Shared CFG	53.27	93.96	96.84	0.279	13.22

Key Findings

• Removing depth scaling reduces MS from 99.22 to 93.74, indicating an effect on motion continuity as well as beat alignment; reversing the schedule further supports the importance of its direction.
• Removing Stage I hurts both quality and alignment, whereas removing Stage II raises BAS to 0.293 but lowers AQ to 51.38, supporting semantic/visual anchoring rather than claiming that every stage improves every metric.
• Modality-specialized CFG improves BAS and MS over shared CFG but changes OC from 13.22 to 13.08; this supports a multi-objective trade-off, not superiority in text consistency as well.

Highlights & Insights

• Filtering is designed around the generation-time reference image and single-dancer assumption rather than generic video quality alone. In particular, excluding mirrors and picture-in-picture examples reduces ambiguity about which person a reference image should animate.
• Applying a consistent division of conditioning roles across depth, curriculum, and sampling progress is a more complete contribution than adding audio cross-attention alone. Ablations test several choices without establishing this division as a universal law.
• Retaining TI2V as an anchor during music integration allows a new modality to be learned without deliberately abandoning existing capability. Opposing changes in synchronization and appearance also show why one beat score is insufficient for model selection.

Limitations & Future Work

• The authors explicitly acknowledge that generation is not real-time, and minute-scale latency limits interactive use; the conclusion suggests combining long-video generation with distillation but reports no completed acceleration result.
• The test set contains only 100 randomly held-out clips, with no stated split by creator, song, or source video and no confidence intervals. Strong generalization across people, music, and sources is therefore not established.
• Filtering favors single performers, clear front-facing references, and stable scenes, leaving group dance, back-facing starts, occlusion, and strong camera motion insufficiently tested; sliding-window extension also lacks long-duration quantitative results.
• Pose-estimation errors may affect 2D keypoint metrics, and automated choreography captions lack a full human accuracy audit. Threshold labels, table references, and frame-count conventions also need implementation-level verification rather than being settled by this note.

Related Work & Insights

• Compared with X-Dancer, M2PE-Diff, and STG-Mamba: these methods first generate 2D poses and then animate images, whereas OmniDance learns music conditioning directly in video latent space; it consequently lacks the same direct motion-editing interface as an explicit pose sequence.
• Compared with ChoreoMuse and MACE-Dance: intermediate 3D SMPL representations provide geometric structure, whereas this work uses Internet video scale to obtain broader appearance and motion examples, with different supervision costs and control mechanisms.
• Compared with Wan2.2-TI2V-5B: the paper does not build a new video backbone from scratch but studies how an existing generative prior can accept music. Retaining the old task while gradually adding a modality is a useful empirical result, although effectiveness on other tasks remains untested.

Rating

• Novelty: 4/5. The coordination of data, model depth, curriculum, and sampling is clear, although individual components are not entirely new.
• Experimental Thoroughness: 3/5. Three-task results and architectural/strategy ablations are substantial, but test size, adaptation differences, and limited long-video validation constrain the conclusions.
• Writing Quality: 3/5. The main argument is clear, but actual CFG coefficients require distinguishing relative influence from absolute growth, and several figure/text and table-reference details complicate reproduction.
• Value: 4/5. The work directly informs music integration into strong video foundation models, while real-time generation and complex multi-person scenes remain unresolved.