Music-to-Dance Generation via Atomic Movements¶

Conference: ECCV 2026
Paper: ECCV 2026
Code: https://github.com/oceanflowlab/AtomicDance
Area: Audio & Speech
Keywords: music-to-dance generation, atomic movement, discrete diffusion, structure-aware generation, editable choreography

TL;DR¶

The paper models dance explicitly as a sequence of semantically readable, reusable "atomic movements": a discrete diffusion model first plans the type, duration, and temporal position of every atomic movement conditioned on the full music (a symbolic dance score), and a transition-aware continuous diffusion generator then fills in the transitions, retrieves instances by duration-nearest matching and re-creates them under masked noise, achieving the lowest FID (25.26 / 9.03), the highest BAS (0.2470), and the highest structural consistency R (26.6) on AIST++.

Background & Motivation¶

Music-to-dance generation has to satisfy two things at once: the motion must be kinematically plausible and visually natural, and its rhythm and style must match the music. Recent years have largely been taken over by generative models — FACT autoregressively predicts joint trajectories with a cross-modal Transformer, Bailando and TM2D learn discrete motion codebooks with VQ-VAE / GPT to strengthen music-conditioned controllability, EDGE brings diffusion into dance generation with local editing and continuation, and LODGE and EDMG target long-sequence dance with coarse-to-fine or efficient generation. All of them share one premise: dance is a continuous signal, and the task is a direct music-frame-to-motion-frame sequence-to-sequence translation. What they are therefore good at is frame-level or short-term correspondence — which beat the foot lands on, whether a bar is locally rhythm-consistent — while the organization that real choreography depends on is discarded. The granularity problem shows up in two concrete places: each discrete code in a VQ codebook typically corresponds to a very short motion segment, so decoded outputs are essentially static poses with no process information; and diffusion-based methods are usually trained within a short temporal window, which constrains their receptive field so they cannot capture global choreographic structure unfolding over longer timescales, while their inpainting edits happen at the signal level with no symbolic interpretability.

The tension is this: choreography theory and music theory both stress that a performance is organized around discrete, meaningful, reusable units — the same movement reappears throughout a dance and changes in rhythm, dynamics, range, and direction, sped up or slowed down, enlarged or minimized, reversed, fragmented, and reassembled. In other words, the structure is already in the data, but none of the existing methods has a layer that explicitly states "what movement occupies this stretch, how long it lasts, when it appears, and where the connections are." Without that layer, generated dances lack segment-level coherence, and users cannot edit them in the form of "replace the second movement with a spin."

This paper's choice is to first mine that structure out of the data and then generate on top of it, rather than hoping the network learns it implicitly. The authors define an atomic movement as a basic motion process that is repeatable, comes with variation, and can be freely combined, and they ground it in three criteria: it must be a process rather than a single frame (a single pose fully describes one instant but is frozen in time, so it cannot express shifts in emotion or energy and cannot interact with the rhythm and phrasing of the music); it must recur with variation (this repetition-plus-variation is exactly what keeps a whole dance unified yet non-monotonous); and it must be describable in natural language, which enables retrieval and human-readable labeling. On this basis the authors design a two-stage framework: the first stage acts as the choreographer, predicting the type, duration, and position of atomic movements conditioned on the full music and emitting a symbolic dance score; the second stage acts as the dancer, realizing each movement according to that plan and generating the transitions between them. Core idea: split music-to-dance generation explicitly into atomic movement planning plus transition completion — a discrete diffusion model writes the symbolic score conditioned on the whole music, and a continuous diffusion model with masked noise fills the transitions and re-creates the movements, preserving motion naturalness while gaining segment-level structure, interpretability, and editability.

Method¶

Overall Architecture¶

Given a music condition \(M^{1:L}\) of length \(L\), the goal is to generate a dance sequence \(X^{1:F}\) aligned with the music, where \(F = L \times R_F\) and \(R_F\) is a fixed frame rate. The method is really two things stitched together: offline, it builds a vocabulary of atomic movements out of the dance data (there is no segment-level annotation anywhere in AIST++, so this vocabulary is the foundation for everything else); online, it generates in two stages — the first plans a sequence of atomic movement labels (each with a type, a starting position, and a duration) conditioned on the full music, and the second translates that symbolic score back into continuous motion, filling in the transitions and allowing each atomic movement to be re-created with variation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    A["AIST++ dance videos<br/>+ 3D motion"] --> D1
    subgraph D1["Atomic Movement Discovery"]
        direction TB
        S1["Segmentation: similarity<br/>matrix + adaptive cuts"]
        S2["Clustering: TMR encoding<br/>+ K-Means prototypes"]
        S3["Re-clustering: LLM labeling<br/>+ semantic splitting"]
        S1 --> S2 --> S3
    end
    D1 --> D2
    M["Input music (full track)"] --> D2
    D2["Atomic Movement Planning<br/>discrete diffusion score + post-process"] --> D3
    M --> D3
    subgraph D3["Dance Completion"]
        direction TB
        C1["Duration-nearest retrieval<br/>+ masked noise"]
        C2["Transition-aware diffusion<br/>fill transitions + re-create"]
        C1 --> C2
    end
    D3 --> O["Complete dance sequence"]

Notably, the music-to-motion correspondence is split across two levels. The symbolic level is established by the planner: it conditions on the whole music rather than a clip, deciding which atomic category occupies each moment, for how long, and which frames should be left as transitions, so it can align with phrase- and section-level structure rather than just beats. The signal level is established by the completion module: the denoising network conditions on music features and adds an explicit transition loss that smooths the seams at movement boundaries. Unlike end-to-end approaches that treat music merely as a beat-alignment signal or model it implicitly through short-window diffusion, here the music determines the symbolic arrangement of the motion sequence.

Key Designs¶

1. Atomic Movement Discovery: turning dance data into a reusable movement vocabulary via segmentation, clustering, and LLM re-clustering

AIST++ contains only whole dances paired with music and no segment-level annotation whatsoever, and existing VQ codebooks are so fine-grained that decoding yields little more than poses, so the authors have to build the vocabulary themselves. The first step is segmentation, whose goal is to cut continuous motion into units that are process-complete and distinguishable from their temporal context: a kick is a complete atomic process comprising the preparatory weight shift, the leg extension, and the recovery, whereas any single frame of the extended leg or a very brief leg movement fails to convey the full event. Following the zero-shot temporal event proposal scheme (Algorithm 1), the authors first extract per-frame visual features with an I3D encoder and compute a frame-wise cosine similarity matrix \(A\); each frame's similarity row \(c_t\) is concatenated with the normalized frame index before clustering, i.e.

\[\tilde{c}_t = [\,c_t;\ t/T\,]\]

Cut points are then placed at label transitions, and segments shorter than \(L_{\min}\) are iteratively merged with the shorter of their temporal neighbors. The intuition is that frames within one motion segment resemble each other and resemble frames from other segments less, so they share the same similarity pattern and get clustered together, which puts the cut points at action boundaries; appending the frame index suppresses spurious merges of similar frames that are far apart in time. Each video segment is then described by Gemini-2.5-Pro along two axes, a signature pose and movement dynamics, where the signature pose is produced by PoseScript from selected keyframes identified as motion beats (local minima of segment-wise joint velocities) and fed to the VLM as an auxiliary cue, yielding fine-grained natural language descriptions.

The second step is dataset-wide clustering, which finds the units that repeat with variation. Every segment is encoded by a pretrained TMR motion encoder and projected into the joint motion-text embedding space, where K-Means groups segments into recurring atomic movement prototypes. Because TMR was trained with text-motion contrastive learning on HumanML3D, whose captions emphasize high-level action descriptions rather than dance-specific detail, the encoder naturally groups segments that are high-level similar while retaining intra-class variation — exactly the property "prototypes preserve variation" requires. For quality, only segments near cluster centers are kept and ambiguous edge points are discarded, yielding 100 atomic movement prototypes over the complete AIST++ dataset, with 268.57 segments per prototype on average.

The third step is in-group re-clustering, which makes the prototypes interpretable. Each prototype is first pre-split by dance genre, since motions from different genres tend to be semantically distinct; then a summarizing LLM iteratively (i) identifies a subset of mutually similar captions as a sub-prototype and (ii) distills it into a concise semantic tag, repeating until the number of ungrouped segments falls below a threshold. On average each prototype yields 7.3 sub-prototypes with 31.8 samples each — preserving fine-grained diversity without severe data sparsity. What the language model contributes here is a merging criterion in the dimension of "semantically similar yet kinematically diverse," which makes the vocabulary closer to how humans understand the movements than purely kinematic clustering would.

2. Atomic Movement Planning: a discrete diffusion model writing a symbolic score conditioned on the full music

Conventional methods let the network regress dense motion frames directly from music, which captures local rhythm but never sees long-range composition. In choreography, however, the first decision is which movements to use, where to place them, and how long they last; this paper makes that step an explicit symbolic sequence prediction, assigning every frame either an atomic movement category or a token indicating that the frame belongs to a transition. Because the output is discrete, the authors adopt a Discrete Denoising Diffusion Probabilistic Model (D3PM) rather than continuous diffusion. With \(K\) atomic movement types numbered from \(0\) to \(K\), where \(0\) means the frame belongs to no atomic movement (i.e. it is a transition frame), the forward process gradually perturbs the true labels into random noise according to a transition matrix \(Q_t\):

\[q(y^t \mid y^{t-1}) = \mathrm{Cat}\big(y^t;\, Q_t[y^{t-1}, :]\big)\]

At each step a token is retained with probability \(\alpha_t\) or replaced with a uniformly random category with probability \(1-\alpha_t\), so after \(T\) steps the sequence approaches a uniform distribution. The reverse process is parameterized by a Transformer-based diffusion model conditioned on music features \(c_{music} = \mathrm{Enc}(M)\); at inference the model starts from a uniformly random sequence \(y^T\) and iteratively samples down to \(t=0\), producing the plan \(\hat{Y} = [\hat{y}_1, \dots, \hat{y}_F]\) that specifies both the category and the temporal arrangement of every motion event. Conditioning on the full music rather than a clip is the most fundamental difference from short-window methods: the planner sees phrase- and section-level structure, so it can allocate movements to appropriate positions and decide where transitions belong, which directly explains the gains in BAS and structural consistency. Moreover, because the intermediate representation is discrete and temporally grounded, users can modify a specific movement, adjust its duration, or reassemble sections before the second stage, without any retraining.

The authors also report a practical failure mode: any mislabeled frame inside a continuous atomic movement segment can severely disrupt the planning result. A lightweight post-processing module therefore follows the first stage — a sliding-window majority vote removes isolated frame-level errors and smooths local inconsistencies, and a minimum-duration merging heuristic detects abnormally short segments and reassigns them to the most semantically compatible neighbor. The two steps target over-segmentation and local jitter respectively, correcting errors while preserving genuine action boundaries.

3. Dance Completion: retrieved instances, masked noise, and transition-aware diffusion stitching and re-creating at once

The plan is only a skeleton — it specifies the categorical composition of the movements but neither realizes them nor connects neighboring ones — and choreographic theory holds that structured movements should exhibit variation when they recur. If one simply concatenated retrieved instances end to end under temporal rescaling, the motions would be faithful but diversity, beat alignment, and overall dance quality would collapse. The approach here is to retrieve, for each predicted label, the motion instance whose duration most closely matches the planned temporal length, rescale it to fit the target range, and place it into the corresponding frames; masked noise (with the ratio controlled by a mask \(w\)) is then applied to those chosen atomic movements so that denoising can rewrite them, while unallocated transition frames remain uninitialized — producing a coarse sequence \(M_0\) whose atomic segments are filled and whose transitional frames are empty. A DDPM denoiser \(f_\theta(m_t, t, x_{music}, M_0, w)\), conditioned on the music features, the draft sequence, and the mask, then predicts the clean signal \(m_0\) from \(M_0\), filling the empty transitions and re-creating the noised atomic segments into versions that better fit the music and cohere with their neighbors. The noise level acts as an explicit fidelity-versus-diversity knob: little noise means the retrieved instances are copied almost verbatim, faithful but monotonous, while more noise buys freedom to follow the music at the risk of drifting from the category semantics. Three retrieval strategies are compared — a fixed single instance suppresses diversity, random choice hurts both FID and structural consistency because it demands stronger temporal scaling, and duration-nearest retrieval gives the best balance across diversity, structural consistency, and rhythmic coherence. In addition, a transition loss constrains the two frames on either side of each movement boundary so that the seams do not jump.

Loss & Training¶

The first stage uses the standard D3PM variational objective, \(\mathcal{L}_{\text{D3PM}}\), minimizing the negative log-likelihood of recovering the perturbed labels given the music condition. The second stage follows the standard denoising loss, where the network predicts the clean signal \(m_0\) from \(m_t\):

\[\mathcal{L}_{diff} = \mathbb{E}_{t, m_0}\big\| f_\theta(m_t, t, x_{music}, M_0, w) - m_0 \big\|\]

On top of it sits a transition loss that penalizes discontinuities at every atomic movement boundary, taking the distance between the frames just before and just after the boundary:

\[\mathcal{L}_{trans} = \sum_{b \in \text{boundaries}} \big\| M_{b^-} - M_{b^+} \big\|_1, \qquad \mathcal{L} = \mathcal{L}_{diff} + \lambda_{trans}\mathcal{L}_{trans}\]

Both modules use a latent dimension of 512, 8 Transformer layers, 8 attention heads, a feed-forward dimension of 1024, and dropout 0.1; the completion module adapts the EDGE denoiser into a transition-aware version. Optimization uses AdamW with a learning rate of \(2\times10^{-4}\), weight decay 0.01, and gradient clipping 1.0. Experiments use AIST++, which contains 1,408 music-paired 3D dance motion sequences totaling 5.2 hours, covering 10 dance genres performed by 30 subjects with multi-view annotations.

Key Experimental Results¶

Main Results¶

Comparison with representative baselines on AIST++ (\(\uparrow\) higher is better, \(\rightarrow\) closer to Ground Truth is better). The method leads on FID, BAS, and structural consistency R, and its diversity is closest to real dance:

Method	FID_k ↓	FID_g ↓	Div_k →	Div_g →	BAS ↑	R ↑
Ground Truth	17.1	10.6	8.19	7.45	0.2374	42.1
DanceNet	69.18	25.49	2.80	2.85	0.143	14.1
DanceRevolution	73.42	25.92	3.52	4.87	0.195	13.7
Bailando	28.16	9.62	7.83	6.34	0.2332	17.9
EDGE	42.16	22.12	3.96	4.61	0.2334	16.3
LODGE	37.09	18.79	5.58	4.85	0.2423	18.2
Ours	25.26	9.03	8.01	6.69	0.2470	26.6

Among the metrics, FID_k / FID_g measure the distribution distance between generated dances and the corresponding ground truth in kinematic and geometry feature spaces; Div_k / Div_g are the mean pairwise Euclidean distance in those feature spaces, better when closer to ground truth; and BAS measures rhythmic synchronization between the generated dance and the input music. R (R-precision) is the authors' own metric for structural consistency: the music is split into segments whose features are computed, and one checks whether, for each pair of most-similar music clips, the motion features of the dances fall among the three most-similar dance clips, with the precision reported as R — ⚠️ the original paper states this definition only briefly, so the description here follows the source verbatim and details should be checked against the paper. MultiModality takes the variance across 5 samples generated from the same music to measure output diversity.

Ablation Study¶

Configurations and re-clustering for atomic movement discovery (Table 2):

Cluster num	Re-clustering	FID_k ↓	FID_g ↓	BAS ↑	R ↑
75	✗	33.24	13.31	0.2413	20.2
100	✗	32.68	12.09	0.2420	21.3
125	✗	34.57	14.28	0.2415	21.1
100	w/o LLM	30.11	11.27	0.2431	23.3
100	w/ LLM (full)	25.26	9.03	0.2470	26.6

Flexible re-creation versus fixed instances in dance completion (Table 4):

Config	FID_k ↓	FID_g ↓	Div_k →	Div_g →	BAS ↑	R ↑	MultiModality ↑
Ground Truth	17.1	10.6	8.19	7.45	0.2374	42.1	–
Fixed Atomic Movements	24.13	8.44	6.99	4.61	0.2356	27.2	1.43
Flexible Atomic Movements (full)	25.26	9.03	8.01	6.69	0.2470	26.6	2.25

⚠️ In Table 2 the "Re-clustering" column takes the values ✗ / w/o LLM / w/ LLM; this note reads them as "no re-clustering / re-clustering without LLM semantic tags / full method," but the exact configuration should be checked against the original paper.

Key Findings¶

All three designs contribute, and the language prior contributes the most. At the same 100-cluster setting, adding LLM-assisted re-clustering lowers FID_k from 30.11 to 25.26 and raises R from 23.3 to 26.6, showing that merging semantically similar yet dynamically diverse movements into more readable categories gives the later planning and completion stages a more reliable foundation. Clustering granularity has a clear sweet spot: both 75 and 125 clusters are worse than 100, since too few clusters over-merge distinct patterns and weaken geometric fidelity, while too many over-fragment similar motions and hurt coherence.
Planning accuracy sets the ceiling for final quality. Table 3 shows that handing the raw plan to the second stage leaves frame-level mispredictions fragmenting continuous movements and polluting completion; after majority-vote smoothing and minimum-duration merging, FID_k drops from 25.26 to 24.02 and R rises from 26.6 to 27.5. Using ground-truth atomic labels pushes FID_k down to 21.59 and R up to 29.8, which also evidences compatibility with human editing — manually adjusting the plan before completion can plausibly yield better structure and quality. ⚠️ Note that the Ours row of Table 1 is numerically identical to the "w/o Post-process" row of Table 3, so the two tables conflict; refer to the original paper.
Re-creation buys dance quality, not fidelity. The fixed setting (direct concatenation with temporal rescaling only) actually achieves lower FID_g (8.44 vs 9.03), but trails on Div_k (6.99 vs 8.01), BAS (0.2356 vs 0.2470), and MultiModality (1.43 vs 2.25). Copying retrieved results costs expressiveness and synchronization, whereas allowing diffusion to re-create behaves more like a real dancer naturally introducing variation when repeating a movement — the model does not simply copy motions from the dataset.
Retrieval strategy is a three-way trade-off among diversity, structure, and rhythm. Random choice has the highest diversity (MultiModality 2.62) but the worst FID_k (27.94) and the lowest R (24.5) because it demands stronger temporal scaling; a single fixed candidate has the lowest diversity (Div_k 6.42, MultiModality 2.18) but the highest R (28.9); duration-nearest retrieval sits at the balance point and its higher MultiModality confirms that duration-aware retrieval produces diverse dances while preserving temporal compatibility.
The structure is visible qualitatively. Against LODGE, the method repeatedly exhibits typical atomic movements with variation in duration and range, while LODGE produces an over-smooth dance with no apparent dance primitive.

Highlights & Insights¶

The movement vocabulary is built as a genuine three-stage pipeline, not a throwaway K-Means. Segmentation guarantees each unit is a complete process rather than a fragment, clustering preserves intra-group variation, and LLM re-clustering adds semantic readability — each step matches one of the three criteria for an atomic movement, so the motivation and the mechanism line up instead of piling up modules.
Writing the score with discrete diffusion is the key lever of the framework. D3PM naturally outputs a discrete label sequence, which turns "which category does this frame belong to, or is it a transition" into something samplable, and makes the intermediate result human-readable and hand-editable — something purely continuous diffusion (EDGE's signal-level inpainting) cannot do.
Splitting the music-to-motion correspondence into a symbolic and a signal level is the mechanistic explanation of the gains. Full-music conditioning lets the planner see phrase structure, while music-conditioned denoising plus the transition loss smooths the signal-level seams; this explains the simultaneous improvements in BAS and R over methods that learn beat correspondence in a short window.
Masked noise is a reusable knob-style design. The mask controls how much of a retrieved instance is preserved versus re-created, handing the fidelity-versus-diversity trade-off explicitly to inference time; the same idea transfers to any retrieval-plus-generation pipeline (e.g. text-driven motion editing, retrieval-augmented video continuation) by defining the mask on the tokens whose semantics must be preserved.
Exposing the plan to the user is a practical product angle. Because the plan is a readable symbol sequence decoupled from completion, changing a movement, adjusting durations, or reassembling sections needs no retraining.

Limitations & Future Work¶

The paper has no separate limitations section. Judging by the experimental setup, the vocabulary is fixed at 100 prototypes (7.3 sub-prototypes each on average) and its coverage of broader genres and more complex choreography is not discussed; with only 31.8 samples per sub-prototype, long-tail categories have even less data, which makes generation quality for rare movements more likely to fluctuate.
Discovery and generation are validated only on AIST++ (1,408 sequences, 5.2 hours, 10 genres), with no cross-dataset or cross-genre generalization experiments; vocabulary construction also depends on Gemini-2.5-Pro and PoseScript annotations, whose cost and labeling noise are not quantified.
The second stage is essentially retrieval plus conditional inpainting: the diversity ceiling is bounded by the instances in the database, and once the plan is wrong, completion can only patch locally rather than reorganize globally. Planning and completion are also trained separately, so the second stage's error cannot be back-propagated to the planner.
Reproducibility has numerical risks: Tables 1 and 3 conflict (the Ours row equals the "w/o Post-process" row), and R-precision is a self-proposed metric with a terse definition, so cross-paper comparisons should be treated with caution.
Quality still trails ground truth noticeably (FID_k 25.26 vs 17.1), suggesting that structure awareness addresses coherence and controllability rather than the fundamental fidelity gap.
Concrete improvement directions: make transitions symbolic as well (they are currently unlabeled free frames, a gap in the structural representation); jointly fine-tune the two stages so the completion reconstruction error can flow back to the planner; and expand the vocabulary on larger multi-genre data while studying how vocabulary size relates to generation quality.

vs Bailando / TM2D: They learn discrete motion codebooks with VQ-VAE / GPT for music-conditioned controllable generation, but each code corresponds to a very short motion segment, so decoded outputs are essentially static poses lacking process and segment-level structure. This paper's discrete unit is a complete motion process, raising granularity from frames to segments so the planning layer can express choreographic structure rather than a string of local poses.
vs EDGE: EDGE applies diffusion to dance generation with local editing and continuation, but its short training window limits the receptive field so it cannot capture global choreographic structure over longer timescales, and its inpainting edits at the signal level without symbolic interpretability. This paper plans symbolically over the whole music and then conditions generation on that plan, so editing happens at the level of movement type, duration, and position.
vs LODGE / EDMG: Both target long dance sequences (coarse-to-fine diffusion and efficient generation respectively) but produce keyframes without a clear process or semantic interpretability, which hinders composing a structural dance sequence conditioned on the music. This paper supplies the structural layer they lack; the two lines are complementary in spirit.
vs text-to-motion (TEMOS / T2M-GPT / MDM): Text conditioning provides concrete instructions and demands precise motion, whereas dance is an artistic and compositional form requiring both kinematic realism and aesthetic creation, where similar movements must be creatively diversified. Directly applying text-conditioned motion frameworks fails to capture these artistic and hierarchical characteristics, which is precisely why a dedicated atomic movement vocabulary is needed.
Transferable insight: modeling "recurring structural units plus variation" explicitly as a discrete vocabulary, then letting a planning layer predict vocabulary sequences under long-range conditioning and a generation layer perform conditional repair, transfers to long video generation, other music-driven performance tasks (music-to-gesture, music-to-gymnastics), and any sequence-generation problem that suffers from weak structural consistency yet needs human editability.

Rating¶

Novelty: ⭐⭐⭐⭐ Casting choreography's "atomic movements with repetition and variation" into an automatically constructed discrete vocabulary and planning the symbolic score with discrete diffusion is a novel and self-consistent angle; each individual component (K-Means plus LLM labeling, D3PM, DDPM completion) is an existing technique.
Experimental Thoroughness: ⭐⭐⭐ The main comparison plus four ablations cover vocabulary granularity, LLM re-clustering, post-processing, re-creation, and retrieval strategy with good analysis; however, validation is limited to a single dataset, Tables 1 and 3 conflict, and annotation cost is not quantified.
Writing Quality: ⭐⭐⭐ The correspondence between motivation and method is clear and the two-stage schematic is intuitive; but the formula regions are corrupted in several places after OCR, some tables are self-contradictory, and the R-precision definition is too terse.
Value: ⭐⭐⭐⭐ It offers an editable, interpretable paradigm for dance generation, and the symbolic planning layer is practically useful for long-sequence dance and human-in-the-loop choreography; the masked-noise fidelity-versus-diversity knob transfers to other retrieval-plus-generation pipelines.