Enhancing Dance-to-Music Generation via Negative Conditioning Latent Diffusion Model

Conference: CVPR 2025
arXiv: 2503.22138
Code: https://github.com/Changchangsun/PN-Diffusion
Area: Diffusion Models
Keywords: Dance-to-Music Generation, Negative Conditioning Diffusion, Bi-directional Diffusion Process, Beat Alignment, Latent Diffusion Models

TL;DR

This work proposes PN-Diffusion, which extracts positive and negative beat conditions from forward-played and backward-played dance videos respectively. It designs a dual diffusion and reverse process to jointly train a U-Net, enhancing the beat consistency and music quality of generated music with dance movements. On the AIST++ and TikTok datasets, it improves BCS by 1.80/3.85 and BHS by 4.22/5.90.

Background & Motivation

Background: Conditional diffusion models have achieved remarkable success in cross-modal generation (T2I, T2A, T2V). The Dance-to-Music (D2M) task requires generating beat-synchronized accompaniment music from dance videos and represents an important application for video-sharing platforms (such as TikTok, YouTube). Existing D2M diffusion methods like CDCD and LORIS extract visual beats and motion information from dance videos as conditional inputs for the U-Net.

Limitations of Prior Work: Existing approaches only utilize forward-played dance videos to extract positive beat cues and motion information as conditions, ignoring the "negative" beat information carried by backward-played videos. Analogous to bi-directional guidance using positive and negative samples in machine learning—where positive samples tell the model "what to do" and negative samples tell it "what to avoid"—training with only positive conditions is incomplete.

Key Challenge: How to effectively define and utilize "negative conditions" in conditional diffusion models? Should negative conditions directly affect the noise prediction of the forward process, or should an independent negative noise process be introduced? How can this be seamlessly integrated into a sequential multi-modal U-Net architecture?

Goal: (1) How to define negative conditions in the D2M task? (2) How to integrate positive and negative conditions in the LDM framework for joint training? (3) How to enable negative conditions to truly improve the beat alignment and quality of the generated music?

Key Insight: Artfully utilizing the temporal characteristics of dance videos—where playing forward and backward provides opposite temporal beat information. Backward playback preserves the same poses and transitions but in reverse, constituting the most faithful pairing of negative samples.

Core Idea: Utilizing backward-played dance videos as negative conditions, the proposed method designs a dual diffusion process (adding positive/negative noise) and a bi-directional reverse process to jointly train the U-Net. This allows positive conditions to guide the recovery of positive noise and negative conditions to guide the prediction of negative noise, bidirectionally reinforcing beat learning.

Method

Overall Architecture

The input is a dance video, and the output is a Mel-spectrogram (convertible back to audio). The workflow is as follows: (1) The audio is converted into a 256×256 Mel-spectrogram, compressed into a 32×32 latent space using a VAE; (2) Visual embeddings (I3D) and motion information (BlazePose + ST-GCN) are extracted from forward/backward played videos respectively and concatenated into positive conditions \(c^+\) and negative conditions \(c^-\); (3) Dual diffusion and bi-directional reverse processes are executed in the latent space to train the U-Net; (4) During inference, generation is sampled using only the positive conditions.

Key Designs

1. Positive & Negative Conditioning:

   • Function: Extract bi-directional beat and motion conditions from dance videos.
   • Mechanism: The positive condition \(c^+\) is formed by concatenating the I3D visual embedding \(p \in \mathbb{R}^{2048}\) of the forward-played video and the ST-GCN motion embedding \(q \in \mathbb{R}^{1024}\). The negative condition \(c^-\) is formed via the same process but applied to the backward-played video. I3D captures the visual beats of video frames, while BlazePose extracts 33 human keypoints, and ST-GCN encodes motion patterns within the spatial-temporal graph sequence. Backward playback reverses the temporal direction of keypoints, thereby inverting the motion tempo/beats.
   • Design Motivation: Directly defining "completely opposite" negative samples is difficult, but backward playback naturally provides opposition in the temporal dimension—retaining the same poses and transitions but in reverse temporal direction. This is more targeted than randomly choosing unrelated videos as negative samples, forming a more faithful positive-negative pair.

2. Dual Diffusion Process:

   • Function: Construct two parallel forward diffusion processes that add noise in opposite directions.
   • Mechanism: Starting from the same initial point \(z_0\), the forward (positive) diffusion adds noise \(\epsilon\): \(z_t^+ = \sqrt{\bar{\alpha}_t} z_0 + \sqrt{1-\bar{\alpha}_t} \epsilon\); the negative diffusion adds the opposite noise \(-\epsilon\): \(z_t^- = \sqrt{\bar{\alpha}_t} z_0 + \sqrt{1-\bar{\alpha}_t} (-\epsilon)\). Both processes share the same noise schedule and starting point, differing only in the direction of the added noise.
   • Design Motivation: Traditional diffusion models have only one forward process, where the U-Net only needs to learn to predict noise in one direction. Introducing symmetric negative diffusion forces the U-Net not only to recover positive noise but also to determine the direction of negative noise under the negative condition. This bi-directional contrastive training reinforces the sensitivity of the U-Net to temporal beat direction.

3. Bi-directional Denoising Objective:

   • Function: Unify the noise prediction objectives under positive and negative conditions into a training loss.
   • Mechanism: The U-Net uses the positive condition to predict the positive noise \(\epsilon_\theta^+(z_t^+, t, c^+)\) and the negative condition to predict the negative noise \(\epsilon_\theta^-(z_t^-, t, c^-)\). The total loss is \(L_\epsilon = \alpha \|\epsilon - \epsilon_\theta^+(z_t^+, t, c^+)\|_2^2 + (1-\alpha) \|-\epsilon - \epsilon_\theta^-(z_t^-, t, c^-)\|_2^2\), where \(\alpha\) controls the weight balance between positive and negative tasks. Inference uses only the positive condition for sampling.
   • Design Motivation: Positive conditions guide "what kind of beats to generate," while negative conditions guide "what kind of beats to avoid generating." The bi-directional objective enables the same U-Net to learn both capabilities simultaneously. The prediction performance of positive noise is enhanced by contrasting with the negative noise, similar to the effect of contrastive learning.

Loss & Training

• First Stage: Train the VAE encoder/decoder to achieve perceptual compression of Mel-spectrograms (perceptual loss + patch-based adversarial loss).
• Second Stage: Train the conditional DDPM in the latent space using the bi-directional denoising objective \(L_\epsilon\).
• \(\alpha\) is determined through grid search to find the optimal value.
• Sampling rate of 22,050 Hz, 5-second audio clips, diffusion steps 1000, batch size 32.
• Inference uses only the positive condition (forward-played dance video) with 1000 sampling steps.

Key Experimental Results

Main Results

Dataset	Metric	Ours	Prev. SOTA (LORIS)	Gain
AIST++	BCS ↑	97.72	95.92	+1.80
AIST++	BHS ↑	93.73	89.51	+4.22
AIST++	F1 ↑	95.69	92.67 (CMT)	+2.01*
AIST++	FAD_v ↓	5.62	6.27	-0.65
AIST++	FAD_p ↓	13.47	16.49	-3.02
TikTok	BCS ↑	92.68	88.83	+3.85
TikTok	BHS ↑	88.56	82.66	+5.90
TikTok	F1 ↑	90.56	85.71	+4.85

Ablation Study

Configuration	BCS	BHS	F1	Description
Only positive (baseline)	~95.9	~89.5	~92.7	Only positive condition
+ Negative conditioning	97.72	93.73	95.69	Integrating negative condition with bi-directional training
\(\alpha=1.0\) (Positive only)	Lower	Lower	Lower	Degenerates to standard diffusion
\(\alpha=0.5\)	Near optimal	Near optimal	Near optimal	Balanced positive & negative
\(\alpha=0.0\) (Negative only)	Lowest	Lowest	Lowest	No positive signal

Key Findings

• On the two core beat-alignment metrics, BCS and BHS, PN-Diffusion significantly outperforms all baselines, indicating that negative conditions indeed enhance beat learning.
• The FAD metrics (measuring generated music quality) also lead across the board, demonstrating that negative condition training improves both alignment and music quality.
• The performance gain on the TikTok dataset is even larger than on AIST++ (BHS +5.90 vs +4.22), likely because TikTok videos feature more diverse movements, making the contrastive effect of negative conditions more pronounced.
• The parameter \(\alpha\) requires careful tuning; overemphasizing either positive or negative conditions degrades performance, with the optimum lying around 0.5.
• Subjective evaluations (MOS and Turing Test) confirm the trends observed in objective metrics.

Highlights & Insights

• Clever Definition of Negative Conditions via Backward Playback: Identifying or constructing "negative samples" in D2M tasks has typically been a difficult problem. The authors elegantly exploit backward video playback to invert the rhythm, presenting a zero-cost, zero-annotation, and semantically logical scheme for negative sample construction. This concept can be transferred to other video-conditioned generation tasks.
• Symmetric Design of the Dual Diffusion Process: The positive and negative noise directions are opposite but share the same starting point and schedule, achieving contrastive training without increasing model parameters (by sharing the same U-Net). This dual-process mechanism can be generalized to other conditional generation tasks that require directional guidance.
• Zero Overhead during Inference: Negative conditions only participate in training. No extra computation is introduced during inference, aligning the deployment exactly with standard LDMs.

Limitations & Future Work

• Experiments were only conducted on 5-second clips; quality and coherence when generating longer music (e.g., full-length songs) remain unexplored.
• The assumption that "backward playback = negative sample" might not hold for certain highly symmetric dances (e.g., waltzes that cycle back and forth).
• More complex negative sample construction strategies (e.g., speed variation, shuffled video frames) have not been explored.
• Under-evaluation of music diversity—can the model generate appropriate music of different styles for the same dance?
• Focus is purely on beat alignment, without considering higher-level musical features like melodic structure and harmony.
• FAD metrics rely on pre-trained feature extractors, which may introduce evaluation biases.

Related Work & Insights

• vs CDCD: CDCD and LORIS use standard LDMs with only forward video conditions. PN-Diffusion introduces negative-condition bi-directional training on top of this, achieving significant improvements within a fully compatible framework.
• vs Classifier-Free Guidance: CFG guides generation via the difference between unconditional and conditional predictions. PN-Diffusion extends this concept from "guidance direction" to contrasting positive/negative conditions.
• vs D2M-GAN: GAN-based methods struggle to model long-term temporal dependencies. PN-Diffusion is based on diffusion models, which naturally handle sequence generation, leading to comprehensively superior performance.

Rating

• Novelty: ⭐⭐⭐⭐ The concept of negative conditions combined with dual diffusion is a first in D2M, and using backward playback for negative samples is highly creative.
• Experimental Thoroughness: ⭐⭐⭐⭐ Evaluated on two datasets across eight metrics, incorporating both subjective and objective assessments.
• Writing Quality: ⭐⭐⭐⭐ Motivation and methodology are clearly described with comprehensive mathematical formulation.
• Value: ⭐⭐⭐⭐ The general idea of negative-condition training is inspiring, although the application scope of D2M itself is relatively niche.

Related Papers

• [NeurIPS 2025] MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extraction
• [ICCV 2025] Latent Swap Joint Diffusion for 2D Long-Form Latent Generation
• [CVPR 2025] VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generation
• [AAAI 2026] Diff-V2M: A Hierarchical Conditional Diffusion Model with Explicit Rhythmic Modeling for Video-to-Music Generation
• [NeurIPS 2025] MEGADance: Mixture-of-Experts Architecture for Genre-Aware 3D Dance Generation