Step-by-Step Video-to-Audio Synthesis via Negative Audio Guidance¶

Conference: ECCV 2026
arXiv: 2506.20995
Code: Project Page https://ahykw.github.io/sbsv2a/
Area: Audio and Speech / Video Generation / Diffusion Models
Keywords: Video-to-audio generation, Negative guidance, Flow Matching, Controllable generation, Foley

TL;DR¶

Addressing the limitation where existing video-to-audio (V2A) models can only generate a complete audio track all at once instead of adding sounds layer-by-layer like Foley artists, this paper proposes Negative Audio Guidance (NAG). By training a branch conditioned on "already-generated audio tracks" and utilizing it inversely during sampling, the current generation is pushed away from existing sounds. This enables step-by-step generation of complementary audio tracks to mix into high-quality synthesized sound, using only standard single-reference audio-visual datasets.

Background & Motivation¶

Pairing footage with matching sound effects in movies and games is called Foley recording. The traditional workflow requires Foley artists to first lay down a background noise track, and then incrementally add missing acoustic elements such as footsteps or rustling clothes layer-by-layer. A few seconds of footage often contains numerous audible events, making this process highly time-consuming and labor-intensive. Recent video-to-audio (V2A) models (e.g., Diff-Foley, FoleyCrafter, MMAudio) have succeeded in generating high-quality screen-aligned audio in both semantics and timing, making automated dubbing possible. However, they almost exclusively adopt a non-interactive "single-forward, full-generation" design where the model outputs a single complete audio track without providing any mechanism for incremental addition. This poses a practical issue: if an output misses a specific sound event, creators must regenerate the entire track rather than just patching that single part, which is highly inefficient in human-in-the-loop workflows.

A seemingly natural approach is to train a conditional generative model to produce multiple complementary audio tracks for the same video. However, this direction is bottlenecked by data: it requires "multi-reference" audio-visual pairs (a single video mapped to multiple separate tracks), which are nearly impossible to acquire at scale. Even in a single scene, various sounds are mixed together, and separating them into individual tracks is itself an unsolved challenge of vision-guided audio source separation. Another route is to use text-conditioned V2A directly to generate different sounds with varying prompts. However, existing text-conditioned V2A models struggle to suppress prominent sounds that have already been generated in the frame. The paper gives an example: in a video of a moose walking through water, regardless of the text prompt, every audio track generated by MMAudio contains the sound of the moose splashing, because this visual event is too dominant.

Key Insight: Since the actual difficulty lies in "what NOT to generate" rather than "what TO generate," the task of "excluding existing sounds" can be delegated to guidance instead of retraining a multi-reference model.
Core Idea: Treat the "already-generated audio track" as a negative audio condition and train an audio-conditioned branch to predict the flow direction "toward this audio." During sampling, this term is subtracted (used negatively) to push the current generation away from existing sounds. This branch can be trained on standard single-reference datasets using two non-overlapping audio segments from the same video as mutual conditions, entirely bypassing the requirement for multi-reference data.

Method¶

Overall Architecture¶

The method is built on top of MMAudio, a pretrained flow-matching V2A model, and aims to address the problem of "step-by-step complementary audio track generation." Given a video \(V\) and several text captions describing different sound events, the model sorts the events by semantic prominence and generates one audio track per step: at step \(k\), already-generated audio tracks from steps \(1\) to \(k-1\) are mixed and normalized into a "conditional audio" \(x^{(1)}\). The current step then generates the target audio track \(x^{(2)}\) under the positive guidance of text \(C_2\), while being negatively repelled by the conditional audio to ensure that \(x^{(2)}\) only covers complementary sounds not present in \(x^{(1)}\). Finally, all tracks are normalized by loudness and summed to obtain the synthesized audio.

The entire system consists of three parts, corresponding to the three key designs below: first, step-by-step generation is formalized into a target distribution of "concept negation" to derive multi-conditional guided flows (the theoretical backbone that decides how multiple flow terms are combined during sampling); next, the audio-conditioned branch that carries the "negation" is implemented as a ControlNet bypass for MMAudio, whose output is subtracted during sampling; lastly, the training trick that enables this branch to be trained on single-reference data uses two non-overlapping audio segments from the same video as "target/condition" training pairs.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Video V + Text C2<br/>+ Mixed Prior Tracks x(1)"] --> B["Guided Flow via Concept Negation<br/>Pushed toward V/C2, Repelled from x(1)"]
    B --> C["Audio-conditioned Branch (ControlNet)<br/>Predicts flow direction toward x(1)"]
    C -->|Subtracted during sampling| D["Current Complementary Track x(2)"]
    D -->|Iterated for k steps| E["Loudness-normalized Summation<br/>Synthesized Audio"]

Key Designs¶

1. Rewriting the Target Distribution with "Concept Negation" to Derive Multi-Conditional Guided Flows: Translating "Do Not Repeat Existing Sounds" into a Samplable Equation

Step-by-step generation aims to sample from \(p(x^{(2)}\mid V, C_2, x^{(1)})\). Intuition suggests training on quadruplets like \((x^{(2)}, V, C_2, x^{(1)})\), which are unavailable. The key observation of this paper is that conditioning on \(x^{(1)}\) while requiring the generated \(x^{(2)}\) to exclude concepts in \(x^{(1)}\) is essentially a form of "concept negation." Following the energy-based perspective of Du et al., negation conditions can be expressed as the reciprocal of probability—meaning that when generating \(x\), the likelihood of positive concept \(c_p\) is multiplied by the reciprocal of the likelihood of negative concept \(c_n\):

\[p(x, c_p, \neg c_n) \propto p(x)\, p(c_p\mid x)\, p(c_n\mid x)^{-1}.\]

Substituting this relationship into the target distribution and expanding each term via Bayes' theorem decomposes \(p(x^{(2)}\mid V,C_2,\neg\mathcal{E}(x^{(1)}))\) into four factors: an unconditional term, a video term, a video+text term, and a ratio term of "video vs video+conditioned audio." Correspondingly in flow-matching, this yields a guidance comprising "one unconditional flow + three guided flows." Tuning four coefficients is tedious, so the authors adopt the experience of VinTAGe—simplifying by eliminating an intermediate term \(u_\theta(x_t,t,V,\varnothing)\) without sacrificing performance—setting the sum of the first and third guidance coefficients equal to the second. This converges into a clean formula containing only the unconditional, text-guided, and negative audio terms:

\[\tilde{u}_{\theta,\psi}(x_t) = u_\theta(x_t,t,\varnothing,\varnothing) + \alpha\big(u_\theta(x_t,t,V,C_2) - u_\theta(x_t,t,\varnothing,\varnothing)\big) + \beta\big(u_\theta(x_t,t,V,C_2) - u_{\theta,\psi}(x_t,t,V,\varnothing,x^{(1)})\big).\]

The second term is standard text+video guidance, reinforcing fidelity to the prompt and visuals. The third term is the newly introduced negative audio guidance—pushing the generation direction away from "becoming more like conditional audio \(x^{(1)}\)," which is key to preventing re-generation of already-generated events, thereby enabling step-by-step generation without overlapping data. Coefficients default to \(\alpha=4.5, \beta=1.5\).

2. Adding an Audio Conditional Branch to MMAudio via ControlNet: Learning the Flow Directions "Toward Audio" and Reversing It

All flows in Equation (7) except the last one can be computed using the off-the-shelf text-conditioned V2A model. The only missing part is \(u_{\theta,\psi}(x_t,t,V,\varnothing,x^{(1)})\)—a flow estimator that takes "conditional audio" as an extra input. The authors implement this as a ControlNet bypass: since MMAudio uses an extended multimodal diffusion transformer (MM-DiT), copying Stable Diffusion 3.5's approach, they stack several unimodal transformer blocks to extract features from the conditional audio. Features from each block are added to intermediate MMAudio backbone features via zero-convolution. During training, all pretrained parameters of MMAudio are frozen, and only a new set of parameters \(\psi\) (around 107M) for the bypass are updated. Thus, training is highly efficient, taking about 10 hours on 8 H100 GPUs for 200K steps.

The ingenuity of this design lies in the opposite behaviors of "training target" and "sampling application": during training, this branch behaves as positive guidance, learning to generate audio aligned with the acoustic features of the conditional audio under the constraint of \(V\). During sampling (where it is preceded by a minus sign), it acts as a repulsive prior, pulling the step trajectory away from the direction of fitting the conditional audio. In other words, it does not need to learn how to perfectly "copy" the conditional audio; it only needs to point out "which direction makes the output more like the conditional audio" so that this direction can be subtracted—which naturally fits the negative application of NAG.

3. Training with Non-overlapping Audio Segment Pairs from the Same Video: Bypassing Multi-Reference Data and Forcing the Branch to Learn "Segment-Level Shared Acoustic Context"

This is the key trick that allows the entire method to be trained on standard single-reference datasets. Following MMAudio's training configuration, VGGSound (text-video-audio) is jointly used with Clotho, AudioCaps, and WavCaps (text-audio). For each audio clip, a 4-second segment is sampled as target \(x_{\text{tgt}}\), and another non-overlapping 4-second segment is sampled as condition \(x_{\text{cond}}\) (for VGGSound, the corresponding video segment is also taken as condition \(V\), while other datasets use a null token placeholder). Non-overlapping is crucial; if they overlapped, the branch would learn a trivial solution of directly copying the conditional waveform. By choosing adjacent but non-overlapping segments from the same clip, they are highly likely to share high-level acoustic contexts like environment, timbre, and recording conditions, while not containing identical sound content.

The branch trained this way learns "which direction the overall acoustic ambiance of the video goes" instead of copy-pasting sample-by-sample, which is precisely what NAG needs during inference: it wants to reject "components already present in prior tracks," which are characterized by such segment-level shared contexts. As the authors put it, because the branch is utilized negatively, it only needs to identify "which directions increase consistency with \(x_{\text{cond}}\)" and subtract them during sampling. This rejects existing sounds while avoiding the degenerate case of waveform copy-paste.

A Complete Example¶

Take a video of a "moose walking through water" with three captions sorted in descending order of semantic prominence. Step 1 generates the core audio track "moose splashing"—with no prior tracks generated, standard text+video CFG (\(\beta=0\)) is used to output \(x^{(1)}\). Step 2 aims to generate "distant birds chirping": \(x^{(1)}\) is normalized as conditional audio and fed into the audio branch. The third term of the formula pushes the generation direction away from the "splashing sound," so almost no splashing is heard in this track, leaving only bird chirping. Step 3 generates "muffled hoofbeats": prior tracks are mixed and normalized into a new conditional audio, which is again negatively repelled to yield a clean third track. Finally, the three tracks are normalized and summed. In comparative experiments, standard MMAudio's three tracks all contain periodic splashing sounds (visible as vertically spaced bars on the spectrogram), whereas NAG's second and third tracks successfully suppress the splashing noise while maintaining alignment with their respective text captions.

Loss & Training¶

The training objective is the standard flow-matching loss—minimizing the mean squared error between the estimated flow \(u_{\theta,\psi}(x_t^{\text{tgt}}, t, V, \varnothing, x_{\text{cond}})\) computed from the sampled \((x_{\text{tgt}}, x_{\text{cond}}, V)\) and the ground truth velocity field, optimizing only the ControlNet parameters \(\psi\). The optimizer is AdamW (lr \(10^{-4}\) , \(\beta_1=0.9\), \(\beta_2=0.95\), weight decay \(10^{-6}\)) with a batch size of 512, training for 200K steps (fewer than MMAudio's default 300K due to faster convergence). A linear warmup of 1K steps is applied, with decay steps of 10x at 80% and 90% of training. Bf16 mixed precision is used, and a post-hoc EMA (\(\sigma_{\text{rel}}=0.05\)) is applied after training. One ControlNet block is added to every two backbone blocks (\(N_1+N_2=12\), \(M=6\)).

Key Experimental Results¶

The authors constructed a new evaluation dataset, Multi-Caps VGGSound: using Qwen2.5-VL to generate 5 captions describing different acoustic events for each video in the VGGSound test set (15,221 videos × 5). For each video, the first 8 seconds are used to generate 5 sequential audio tracks based on different captions, which are then loudness-normalized (target -20 LUFS) and summed. Baselines include three open-source text+video conditional V2A models (Seeing-and-Hearing, FoleyCrafter, MMAudio), comparing NAG with MMAudio using CFG and negative prompt strategies for multi-track generation.

Main Results¶

Objective evaluation of synthesized audio (Table 1), where NAG achieves the best performance among step-by-step generation methods in all metrics except IS:

Generation Method	Method	FD_PANNs↓	FD_VGG↓	KL_PANNs↓	IS↑	IB-score↑	DeSync↓
Independent Generation	MMAudio-S-16k	7.76	1.35	2.02	10.42	28.13	0.42
Step-by-Step with Negative Prompt	MMAudio-S-16k	9.21	1.77	2.15	9.08	25.89	0.45
Step-by-Step with NAG (Ours)	Ours	6.47	0.98	2.01	10.58	28.65	0.42

Single-track evaluation (Table 2) proves the point—the core metric is the "separability" CLAP A-A between tracks (lower is more separated):

Method	Separability CLAP A-A↓	Quality IS↑	Text Alignment CLAP T-A↑	Video Alignment IB↑
MMAudio-S-16k	79.75	12.47	28.36	27.76
MMAudio + Negative Prompt	75.57	11.19	27.14	24.53
MMAudio + NAG (Ours)	71.38	12.01	28.91	26.67

Clear contrast: original MMAudio's tracks are highly repetitive (poor separability), and while negative prompts improve separability, they drastically degrade other metrics (severe drops in text and video alignment). In contrast, NAG achieves the best separability while maintaining high quality and video alignment, and actually improving text alignment.

Subjective evaluations support this: for synthesized sound, NAG's win rate over MMAudio is 71.4% for audio quality, 76.0% for semantic alignment, and 61.1% for temporal alignment (Table 3); for single-track ratings, separability is 3.35 vs 2.24, quality is 3.30 vs 2.89, and text fidelity is 3.12 vs 2.42 (Table 4, 5-point scale).

Ablation Study¶

Guidance coefficient sensitivity (Appendix Table A1, excerpt) demonstrates the impact of NAG strength \(\beta\):

Configuration	CLAP A-A↓	CLAP T-A↑	FD_PANNs↓	Note
\(\alpha{=}4.5,\beta{=}0\)	79.75	28.36	7.76	Equivalent to independent MMAudio generation
\(\alpha{=}4.5,\beta{=}1.0\)	76.21	28.69	7.30	Moderate NAG, optimal FD
\(\alpha{=}4.5,\beta{=}1.5\)	74.74	28.79	7.32	Default setting
\(\alpha{=}4.5,\beta{=}2.0\)	73.44	28.88	7.52	Better separability, slight FD degradation

Generation order ablation (Table A4): descending order by text-video ImageBind similarity (generating the most visually prominent core event first) yields the best results across all metrics, with an FD_PANNs of 6.47, markedly superior to random (7.32) and ascending (7.36).

Key Findings¶

Negative utilization is crucial: training the same audio branch positively but applying it negatively during sampling excludes existing sounds without copying waveforms. Directly using textual negative prompts severely degrades text/video alignment.
Higher \(\beta\) values lead to better separability and higher text alignment (fewer contamination), but FD/IB are optimized near \(\beta=1.0\); establishing a trade-off yields 1.5. A small \(\alpha\) hampers the baseline MMAudio performance.
Generating prominent events first is crucial—descending generation allows core sounds to "occupy" the space first, allowing subsequent steps to successfully repel them.
More captions highlight NAG's advantages (experiments on VGGSounder with 2-5 captions); generalization is stable across AudioCaps and Movie Gen Audio Bench. Swapping the base model to ControlFoley maintains NAG's effectiveness (separability 60.67 \(\rightarrow\) 53.81), showing the method is not tied to MMAudio.

Highlights & Insights¶

Reframing "incremental sound design" as "concept negation": Instead of teaching the model "what to generate" (which requires multi-reference data), it is told "what NOT to generate". Transforming this idea into a clean guidance equation via energy-based reciprocal likelihoods is the most elegant translation in the paper.
Highly reusable "train positively, use negatively" branch design: Any generation task involving the rejection of certain existing contents (image inpainting/object addition, music accompaniment addition, video continuation) can benefit from this—train a bypass conditioned on the content, and subtract its guided direction during sampling.
Cheap and clever training pair formulation with non-overlapping segments: Using adjacent but non-overlapping segments from the same clip forces the branch to learn segment-level acoustic context rather than sample-by-sample replication. This avoids degenerate solutions and utilizes only single-reference data, neatly sidestepping the formidable challenge of vision-guided audio source separation.
Inherent value in workflow interactivity: Foley-style step-by-step sound addition allows users to focus on one acoustic event at a time and modify any track independently, which is far more practical than "jointly generating and then applying source separation" (which Appendix shows achieves worse quality and more cumbersome interaction).

Limitations & Future Work¶

The authors acknowledge slightly degraded audio quality on individual tracks: Although NAG suppresses contamination, outputs occasionally suffer from poor text alignment or quiet/muffled segments. The root cause lies in the base MMAudio's struggle with rare or faint sounds (e.g., "gentle carpet rustling", "snow falling")—NAG only repels existing sounds, so overall quality is heavily bounded by the base V2A model's capabilities.
Oversimplified mixing strategy: Directly summing tracks with equal weights and applying loudness normalization does not account for the natural relative volumes of different tracks, which might be suboptimal. Future work could introduce generative models to learn mixing.
Evaluation captions were generated by a VLM (Qwen2.5-VL) purely from visual cues: This may drift from actual audio (e.g., voiceovers). While this is not an issue for "visually plausible sound addition," it does mean evaluations do not measure the reproduction of original recordings.
Self-observation: The method is inherently "serial," meaning step \(k\) depends on the mixed results from steps \(1\) to \(k-1\). Sensitivity to generation order (descending is optimal) indicates that early errors propagate through the chain, lacking fallback/correction mechanisms. Also, running ControlNet at every step (2.07 seconds per step on H100) causes latency to scale linearly with the number of audio tracks.

vs. Training-based Additive Operations (InstructME, AUDIT, etc.): These explicitly train models to "add new audio given an input audio," requiring triplet data of "(input audio, text, audio-to-add)," which are unavailable in this paper's setting. This work bypasses this data hurdle using negative guidance + single-reference data.
vs. Training-free Additive Operations (Multi-track joint generation / structured noise inversion): These training-free methods require pre-trained models to possess properties like multi-track joint generation, data-space diffusion, or specific setups, which are hard to transfer to V2A. This paper chooses a training-based route but eliminates the need for special data through clever pair formulation.
vs. MultiFoley / Action2Sound / ReWaS, etc. (Audio-conditioned V2A): These use conditional audio to "specify" what to generate (reference audio, background/foreground separation, energy alignment), whereas this work does the opposite, using conditional audio to "specify what NOT to generate," representing the first attempt at utilizing audio conditions for concept negation.
vs. Image Object Addition (Add-it, EraseDraw, etc.): Visual addition relies on segmentation models to create training data or guide generation under the assumption that a specific pixel region is completely replaced. However, audio addition is a "mix" rather than a "replacement," making visual methods inapplicable—this work is specifically designed around the "additive mixture" nature of audio.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Formulating step-by-step sound addition as concept negation, implemented via a branch trained positively but used negatively, all while bypassing multi-reference data. Elegant and highly original.
Experimental Thoroughness: ⭐⭐⭐⭐ Covers objective and subjective evaluations, coefficient/order ablations, and generalization across datasets/base models. Demoted one star because the analysis of synthesized audio quality degradation is primarily qualitative.
Writing Quality: ⭐⭐⭐⭐ Clear logical progression (motivation \(\rightarrow\) derivation \(\rightarrow\) implementation \(\rightarrow\) experiment). Figure 1 and 4 provide highly intuitive comparisons. Derivation of the guidance formula is slightly dense.
Value: ⭐⭐⭐⭐ Directly addresses the real-world workflow of interactive Foley design, and the negative guidance paradigm can easily transfer to other generative tasks requiring rejection of existing content.