---

title: >- [Paper Note] Precise Video-to-Audio Generation with Cross-Modal Alignment in Latent Space description: >- [ECCV 2026][Audio & Speech][Flowley] Proposes Flowley, an end-to-end single-stage flow matching video-to-audio framework with Progressive Soft-masked Cross-Attention (PSCA) and SoundCap, achieving state-of-the-art audio quality and temporal alignment without pretrained alignment modules. tags: - ECCV 2026 - Audio & Speech - Video-to-Audio - Flow Matching - Cross-Modal Alignment date: 2026-09-19 content_hash: 21d522a005267247

Precise Video-to-Audio Generation with Cross-Modal Alignment in Latent Space

Conference: ECCV 2026
Paper: ECCV 2026 Poster
Code: https://flowley-v2a.github.io
Area: Audio & Speech
Keywords: Video-to-Audio / Flow Matching / Progressive Cross-Attention / Audio-Visual Synchronization / Sound-Aware Captioning

TL;DR

Addressing the excessive training overhead and reliance on external alignment networks in video-to-audio (V2A) synthesis, this paper proposes Flowley, a single-stage flow-matching framework incorporating Progressive Soft-masked Cross-Attention (PSCA) for parameter-free temporal synchronization, coupled with a sound-aware captioner (SoundCap) to deliver state-of-the-art generation fidelity.

Background & Motivation

Sound design is fundamental to cinematic storytelling, where Foley post-production demands exact synchronization between synthesized audio effects and on-screen visual movements. Skilled Foley artists historically achieve this through tactile physical manipulation with physical props on dedicated recording stages. Although recent progress in text-to-audio (T2A) generation allows designers to synthesize realistic sound effects from natural language prompts, practitioners must still manually adjust timing markers on digital audio workstations to match video cuts and physical impacts, limiting real-time production throughput.

To automate this cumbersome workflow, video-to-audio (V2A) generation has attracted significant attention, typically following one of two paradigms. The first paradigm translates visual features from silent video into concise textual captions and feeds them into frozen T2A backbones; however, compressing high-frequency continuous visual motion into static textual tokens inevitably discards fine-grained temporal cues necessary for synchronized Foley sound effects. The second paradigm adopts multi-stage pipelines that pretrain separate audio-visual alignment models via contrastive learning or train auxiliary onset/energy regression heads before training the generator. These multi-stage workflows introduce heavy computational overhead, potential compounding errors across stages, and remain bottlenecked by the lack of sound-oriented textual annotations in public benchmarks like VGGSound.

This paper tackles these challenges from a clean perspective: temporal synchronization does not require external pretrained alignment networks or complex multi-stage objectives, but can instead emerge directly within the cross-modal attention operator of latent generative models. Simultaneously, the lack of sound-focused annotations can be resolved by distilling a joint audio-visual foundation model into a visual captioner. Core idea: construct a single-stage end-to-end flow-matching architecture, Flowley, featuring a Progressive Soft-masked Cross-Attention (PSCA) mechanism that injects depth-aware temporal decay into latent attention without extra parameters, accompanied by SoundCap to curate sound-aware textual conditioning for high-fidelity audio synthesis.

Method

Overall Architecture

Flowley takes silent video frames and corresponding descriptive prompts as input, progressively transforms Gaussian noise into target mel-spectrogram latents using a Flow Matching Ordinary Differential Equation (ODE) solver, and reconstructs the final audio waveform via a BigVGAN neural vocoder. The framework couples a multimodal feature extraction backbone with \(N_1\) multi-stream blocks for joint cross-modal fusion, followed by \(N_2\) single-stream refinement blocks equipped with Progressive Soft-masked Cross-Attention (PSCA).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Silent Video Frames + Text Prompt + Gaussian Noise Latent"] --> B["Multi-to-Single Stream Backbone<br/>N1 Multi-Stream Blocks + N2 Single-Stream Audio Refinement Blocks"]
    B --> C["Progressive Soft-masked Cross-Attention<br/>Hard-window with Cosine Decay Kernel + Layer-dependent Factor beta"]
    C --> D["Tangent Noise Scheduling & Velocity Direction Supervision<br/>Tangent-based Time Sampler + Cosine Velocity Direction Loss Lvel"]
    D --> E["Sound-aware Captioner<br/>AV-LLM Ground-truth Distillation into Video-only VLM"]
    E --> F["Output: ODE Integration to Mel-Spectrogram & BigVGAN Waveform Synthesis"]

Key Designs

1. Multi-to-Single Stream Backbone: Balancing Early Multimodal Fusion and Audio Latent Specialization Prior generative architectures either concatenate all modalities naively—exploding attention complexity—or perform superficial late cross-attention that fails to capture deep cross-modal correlations. Flowley adopts a decoupled two-stage backbone design. Video frames sampled at 8 FPS are encoded via CLIP into frame-level features \(f_{\text{vis}} \in \mathbb{R}^{L_{\text{vis}} \times D_{\text{vis}}}\), while text prompts are embedded via FLAN-T5 into \(f_{\text{txt}} \in \mathbb{R}^{L_{\text{txt}} \times D_{\text{txt}}}\). The first \(N_1 = 5\) multi-stream blocks process text, visual, and noisy audio latents concurrently using joint self-attention equipped with QK-Norm and Rotary Positional Embeddings (RoPE) to establish stable cross-modal grounding. The audio latent stream is subsequently routed into \(N_2 = 10\) single-stream blocks for dedicated acoustic refinement, where layer-wise learnable gating parameters \(\alpha^{(\ell)}\) balance textual semantic conditioning against visual temporal conditioning, maintaining high expressive capacity while keeping model size compact at 169M parameters.

2. Progressive Soft-masked Cross-Attention: Inherent Audio-Visual Synchronization Without External Models Standard cross-attention computes unrestricted global affinities between audio query tokens and visual key tokens, allowing audio features to erroneously attend to distant visual frames and causing temporal desynchronization. Instead of adding external alignment networks or auxiliary onset detectors, Flowley introduces Progressive Soft-masked Cross-Attention (PSCA). PSCA applies a dynamic, depth-dependent mask directly to the cross-attention affinity matrix:

Given an audio query token \(i\), its aligned video frame index is defined by \(j_c(i) = \min(\lfloor \frac{r_v}{r_a} i \rfloor, L_{\text{vis}} - 1)\), with temporal distance \(d_{ij} = |j - j_c(i)|\). Across single-stream layer \(\ell\), attention weights are decayed via a layer-dependent progression parameter \(\beta_\ell = 1 - \frac{\ell}{N_2 - 1}\):

\[M_{ij}^{(\ell)} = \begin{cases} 1, & d_{ij} \le \omega \\ \beta_\ell \cdot \frac{1}{2} \left[ \cos\left(\frac{\pi (d_{ij} - \omega)}{\delta}\right) + 1 \right], & \omega < d_{ij} \le \omega + \delta \\ 0, & d_{ij} > \omega + \delta \end{cases}\]

Here, \(\omega\) defines the hard-attention window (set to 0), and \(\delta\) denotes the soft cosine fade window (set to 4). In shallow layers, a larger \(\beta_\ell\) permits the model to aggregate contextual motion cues from adjacent video frames. In deeper layers, \(\beta_\ell \to 0\), naturally contracting the receptive field into a strict local window. This mechanism guides the velocity field toward frame-accurate synchronization with zero added parameters or latency overhead.

3. Tangent-Based Scheduling & Velocity Direction Supervision: Preventing Endpoint Collapse and Stabilizing Vector Fields Standard Flow Matching models frequently rely on logit-normal timestep sampling, which places heavy probability mass on intermediate diffusion steps but collapses to zero density at boundaries \(t=0\) and \(t=1\). This degrades boundary velocity estimation near clean data and pure Gaussian priors. Flowley adopts a tangent-based sampling schedule:

\[t = 1 - \frac{1}{\tan\left(\frac{\pi}{2} u\right) + 1}, \quad u \sim \mathcal{U}[0, 1]\]

This formulation ensures smooth probability coverage across the entire trajectory \([0, 1]\). Furthermore, because MSE velocity regression \(\mathcal{L}_{\text{FM}}\) can yield vector magnitudes that approximate ground truth while drifting in angular orientation, Flowley supplements training with an explicit cosine velocity direction loss:

\[\mathcal{L}_{\text{vel}} = 1 - \text{sim}\big(v_\theta(x_t, t, c), (x_1 - x_0)\big)\]

Optimized under \(\mathcal{L} = \mathcal{L}_{\text{FM}} + \lambda \mathcal{L}_{\text{vel}}\) (\(\lambda = 0.5\)) along with Data-Noise Alignment, the model achieves faster convergence and more consistent trajectory integration.

4. Sound-Aware Captioner: Audio-Visual LLM Distillation for Robust Acoustic Conditioning Existing in-the-wild V2A datasets such as VGGSound provide sparse labels that omit acoustic nuances, while independent multi-model captioning pipelines suffer from severe cross-modal hallucinations (such as mistaking vehicle engine rumbles for animal roars). SoundCap solves this by using a two-stage distillation strategy: first, an audio-visual large language model (video-SALMONN) processes paired video and audio with explicit noise-filtering prompts to generate rich, acoustic-centric captions covering both on-screen actions and ambient sounds; second, these rich captions supervise the fine-tuning of a vision-language model (Qwen2.5-VL). At inference time, only the VLM runs on silent video, generating detailed sound-oriented captions that guide Flowley without requiring test-time audio inputs.

Loss & Training

The framework encodes 16 kHz audio downsampled mel-spectrograms into latent variables via a pretrained VAE. Global conditioning vectors are integrated into transformer layers via adaptive layer normalization (adaLN). To support classifier-free guidance (CFG, set to \(s = 7.5\) during inference), visual and textual tokens are independently masked with a 10% dropout probability during training. Trajectory generation is integrated using Euler's ODE solver in just 25 function evaluations (NFE).

Key Experimental Results

Main Results

On the VGGSound benchmark (evaluated on the first 8 seconds of test clips), Flowley was compared against seven recent state-of-the-art systems across distribution matching, generation quality, semantic alignment, and temporal synchronization (Table 1 of the original paper).

Method	Params	KAD ↓	FAD ↓	KL ↓	IS ↑	IB-Score (×100) ↑	LB-Score (×100) ↑	Align Acc ↑
Frieren	159M	1.27	12.80	2.82	12.02	22.45	19.09	97.13
FoleyCrafter	1.22B	1.54	19.17	2.19	15.09	25.75	24.66	77.15
V2A-Mapper	229M	1.34	11.73	2.50	12.43	22.38	22.32	79.08
MDSGen	131M	5.33	39.68	2.85	6.87	17.75	19.05	91.70
Mel-QCD	859M	1.53	19.17	2.09	10.32	23.79	23.80	73.85
VinTAGe	1.32B	1.08	17.88	2.15	17.34	21.10	21.51	67.11
MMAudio	157M	0.57	7.89	1.91	12.68	28.09	21.98	89.73
MMAudio + SoundCap	157M	0.39	7.09	1.56	14.68	28.85	22.64	90.53
Flowley (Ours)	169M	0.42	7.65	1.57	18.25	29.32	24.87	89.37
Flowley + SoundCap (Ours)	169M	0.39	7.52	1.56	19.68	30.07	25.33	90.02

Note: Frieren and MDSGen achieve high Align Acc partly because their video backbone overlaps with the Align Acc feature extractor, introducing evaluation bias; in subjective blind A/B user studies, human evaluators strongly preferred Flowley's synchronization.

Ablation Study

The impact of cross-attention components in the single-stream block and the effectiveness of PSCA are presented below (Table 3 from the original paper).

Config #	Text CA (\(\text{CA}_t\))	Visual CA (\(\text{CA}_v\))	PSCA (\(\text{PSCA}_v\))	KAD ↓	IS ↑	IB-Score (×100) ↑	Align Acc ↑
1 (Vanilla DiT)	✗	✗	✗	0.44	16.99	27.38	86.26
2 (Text CA only)	✓	✗	✗	0.44	17.29	28.93	87.43
3 (Standard Visual CA only)	✗	✓	✗	0.40	17.58	28.02	87.04
4 (PSCA Visual only)	✗	✗	✓	0.40	17.69	28.51	88.72
5 (Text CA + Standard Visual CA)	✓	✓	✗	0.40	18.15	28.74	87.61
6 (Flowley Full Setup)	✓	✗	✓	0.42	18.25	29.32	89.37

Additional ablations validate the progression parameter \(\beta\) (Table 4): removing \(\beta\) degrades IS from 18.25 to 17.78 and Align Acc from 89.37% to 88.62%. In SoundCap prompt noise conditioning ablations (Table 5), training without explicit noise warnings causes IS to collapse by 16.9% (from 18.25 to 15.16) and Align Acc to drop to 87.13%, underscoring the necessity of noise filtering when pseudo-labeling web-scale video.

Key Findings

• PSCA Eliminates the Need for Pretrained Alignment Modules: Progressive soft masking improves Align Acc from 87.61% to 89.37% over standard visual cross-attention without extra trainable weights, surpassing explicit onset-based baselines like FoleyCrafter (77.15%) and Mel-QCD (73.85%).
• Superior Parameter and Data Efficiency: With only 169M parameters, Flowley outperforms 1.32B VinTAGe on generation quality (IS 18.25 vs 17.34) and distribution divergence (KAD 0.42 vs 1.08).
• Outperforming Closed-Source 13B Models in Zero-Shot Testing: On the MovieGen Audio Bench zero-shot transfer test, Flowley + SoundCap trained on roughly 400 hours of VGGSound achieves an IS of 8.18, surpassing Meta's 13B Movie Gen Audio model (IS 8.01) trained on over one million hours of media.

Highlights & Insights

• Embedding Temporal Inductive Bias Directly into Latent Attention: Instead of treating temporal synchronization as a separate engineering module (e.g., onset predictors, energy trackers, or contrastive encoders), Flowley demonstrates that structuring cross-attention with smooth distance decay and layer-depth progression can achieve tight temporal alignment natively.
• Cross-Modal Distillation for Rich Acoustic Prompting: SoundCap demonstrates that leveraging unified AV-LLMs to generate audio-grounded captions avoids the hallucination pitfalls of disjoint audio and vision models, effectively converting raw video-audio pairs into informative supervision for text-guided generation.

Limitations & Future Work

• Domain Shift in Open-Domain Semantic Generalization: While Flowley achieves higher audio quality (IS) than Movie Gen Audio on synthetic benchmarks, its semantic alignment (IB-Score 25.78 vs 35.86) lags behind, reflecting the data-scale bottleneck of training solely on VGGSound's ~400 hours.
• Handling Off-Screen Audio Dynamics: PSCA enforces localized temporal attention around aligned video timestamps. When handling off-screen narrative cues, ambient reverberation, or anticipatory sound design where sound precedes visual manifestation, strict local masking may constrain expressive flexibility.

Related Work & Insights

• vs MMAudio: While MMAudio also targets joint visual-textual audio synthesis, it depends on an external pretrained audio-visual contrastive encoder; Flowley's PSCA achieves better distribution matching (KAD 0.42 vs 0.57) and audio quality (IS 18.25 vs 12.68) without external alignment modules.
• vs FoleyCrafter & Mel-QCD: Previous methods rely on discrete onset detectors or complex continuous-discrete mel-decomposition pipelines; Flowley streamlines the task into an end-to-end, single-stage flow matching framework that is simpler to train and deploy.

Rating

• Novelty: ⭐⭐⭐⭐ [The depth-progressive soft-masked attention elegantly embeds temporal locality without external alignment models]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Evaluated against 7 competitive baselines across four quantitative dimensions, thorough ablations, and extensive human A/B testing]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical formulations, clean structural hierarchy, and well-designed figures]
• Value: ⭐⭐⭐⭐⭐ [Provides a practical, highly efficient, and high-fidelity blueprint for automated Foley and video-to-audio production]

Related Papers

• [ICML 2026] Probing Cross-modal Information Hubs in Audio-Visual LLMs
• [AAAI 2026] CCFQA: A Benchmark for Cross-Lingual and Cross-Modal Speech and Text Factuality Evaluation
• [AAAI 2026] Cross-Space Synergy: A Unified Framework for Multimodal Emotion Recognition in Conversation
• [NeurIPS 2025] Latent Space Factorization in LoRA
• [ECCV 2026] Meric: A Unified Framework for Multimodal Music Generation and Retrieval via Representation Space Anchoring