Conditional Flow Matching for Visually-Guided Acoustic Highlighting¶
Conference: ECCV 2026
Paper: ECCV Official
Area: Multimodal VLM
Keywords: Audiovisual Learning, Smart Remixing, Flow Matching, Conditional Flow Matching, Acoustic Highlighting
TL;DR¶
Reframes visually-guided acoustic highlighting as a generative Conditional Flow Matching (CFM) problem, introducing a trajectory rollout loss to suppress compounding drift and an early cross-modal adapter in the vision encoder for explicit source selection.
Background & Motivation¶
With the rapid growth of video content creation and consumption, ensuring harmony between visual and auditory cues has become essential for immersive user experiences. While visual post-processing techniques such as viewpoint selection, color grading, and dynamic rendering have matured, audio processing has not kept pace. Standard cameras and handheld recording devices capture environmental sounds indiscriminately through omnidirectional microphones. As a consequence, spoken dialogue is often overwhelmed by background noise or competing soundtrack elements, breaking the perceptual alignment between what viewers see and what they hear.
To address this disconnect, the task of Visually-guided Acoustic Highlighting (VisAH) was established to automatically rebalance audio loudness in alignment with the visual narrative. Existing state-of-the-art methods approach this task through a discriminative paradigm using sound source separation backbones like HybridDemucs. This setup formulates the problem as an explicit one-to-one point mapping from poorly balanced to well-balanced audio. However, given a dynamic video scene, multiple plausible high-quality mixes can exist, just as degraded mixtures present diverse acoustic corruption patterns. Rigid discriminative regression fails to model such many-to-many distributional uncertainty, frequently resulting in over-smoothed or unnatural mixes.
While generative flow matching offers an ideal mechanism to transport the ill-balanced audio distribution toward the target mix distribution, standard flow integration faces a unique hurdle: in acoustic highlighting, the crucial decision of which source to amplify or attenuate must occur at the earliest steps. Small early velocity estimation inaccuracies inevitably compound across successive iterations, leading to catastrophic off-manifold trajectory drift. Core idea: formulate visually-guided acoustic highlighting as conditional flow matching, supervised by a full-trajectory rollout loss that penalizes endpoint drift to enable self-correcting dynamics, and equipped with an early audio-visual cross-attention adapter inside CLIP to offload source selection from the denoising U-Net.
Method¶
Overall Architecture¶
VisAH-FM establishes a continuous probability flow transporting samples from the ill-balanced input distribution \(\pi_0\) to the target balanced distribution \(\pi_1\). Given an unaligned mono waveform \(x_0\), synchronized video frames, and an integration timestep \(t \in [0, 1]\), the framework estimates a time-dependent velocity field \(v_\theta(x_t, t, c)\) to guide the state trajectory toward an optimal acoustic mix via a few-step ODE integration.
The architecture coordinates three major components: first, a multi-modal conditioning pipeline extracts visual features from video frames via CLIP and injects auditory context via a lightweight cross-attention adapter linked to a frozen CLAP audio encoder; second, a sinusoidal timestep embedding is concatenated with cross-modal tokens and attended into the dual-branch U-Net latent space through temporal transformers; third, a dual-branch U-Net (operating simultaneously on STFT spectrograms and raw temporal waveforms) estimates the instantaneous velocity field, iteratively updated under the supervision of the rollout mechanism.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: Ill-balanced Waveform x0 & Video Frames"] --> B["Early Multimodal Adapter<br/>CLAP Audio Injected into CLIP Vision Layers"]
B --> C["Conditional Flow Field Regression<br/>Dual-branch U-Net Predicts Velocity Field vt"]
C --> D["Trajectory Rollout Supervision<br/>T-step Recurrent Integration with Self-correction"]
D --> E["Output: Visually-aligned Rebalanced Audio"]
Key Designs¶
1. Early Multimodal Adapter: offloading source identification to conditioning In prior discriminative baselines, the conditioning module ingested only vision features or caption embeddings, compelling the downstream U-Net to simultaneously decipher audio-visual correspondences and regress audio waveforms. VisAH-FM decouples these responsibilities by introducing a cross-modal adapter directly into intermediate layers (the 18th layer) of the CLIP vision encoder. Given layer-\(k\) visual tokens \(F_k\) and CLAP auditory embeddings \(E\), cross-attention is computed across projected representations: $\(m(F_k, E) = \mathrm{Attn}(E W_{\mathrm{down}}^E, F_k W_{\mathrm{down}}^F) W_{\mathrm{up}}^E\)$ The adapted representation is then added via a gated residual connection: \(\mathrm{adapter}(F_k, E) = F_k + \lambda_E m(F_k, E)\), where \(\lambda_E\) is initialized to 0 to preserve pretrained visual features at initialization. This allows the conditioning module to filter out visual regions irrelevant to active sound sources beforehand, providing the denoising U-Net with audio-aware visual tokens and freeing it to specialize purely in velocity field estimation.
2. Trajectory Rollout Supervision: mitigating error compounding across integration steps Standard flow matching evaluates the velocity field independently at each timestep using ground-truth interpolants \(x_t\), ignoring drift caused by the model's own intermediate errors. Because the decision to highlight speech, music, or sound effects must be made during initial steps, early errors propagate and compound down the trajectory. Capitalizing on the short distribution distance between input and output audio (which requires only \(T = 4\) Euler steps), VisAH-FM implements an end-to-end backpropagation through the flow during training, adding an auxiliary MSE rollout loss at the final trajectory endpoint: $\(\mathcal{L}_{\mathrm{Rollout}}(\theta) = \mathbb{E}_{x_0} \left[ \|\hat{x}_T - x_T\|_2^2 \right]\)$ Supervising the final prediction after \(T\) recurrent applications teaches the network to recover from self-generated intermediate errors, resolving exposure bias and anchoring long-range integration trajectories.
3. Velocity Field Parameterization: leveraging discriminative pretraining priors To exploit pretrained weights from the discriminative VisAH baseline without retraining from scratch, the velocity field is re-parameterized as \(v_\theta(x_t, t, c) = x_0 - u_\theta(x_t, t, c)\), where \(u_\theta\) is the neural network. Training leverages dynamic on-the-fly audio corruption rather than a fixed dataset, exposing the model to diverse degradation variants per video target across training epochs. During inference, fixed-step Euler integration provides an explicit control knob where intermediate steps represent physically valid mixes, allowing users to modulate highlighting intensity by terminating early.
Loss & Training¶
The overall training objective combines conditional flow matching with the auxiliary rollout loss: $\(\mathcal{L}(\theta) = \mathcal{L}_{\mathrm{CFM}}(\theta) + \lambda \mathbb{E}_{x_0} \left[ \|\hat{x}_T - x_T\|_2^2 \right]\)$ where the CFM objective minimizes regression between \(v_\theta(x_t, t, c)\) and the optimal transport vector field \(\frac{x_1 - x_t}{1 - t}\), with \(x_t = (1 - t)x_0 + t x_1\). The rollout weight is set to \(\lambda = 0.3\) and integration steps to \(T = 4\). Initialized with VisAH weights, the model is trained on 44.1 kHz mono audio for 50 epochs (\(\sim\)23,500 iterations) with batch size 32, using a cosine annealing learning rate schedule starting at \(1 \times 10^{-4}\).
Key Experimental Results¶
Main Results¶
On the Muddy Mix movie test set, VisAH-FM is evaluated against discriminative VisAH baselines across multimodal alignment (ImageBind Score / IB Score), semantic divergence (KLD), source loudness difference (LDif), and signal fidelity metrics (Magnitude distance Mag, Envelope distance Env, Wasserstein distance Was). All values except LDif are scaled by 100 as in the original paper:
| Model | Conditioning | IB Score ↑ | KLD ↓ | LDif ↓ | Mag ↓ | Env ↓ | Was ↓ |
|---|---|---|---|---|---|---|---|
| Ill-balanced Input | - | 28.14 | 20.74 | 18.36 | 22.69 | 6.29 | 1.96 |
| VisAH | CLIP (Vision) | 28.84 | 11.37 | 9.66 | 9.99 | 3.38 | 0.84 |
| VisAH | T5 (Text Captions) | 28.92 | 11.71 | 9.63 | 10.22 | 3.44 | 0.88 |
| VisAH | CLIP-CLAP | 28.82 | 11.28 | 9.60 | 10.27 | 3.56 | 0.80 |
| VisAH-FM (Ours) | CLIP-CLAP | 29.12 | 9.70 | 7.77 | 8.28 | 2.74 | 0.63 |
Ablation Study¶
Ablation on rollout supervision components and alternative multi-step consistency mechanisms:
| Objective / Configuration | IB Score ↑ | KLD ↓ | LDif ↓ | Note |
|---|---|---|---|---|
| FM + Rollout (Full Model) | 29.09 | 9.79 | 7.87 | Full flow matching with self-correcting rollout supervision |
| FM (Vanilla Flow Matching) | 28.92 | 10.99 | 9.48 | Lacks end-to-end rollout; compounding error degrades flow |
| Rollout Only (No CFM Loss) | 28.94 | 9.92 | 7.71 | Direct multi-pass recurrent regression; learns non-linear paths |
| FM + Consistency | 28.40 | 14.58 | 10.30 | Local timestep matching propagates and amplifies noise |
| Bridge Matching | 29.04 | 10.85 | 9.62 | Gaussian white noise perturbation fails to model raw waveform errors |
| FM Weighted | 28.86 | 11.16 | 9.44 | Time-weighted loss prioritizing early steps cannot resolve drift |
Key Findings¶
- Rollout Loss eliminates off-manifold drift: Omitting the rollout loss from CFM increases KLD from 9.79 to 10.99 (+1.20) and worsens loudness difference LDif from 7.87 to 9.48 (+1.61). Adjacent consistency loss (FM + Consistency) causes severe cascading degradation (KLD surges to 14.58), whereas Rollout Loss anchors the final prediction to ground truth and prevents catastrophic drift.
- Audio-visual fusion efficiency: In conditioning ablations, early fusion of vision and audio (\(V + A\)) yields the best overall performance (IB Score 29.12, KLD 9.70, LDif 7.77). Incorporating heavy text features (\(V + T + A\)) yields no gain (LDif 7.90) while requiring an expensive forward pass through an 8B/11B VLM and large language model.
- Trajectory geometry confirmation: Trajectory curvature analysis reveals that vanilla FM exhibits severe bending after step 1 (discrete curvature \(\Theta_t\) climbs to \(\sim\)0.70), confirming off-manifold deviation. Adding rollout loss stabilizes curvature between 0.20 and 0.33, ensuring straight, well-behaved integration paths.
Highlights & Insights¶
- Generative rebalancing over discriminative fitting: Formulating acoustic highlighting as continuous distribution transport captures the multi-modal variability inherent in professional audio remixing, avoiding over-smoothed average predictions.
- Differentiable flow rollout for exposure-bias mitigation: Exploiting low-step integration (\(T = 4\)) to backpropagate through the entire ODE integration solves exposure bias in continuous generative modeling without requiring hundreds of sampling steps.
- Decoupled cross-modal architecture: By placing an audio adapter inside CLIP, the model isolates source selection within the conditioning network, allowing the U-Net denoiser to dedicate its capacity entirely to acoustic velocity field regression.
Limitations & Future Work¶
- High-complexity acoustic artifacts: Qualitative analysis reveals occasional residual high-frequency phase distortion or incomplete suppression in densely overlapping multi-speaker acoustic scenes.
- Visual occlusion sensitivity: Because source selection is guided by visual prominence, scenes with off-screen dialogue, severe occlusions, or misleading camera focal points can lead to suboptimal audio weighting.
- Future directions: Integrating adaptive time-step ODE solvers and exposing user-controllable interactive mixing sliders to adjust the relative highlighting gains of individual character stems.
Related Work & Insights¶
- vs VisAH (CVPR 2025): VisAH relies on discriminative HybridDemucs to directly regress balanced audio under an assumed one-to-one mapping; VisAH-FM reformulates this into generative flow matching with rollout supervision, achieving a 45.55% vs 17.17% subjective win rate.
- vs Consistency Models: Consistency models enforce local prediction matching across infinitesimal time steps, which can amplify early estimation errors; VisAH-FM supervises the global endpoint through recurrent flow rollout, ensuring robust trajectory convergence.
- vs Video-to-Audio Synthesis (Movie Gen / MMAudio): Generative video-to-audio synthesis creates sounds from scratch, often altering original timbre and vocal content; VisAH-FM performs source-preserving rebalancing that retains the original recording's fidelity.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ An elegant reformulation of acoustic highlighting into flow matching, coupled with an end-to-end rollout loss and modular cross-modal adapter.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across objective audio metrics, source separation loudness difference, trajectory curvature analysis, and subjective listening tests.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear exposition, thorough problem formalization, and transparent ablation of trajectory dynamics.
- Value: ⭐⭐⭐⭐☆ Offers a practical, principled foundation for automated video sound mixing, mobile video enhancement, and multimedia post-production.