Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interaction¶

Conference: ECCV 2026
arXiv: 2510.03117
Code: https://bridgedit-t2sv.github.io/
Area: Video Generation / Joint Audio-Video Generation / Diffusion Models
Keywords: Text-to-sounding video, dual-tower diffusion, cross-attention, caption decoupling, modality interference

TL;DR¶

For the text-to-sounding video generation (T2SV) task that simultaneously generates visual frames and synchronized audio, this paper proposes a dual-agent caption rewriting framework (CRR) to provide separate, interference-free, modality-pure captions for the video tower and the audio tower. It then employs a dual-tower diffusion model, BridgeDiT, using dual cross-attention (DCA) to bridge and exchange features between the two towers, achieving state-of-the-art (SOTA) performance on most metrics across three benchmarks.

Background & Motivation¶

Human perception is naturally multi-sensory, where visuals and sounds are tightly coupled. Thus, generating sounding videos directly from text (Text-to-Sounding-Video, T2SV) is regarded as a key step toward world models. Text-to-video (T2V, e.g., Wan, HunyuanVideo) and text-to-audio (T2A, e.g., Stable Audio Open, AudioLDM) generation have achieved mature progress individually, prompting the community to shift attention to the more challenging joint generation task. Early methods either generated the two modalities independently, completely lacking temporal synchronization, or used cascaded pipelines (T→V→A or T→A→V). However, downstream models in cascaded pipelines are trained only on ground-truth data, failing to rectify errors from the first stage, which instead propagates and accumulates errors. Consequently, research focus has shifted to joint generation, where the single-tower paradigm learns joint distributions from scratch but is heavily reliant on massive paired data and hard to optimize, making the dual-tower paradigm—reusing pre-trained T2V/T2A backbones and training only a lightweight interaction module—the dominant approach.

However, the dual-tower paradigm still suffers from two fundamental challenges. The first is the conditioning problem: prior dual-tower methods feed the same shared caption (\(T_V = T_A\)) to both towers. Yet, the two backbones are pre-trained on text from different modalities; the shared caption causes semantic interference—color descriptions in the visual scene act as noise to the audio model, and timbre descriptions disrupt the video model. More troublesome, these models are trained on dense, detailed captions, but at inference, users only provide a short prompt. This distribution gap significantly degrades generation quality. The second is the interaction problem: the interaction module is responsible for exchanging information between the two towers to achieve synchronization, but how it should be designed—full self-attention, unidirectional conditioning, or additive fusion—lacks systematic consensus. Notably, recent unified subtitling schemes like LTX-2 and Ovi primarily target speech/dialogue (where lip-sync naturally fits unified text representation), whereas this paper focuses on foley and sound effects, where modality interference is more prominent.

This paper tackles these two problems separately: the conditioning problem is addressed by first cross-referencing captions from both modalities to distill trustworthy semantic anchors, and then rewriting them into two modality-pure, non-interfering captions; the interaction problem is analyzed by systematically comparing various fusion strategies under identical conditions to find the optimal design. Core Idea: Use a dual-agent Cross-Referential Rewriter to generate modality-pure, train-test aligned decoupled captions (\(T_V \neq T_A\)), and build a lightweight "manifold bridge" between the frozen dual towers using dual symmetric Cross-Attention (DCA), which preserves individual feature spaces while achieving bidirectional synchronization.

Method¶

Overall Architecture¶

The system consists of two decoupled components. The frontend is the CRR framework (Cross-Referential Rewriter), a data/prompt-side dual-agent pipeline: during training, given a sounding video, visual LLM and audio LLM extract raw captions respectively, which are cross-referenced by a semantic checker to filter out hallucinations and yield structured "semantic anchors"; then, a cross-modality rewriter generates two modality-pure dense captions, \(T_V\) (for the video tower) and \(T_A\) (for the audio tower), based on these anchors. During inference, the same pipeline expands the user's short prompt into dense captions matching the training distribution. The backend is the generation backbone BridgeDiT: a video DiT tower (Wan 2.1 1.3B) and an audio DiT tower (Stable Audio Open) run in parallel with most parameters frozen. Four lightweight BridgeDiT Blocks are inserted between specified layers, utilizing Dual Cross-Attention (DCA) for bidirectional feature exchange before feeding the updated features back to the respective towers. The entire pipeline takes text as input and outputs 5.4-second synchronized video paired with 44.1kHz audio.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    IN["Input: Sounding Video (Train) / User Prompt (Inference)"] --> RAW["Visual LLM + Audio LLM<br/>Extract Raw Captions"]
    RAW --> CRR["Cross-Referential Rewriter CRR<br/>Semantic Anchors → Decoupled Rewriting"]
    CRR -->|"Video Caption T_V"| VT["Video DiT Tower<br/>(Frozen Backbone)"]
    CRR -->|"Audio Caption T_A"| AT["Audio DiT Tower<br/>(Frozen Backbone)"]
    VT <--> DCA["Bidirectional Interaction Bridge DCA<br/>4×BridgeDiT Block"]
    AT <--> DCA
    VT --> OUT["Output: Synchronized Video + Audio"]
    AT --> OUT

Key Designs¶

1. Semantic Anchors: Grounding the Captions to Prevent Hallucinations

The pain point is straightforward: if a single LLM is used directly to write separate captions for the two modalities, the audio LLM will frequently hallucinate sounds that do not exist in the video—for example, given a scene of "a worker hammering iron", the audio LLM might output "tapping sounds, resembling stone hitting", completely failing to identify the actual sound source. The root of the problem is not "generating captions" itself, but the lack of a grounding mechanism between the two modality captions. CRR's solution is to split caption generation into two agents with a structured bottleneck in between: the semantic checker \(\mathcal{F}_{sc}\) distills the inputs into semantic anchors \(\mathcal{A}\)—a set of verified attributes containing core entities, environments, visual actions, and their corresponding acoustic events (e.g., {Entity: worker, Env: workshop, Action: hammer strike, Sound: metallic clang}). During training, it uses reliable visual descriptions as a reference to correct and filter audio descriptions, retaining only semantically consistent events in \(\mathcal{A}\). Subsequently, the cross-modality rewriter \(\mathcal{F}_{cr}\) strictly follows \(\mathcal{A}\) to write dense captions:

\[\mathcal{A} = \mathcal{F}_{sc}(\mathcal{I}), \quad T_V, T_A = \mathcal{F}_{cr}(\mathcal{A})\]

Because \(\mathcal{A}\) only retains grounded semantics and discards modality-specific noise, the rewriter is structurally prevented from introducing unverified events—modality purity is guaranteed by the architecture, not by hoping the LLM will behave. Ablations demonstrate that removing this anchor bottleneck and letting a single LLM handle the rewrite in one step (Direct Rewrite) degrades the audio FAD from 5.34 to 15.74, which proves that the anchor is not just a prompting trick, but a genuinely functioning grounding mechanism.

2. Train-Inference Prompt Expansion: Smoothing the Distribution Gap with a Single Pipeline

The pain point is that the model is trained on dense, detailed captions, but users provide only ultra-simple prompts like "a worker hammering iron" at inference, creating a huge distribution gap that collapses generation quality if fed directly. The elegance of the proposed method is that inference and training reuse the same CRR pipeline, only changing the input \(\mathcal{I}\): during training, \(\mathcal{I}\) is the raw captions, following the "cross-referencing" logic; during inference, \(\mathcal{I}\) is the user prompt, and the semantic checker switches to "contextual inference" mode—inferring implicit visual scenes and acoustic events from the brief prompt ("worker hammering" \(\to\) the aforementioned anchors), which are then expanded by the rewriter into decoupled \(T_V\) and \(T_A\) with density matching the training captions. This fills in missing details for user prompts and aligns the inference input with the training distribution without requiring complex prompt engineering from the user. Removing this prompt expansion step in the ablation studies led to the most severe performance degradation (FVD soared from 765 to 1463, FAD to 20.84), confirming that the "short user prompt vs. dense training caption" gap is the primary quality bottleneck.

3. Bidirectional Cross-Attention (DCA): Bridging Disjoint Manifolds

This is the core of the interaction module. The challenge lies in the fact that the two towers are pre-trained on their respective modalities and mostly frozen, meaning their features reside in two disjoint manifolds \(\mathcal{M}_V\) and \(\mathcal{M}_A\)—which is fundamentally different from training from scratch (such as MMDiT, where all modalities converge to a shared space). DCA employs two symmetric cross-attention streams to update video latents \(L_V\) and audio latents \(L_A\). Taking the A2V direction as an example, the video latent acts as the query, while the audio latent provides key/value, and the attention result is added back residually to the video latent (the V2A direction is fully symmetric):

\[L'_V = \text{Attention}(Q_V, K_A, V_A) + L_V\]

Why is DCA the correct choice under the "frozen dual-tower" setup? Two reasons: First, DCA preserves individual feature spaces—unlike full self-attention, which concatenates both modalities into a single sequence and mixes intra- and inter-modality dependencies within one attention matrix, DCA keeps the two streams separate, letting each tower receive targeted cross-modal cues. Second, this separation creates an easier optimization target—instead of using limited trainable parameters to force \(\mathcal{M}_V\) and \(\mathcal{M}_A\) into a shared space, it learns a lightweight bridge between the two established manifolds (\(\mathcal{M}_V \leftrightarrow \mathcal{M}_A\)). This is a fundamentally simpler task under a fixed parameter budget, leading to more stable training and stronger synchronization. In contrast, unidirectional variants (V2A or A2V-CrossAttn) feed information in only one direction, causing asymmetric side effects where one modality improves while the other is neglected (e.g., A2V lowers FVD but sacrifices synchronization to an AV-Align of only 0.236). DCA enables both modalities to benefit simultaneously via bidirectional exchange. Moreover, BridgeDiT Blocks perform best when placed in the early-to-middle layers (video [3, 7, 11, 15], audio [2, 5, 8, 11]), as these layers capture both early-stage spatio-temporal details and mid-stage semantic abstractions; relying solely on high-level abstract features from the final layers lacks the precision required for T2SV alignment.

Loss & Training¶

The total loss is the sum of the two tower losses: \(\mathcal{L} = \mathcal{L}_{\text{audio}} + \mathcal{L}_{\text{video}}\). The audio tower uses a v-prediction diffusion loss (with continuous timestep \(t_A \in [0,1]\), noise-to-signal scaling \(\alpha(t_A)=\cos(t_A\pi/2)\), \(\sigma(t_A)=\sin(t_A\pi/2)\), and prediction target \(\alpha(t_A)\epsilon_A - \sigma(t_A)\mathbf{x}_A\)). The video tower uses a flow matching loss (predicting the vector field \(\epsilon_V - \mathbf{x}_V\), with the video timestep defined as \(t_V = 1000 \cdot t_A\) to align with the audio). The backbones are mostly frozen; only the last 5 blocks of each tower and the 4 BridgeDiT Blocks are trained. The model is trained independently on each dataset, taking roughly 8 hours for 15,000 steps on 4 H100 GPUs for AVSync15. During inference, independent CFG scales (both set to 6.0) are applied to control the video and audio generation.

Key Experimental Results¶

Main Results¶

On AVSync15, BridgeDiT achieves the best or second-best performance across most metrics in video quality, audio quality, audio-text alignment, and temporal synchronization. The table below compares several representative baselines (↓ lower is better, ↑ higher is better):

Method	Paradigm	FVD↓	FAD↓	KL↓	CLAP↑	AV-Align↑
Wan + MMAudio	T→V→A Cascade	828.33	7.98	1.40	34.64	0.243
JointDiT	T→I→VA Joint	992.71	6.51	1.77	30.34	0.156
JavisDiT	T→VA Joint	878.70	13.48	3.50	22.99	0.158
CoDi	T→VA Joint	1387.14	16.56	5.24	17.94	0.081
BridgeDiT (ours)	T→VA Joint	765.74	5.34	1.30	35.95	0.275

It also leads on VGGSound-SS and Landscape (e.g., Landscape: FVD 628.07 / FAD 4.78 / AV-Align 0.258, all best). In a 20-person user study, BridgeDiT ranked first across all 5 dimensions: video quality, audio quality, text alignment, sync, and overall (Overall score: 3.34, followed by the cascaded Wan+MMAudio at 3.06). The authors honestly note two minor exceptions: CLIPSIM (28.52) is slightly lower than JointDiT (29.94) due to the latter's stronger T2I backbone Qwen-Image; and VA-IB (34.59) is surpassed by SeeingHearing (35.87) because the latter directly uses ImageBind as guidance during inference to optimize this specific metric—yet, ImageBind was not preferred by human evaluators, validating that automatic metrics do not entirely equate to perceived quality.

Ablation Study¶

Ablation study on caption strategies (AVSync15):

Caption Strategy	FVD↓	FAD↓	CLAP↑	AV-Align↑	Description
Shared, Video Caption Only (\(T_V\))	788.65	16.46	9.67	0.176	Audio tower lacks acoustic details, poor audio alignment
Shared, Merged Caption (\(T_{AV}\))	1362.83	13.75	26.37	0.185	Semantic interference, collapsed video quality
Decoupled, Raw LLM Captions (w/o CRR)	924.36	19.42	8.43	0.161	Hallucinations + cross-modal conflict, worst decoupling
Decoupled, Direct Rewrite (w/o Semantic Anchor)	787.32	15.74	27.34	0.224	Missing anchor bottleneck, obvious drops in audio and sync
CRR (ours)	765.74	5.34	35.95	0.275	Complete framework
CRR, w/o Prompt Expansion (Inference)	1463.81	20.84	22.45	0.155	Distribution gap unresolved, worst degradation

Ablation study on fusion mechanisms (AVSync15):

Fusion Type	FVD↓	FAD↓	CLAP↑	VA-IB↑	AV-Align↑
No-Fusion	828.33	11.90	30.78	26.22	0.205
Full-Fusion (Full self-attention)	781.03	5.62	32.28	34.14	0.253
Additive-Fusion (Additive)	772.23	5.72	34.77	28.34	0.258
V2A-CrossAttn (Unidirectional)	813.02	6.21	35.85	34.20	0.268
A2V-CrossAttn (Unidirectional)	746.37	5.91	31.25	31.54	0.236
DCA (ours)	765.74	5.34	35.95	34.59	0.275

Key Findings¶

Both components are indispensable: Eliminating prompt expansion scales FVD from 765 to 1463 (the largest degradation), showing that the training-testing distribution gap is the number-one bottleneck. Shared captions (either video-only or merged) collapse due to either missing information or semantic interference, proving the necessity of decoupled conditions.
Semantic anchors are not prompt engineering tricks: Direct rewrite (removing the anchor bottleneck) increases audio FAD to 15.74 compared to 5.34 for the full CRR. This huge gap proves that structural anchors act as an essential grounding mechanism to prevent the rewriter from hallucinating events.
DCA comprehensively outperforms full self-attention: The authors attribute this to the frozen backbone constraint. Full self-attention tries to force two disjoint manifolds into a shared space, which is difficult to optimize within a fixed parameter budget (leading to unstable training curves). DCA only learns a lightweight bridge, which is a fundamentally simpler task, resulting in more stable training and superior synchronization throughout.
Unidirectional fusion yields asymmetric side effects: A2V improves video (FVD 746) but sacrifices synchronization (AV-Align 0.236), while V2A does the opposite. Only bidirectional DCA benefits both modalities simultaneously.
Interaction layer placement is sensitive: Placing them in the early-to-middle layers (Uniform Early Bias) is optimal (VA-IB 34.59 / AV-Align 0.275), while placing them strictly in the final layers leads to significant degradation (VA-IB only 19.32), because precise alignment relies on both early-stage spatio-temporal details and mid-stage semantics.
Improvements stem from methodology, not scale: Swapping the 72B LLM in CRR with a 7B LLM still yields strong results (FAD 8.26, AV-Align 0.258). Furthermore, replacing the backbone (SANA + AudioLDM) paired with DCA improves every metric, showing that generalization originates from the core design.

Highlights & Insights¶

Architecting "hallucination prevention" instead of relying on prompts: The semantic anchor serves as a structural bottleneck, physically preventing the rewriter from fabricating unverified sounds. This "information-bottleneck-based constraint" is far more reliable than writing "do not hallucinate sound" in a prompt, and can be generalizable to any scenario requiring cross-verification of multi-source information before generation.
Reusing the same captioning pipeline for both training and inference: By simply changing the input \(\mathcal{I}\) and switching the "cross-reference" mode to "contextual inference," the pipeline simultaneously addresses caption decoupling and prompt expansion. It is engineeringly elegant yet directly tackles the distribution gap, the biggest quality hazard.
Systematic empirical analysis on fusion strategies: Instead of just presenting DCA, the paper directly compares full attention, additive fusion, and unidirectional/bidirectional cross-attention under identical conditions. It explains why DCA wins using a manifold perspective ("building a bridge rather than merging manifolds under a frozen backbone"), offering valuable insights for any multi-tower architecture built on pre-trained backbones.
Honesty towards automatic metrics: The authors proactively identify and analyze the reasons for lagging behind on specific metrics (such as CLIPSIM and VA-IB) and cross-verify with user studies, avoiding misleading metric-hacking.

Limitations & Future Work¶

Evaluation is confined to relatively small-scale datasets (up to ~5K videos), which may not fully represent the diverse acoustics of the real world.
Currently, speech and music generation are not supported. Extending to these modalities is bottlenecked by the scarcity of open-source pre-trained models and would require resource-intensive training from scratch. Moreover, the speech scenario (where lip-sync naturally fits unified representations) was intentionally avoided, leaving it for unified captioning schemes like LTX-2/Ovi.
Post-training refinement (e.g., RLHF) remains unexplored. The authors believe that RLHF with rewards specifically designed for audio-video sync is a promising direction for aligning with human perception.
Self-observation: The model must be trained on each dataset separately, and CRR relies on 72B LLMs for offline expansion (though cached), leading to non-trivial deployment/preprocessing costs. Although OOD experiments show generalization, the benchmark domains are still relatively narrow.

vs Cascaded Pipelines (e.g., T→V→A like Wan+MMAudio): Cascade pipelines generate sequentially. The second stage, trained only on ground-truth data, cannot correct—and instead amplifies—errors from the first stage. This paper's joint generation model eliminates error propagation at the root, outperforming cascade baselines across almost all quality and alignment metrics.
vs Single-Tower Joint Generation (e.g., CoDi, MMDiffusion): They learn the joint distribution from scratch, which is highly data-hungry and difficult to optimize. This paper utilizes a dual-tower setup that reuses pre-trained backbones and trains only a lightweight interaction module, making both data and optimization much more manageable.
vs Dual-Tower + Full Self-Attention (JointDiT): They concatenate both modalities into a single sequence for joint self-attention. With frozen backbones, this forces two disjoint manifolds into a shared space, making optimization hard and synchronization unstable. This paper utilizes DCA to preserve individual feature spaces while building a bidirectional bridge, leading to superior sync metrics in the fusion ablation.
vs Dual-Tower + Unidirectional Conditioning (e.g., V2A in SyncFlow, A2V in MTV): They restrict information flow to one direction (ControlNet-style), causing asymmetric performance improvements. This paper employs bidirectional symmetric exchange, allowing both modalities to benefit concurrently.
vs Shared/Unified Caption Schemes (e.g., CoDi's vision-centric captions, LTX-2/Ovi's unified captions): Vision-centric captions lack acoustic details, while unified captions are tailored for speech/conversations (lip-syncing). This paper introduces a decoupled captioning paradigm for foley and sound effects, adopting semantic anchors to ensure modality purity and prevent interference.

Rating¶

Novelty: ⭐⭐⭐⭐ The semantic anchor bottleneck in CRR and the manifold-bridge perspective of DCA introduce substantial innovations. Although individual techniques (such as cross-attention and caption rewriting) are not brand new, their combination and theoretical backing are highly robust.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Three benchmarks + user study + multi-dimensional ablations (including caption strategy, fusion, layer placement, LLM scale, backbone variation, and OOD generalization) successfully validate every claim.
Writing Quality: ⭐⭐⭐⭐⭐ The problems (C1/C2) are clearly deconstructed. Lags in automatic metrics are honestly analyzed, and the manifold perspective explains why DCA wins exceptionally well.
Value: ⭐⭐⭐⭐ Provides direct, reusable empirical findings on captioning design and fusion architectures for dual-tower T2SV systems, offering highly practical guidance for the foley and sound-effect domain.