JoVA: Unified Multimodal Learning for Joint Video-Audio Generation and Editing

Conference: ECCV 2026
Paper: ECCV Official
Project: https://visual-ai.github.io/jova
Area: Audio & Speech
Keywords: Joint Video-Audio Generation, Video-Audio Editing, Joint Self-Attention, Lip-Sync, Unified Multimodal Learning

TL;DR

JoVA introduces a streamlined, unified diffusion framework for joint video-audio generation and editing, replacing fragmented cross-modal attention layers with native joint self-attention and channel-wise conditioning alongside an adaptive mouth-area flow matching loss to achieve state-of-the-art multimodal synchronization and instruction-guided editing.

Background & Motivation

Recent years have witnessed rapid progress in AI-driven generative modeling, transitioning from single-image synthesis to photorealistic video generation. This momentum has catalyzed several specialized subfields: open-domain video generation, joint video-audio synthesis, instruction-guided video editing, and speech-driven talking-head animation. However, these research streams currently operate in near-total isolation. To address individual sub-tasks, practitioners commonly engineer task-specific architectures or splice auxiliary multimodal fusion layers (such as separate cross-attention blocks) into pretrained visual backbones. This fragmented approach not only multiplies architectural complexity and maintenance overhead, but also risks catastrophic disruption to pretrained visual priors because the auxiliary alignment modules are typically trained on multimodal corpora orders of magnitude smaller than the foundational video datasets.

The core tension stems from a fundamental conflict: immersive content creation intrinsically demands tight temporal and spatial synchrony between visual actions and acoustic events across both generative synthesis and fine-grained editing. Yet existing unified video models are largely silent, lacking audio generation capabilities entirely, while state-of-the-art joint video-audio generators rely on rigid dual-branch setups that cannot generalize to instruction-driven editing. Furthermore, extending diffusion models to reference-conditioned video editing via standard token sequence concatenation triggers severe quadratic computational explosions; simultaneously, the human mouth region occupies an average of merely ~2.3% of the total pixel area, causing standard global diffusion objectives to dilute the weak supervision signal required for precise phoneme-to-viseme lip alignment.

The angle of attack in this paper is that rather than stacking superficial cross-modal bridge networks, video, audio, and textual representations should interact directly within a shared latent space via native Transformer attention. Core idea: eliminate external cross-modal alignment modules via native Joint Self-Attention, incorporate visual references through Channel-Wise conditioning with binary masks to prevent token sequence explosion, and introduce an adaptive localized Mouth-Area Loss to achieve unified, high-fidelity joint video-audio generation and editing within a single architecture.

Method

Overall Architecture

JoVA is constructed upon a transformer-based diffusion backbone (specifically scaling the 12B Waver architecture) optimized under the flow matching formulation. The input pipeline ingests visual latents compressed spatio-temporally by the Wan2.1 VAE, audio spectrogram latents encoded via the MMAudio VAE, and multimodal instruction prompts jointly encoded by T5-XXL and Qwen2.5-32B. The model comprises a visual branch, an audio branch, and shared language conditioning, executing joint self-attention across concatenated token sequences within every transformer block.

To enforce temporal coherence between modalities, JoVA incorporates the temporally aligned Rotary Position Embedding (RoPE) from MMAudio, synchronizing position encodings along the shared temporal dimension. For video editing and image-to-video conditions, visual reference features are concatenated along the channel dimension with target noise and an explicit binary mask before passing through a linear projection layer. For human speech synthesis, facial keypoint tracking extracts a localized mouth-region bounding box, mapping it to latent resolution to impose focused flow matching supervision.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Inputs<br/>Video/Image Reference + Text Instructions + Target Noise"] --> B["Modality-Specific Encoding & Channel Injection<br/>Video VAE Channel Concat / MMAudio VAE"]
    B --> C["Native Joint Self-Attention Interaction<br/>Concatenated Video-Audio-Text Tokens in Shared Transformer"]
    C --> D["Temporal Alignment & Flow Matching<br/>Temporally Aligned RoPE + Unified Velocity Prediction"]
    D --> E["Localized Mouth-Area Loss Supervision<br/>Facial Keypoint Tracking + Latent Mask Reweighting"]
    E --> F["Joint Video-Audio Output<br/>High-Fidelity Synchronized Video & Speech/Audio"]

Key Designs

1. Native Joint Self-Attention: Eliminating External Fusion Layers and Prior Drift

Prior joint video-audio architectures typically maintain disjoint video and audio branches, relying on newly initialized cross-attention blocks to exchange cross-modal signals. Such newly introduced alignment weights are prone to undertraining and often distort the rich spatio-temporal priors of the visual backbone. JoVA resolves this by duplicating the pretrained video backbone parameters to initialize the audio diffusion branch and processing all modalities inside a single unified self-attention operation. Within each Transformer block, video tokens \(\mathbf{h}_v\), audio tokens \(\mathbf{h}_a\), and instruction tokens \(\mathbf{h}_t\) are concatenated directly along the sequence axis: $\([\mathbf{h}'_v; \mathbf{h}'_a; \mathbf{h}'_t] = \text{JointAttn}([\mathbf{h}_v; \mathbf{h}_a; \mathbf{h}_t])\)$ This full token-level self-attention enables symmetric bidirectional information flow, allowing acoustic and visual features to condition each other dynamically across every layer. Synchronized temporal RoPE locks the relative frame-to-audio positional relationships, ensuring tight temporal correspondence across the continuous sequence without extraneous parameters.

2. Channel-Wise Conditioning for Unified Video-Audio Editing

Introducing visual reference frames into diffusion models via token concatenation leads to prohibitive quadratic complexity growth with sequence length. JoVA addresses this by adopting modality-tailored conditioning paths. For the audio stream, since audio latent sequences are exceptionally compact (accounting for less than 1% of the visual token length), reference audio latents can be appended directly via token concatenation without noticeable overhead. For the visual stream, JoVA feeds the reference video or image latents, the noisy target latents, and a binary guidance mask along the channel dimension into a lightweight projector. The binary mask indicates the availability of reference context (1 for valid reference frames, 0 for default padded frames). This design preserves fixed spatial-temporal sequence lengths, eliminating computational bottlenecks while allowing a single unified network to natively handle text-to-video-audio generation, reference-image-to-video-audio animation, and instruction-driven joint video-audio editing.

3. Localized Mouth-Area Loss for Fine-Grained Lip-Speech Synchronization

In full-frame video synthesis, the mouth region occupies approximately 2.3% of the visual canvas, causing standard global velocity-prediction losses to be overwhelmingly dominated by scene background and general motion. JoVA overcomes this limitation through an adaptive localized supervision strategy. During preprocessing, facial landmark detectors locate mouth bounding boxes across raw frames, which are downsampled via spatial scaling and sliding-window temporal pooling to match the VAE latent dimensions into a precise binary mouth mask \(\mathcal{M}\). The comprehensive training loss is formulated as: $\(\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{video}} + \mathcal{L}_{\text{audio}} + \lambda \mathcal{L}_{\text{mouth}}\)$ The localized loss \(\mathcal{L}_{\text{mouth}}\) penalizes flow velocity prediction errors strictly inside the masked mouth latents. Crucially, the weighting parameter \(\lambda\) is dynamically conditioned on sample semantics: for human talking scenarios, \(\lambda = 5.0\) provides strong targeted guidance to lock lip movements to speech phonemes; for natural environments and ambient soundscapes, \(\lambda\) is set to 0. This selective enforcement achieves superior lip-sync fidelity without compromising visual naturalness on non-speech content.

4. Large-Scale 3M Corpus Construction and Automated Editing Synthesis Engine

To support end-to-end learning across diverse multimodal generative and editing scenarios, the authors constructed a high-grade dataset comprising ~3 million video-audio-text pairs. The corpus spans natural environmental scenes (~650K samples, 21.6%), synchronized human speech dialogues (~1M samples, 33.2%), and paired video-audio editing data (~1.36M samples, 45.2%). For the editing corpus, an automated synthesis pipeline was developed: input video keyframes are modified via GPT-4o prompts and FLUX.1-Kontext-dev, paired with Canny edge structures to guide Wan2.2-Fun-A14B-Control in generating visually consistent edited videos. Concurrently, Qwen3 scripts and Higgs-Audio voice synthesis produce corresponding acoustic tracks, which are aligned via talking-head synchronization models to supply rich, paired source-target editing data.

Loss & Training

The framework is trained under conditional flow matching objectives across both modalities: $\(\mathcal{L}_{\text{video}} = \mathbb{E}_{t, \mathbf{v}_0, \mathbf{v}_1, \mathbf{C}} \left\| v_\theta(t, \mathbf{v}_t, \mathbf{C}) - u(\mathbf{v}_t | \mathbf{v}_0, \mathbf{v}_1) \right\|^2\)$ $\(\mathcal{L}_{\text{audio}} = \mathbb{E}_{t, \mathbf{a}_0, \mathbf{a}_1, \mathbf{C}} \left\| v_\theta(t, \mathbf{a}_t, \mathbf{C}) - u(\mathbf{a}_t | \mathbf{a}_0, \mathbf{a}_1) \right\|^2\)$ $\(\mathcal{L}_{\text{mouth}} = \mathbb{E}_{t, \mathbf{v}_0, \mathbf{v}_1, \mathbf{C}} \left\| \left( v_\theta(t, \mathbf{v}_t, \mathbf{C}) - u(\mathbf{v}_t | \mathbf{v}_0, \mathbf{v}_1) \right) \odot \mathcal{M} \right\|^2\)$ where \(\mathbf{v}_t = t \mathbf{v}_1 + (1-t) \mathbf{v}_0\) and \(u(\mathbf{v}_t | \mathbf{v}_0, \mathbf{v}_1) = \mathbf{v}_1 - \mathbf{v}_0\). Training is structured into two sequential stages: stage one warms up the audio branch on text-to-audio and text-to-speech datasets with a batch size of 256 and learning rate \(1 \times 10^{-4}\) for 80K iterations; stage two unifies both branches on the full 3M multimodal dataset with a batch size of 32 and learning rate \(3 \times 10^{-5}\) for 100K steps. A Classifier-Free Guidance (CFG) scale of 10.0 is used during sampling.

Key Experimental Results

Main Results

The model is rigorously evaluated on JoVABench-Gen, JoVABench-Edit, and the out-of-domain Verse-Bench suite, benchmarked against both dedicated audio-driven synthesis models (Wan-S2V, Fantasy-Talking) and leading joint video-audio generators (UniVerse-1, JavisDiT, Ovi, LTX-2, UniAVGen).

The following table summarizes performance on JoVABench-Gen:

Paradigm	Method	LSE-C↑	WER↓	FD↓	KL↓	CE↑	PQ↑	MS↑	AS↑	ID↑
Audio-driven	Wan-S2V	6.43	-	-	-	-	-	0.82	0.44	0.72
Joint Generation	UniVerse-1	1.62	0.37	1.04	0.83	3.68	4.39	0.43	0.42	0.82
Joint Generation	JavisDiT	1.04	1.08	1.15	0.64	3.36	4.76	0.20	0.44	0.30
Joint Generation	Ovi	6.41	0.23	0.75	0.66	5.00	5.77	0.94	0.41	0.75
Joint Generation	LTX-2	6.21	0.15	0.80	0.64	5.06	5.64	0.92	0.44	0.74
Joint Generation	UniAVGen	5.87	0.17	0.70	0.62	5.32	6.30	0.90	0.44	0.75
Ours	JoVA	6.70	0.19	0.67	0.64	5.40	6.49	0.97	0.48	0.78

On JoVABench-Edit, where no unified baseline exists, JoVA is evaluated against cascaded two-stage pipelines combining state-of-the-art video editing models with specialized lip-sync systems supplied with ground-truth audio:

Cascaded Baseline Pipeline	LSE-C↑	LSE-D↓	FC↑	IB-TV↑	IS↑	AS↑	MS↑
Ditto + Diff2Lip	2.44	5.30	0.98	0.23	2.13	0.34	0.35
OmniVideo + Diff2Lip	4.07	5.50	0.98	0.19	3.03	0.35	0.71
Ditto + Wav2Lip	5.85	2.83	0.98	0.23	2.33	0.42	0.46
OmniVideo + Wav2Lip	4.86	1.75	0.97	0.18	2.27	0.37	0.63
JoVA (Ours)	5.88	1.66	0.98	0.21	3.21	0.47	0.66

Ablation Study

Sensitivity analysis on the mouth-area loss weighting parameter \(\lambda\) on JoVABench-Gen:

Mouth-Area Weight \(\lambda\)	LSE-C↑	WER↓	FD↓	KL↓	PQ↑	AS↑
\(\lambda = 0.0\)	1.39	0.17	0.67	0.63	6.44	0.48
\(\lambda = 2.0\)	6.53	0.18	0.67	0.62	6.47	0.46
\(\lambda = 5.0\) (Default)	6.70	0.19	0.67	0.64	6.49	0.48
\(\lambda = 8.0\)	6.65	0.18	0.68	0.63	6.40	0.47

Ablation across different audio-video cross-modal interaction designs (under identical parameter budget, training data, and step count):

Multimodal Interaction Strategy	LSE-C↑	WER↓	PQ↑	AS↑	ID↑
Sequential Cross-Attn (Ovi-style)	6.51	0.24	6.25	0.48	0.75
Parallel Cross-Attn (w/ Linear Adaptation)	1.59	0.29	6.31	0.46	0.74
Parallel Cross-Attn (w/o Linear Adaptation)	6.57	0.28	6.14	0.47	0.76
Joint Self-Attention (Ours)	6.70	0.19	6.49	0.48	0.78

Key Findings

• Mouth-area loss is the make-or-break factor for lip-sync: With \(\lambda=0.0\), joint self-attention alone yields an LSE-C of only 1.39, indicating complete failure to synchronize phonemes to mouth shapes. Activating the localized loss at \(\lambda=2.0\) causes LSE-C to surge to 6.53, peaking at 6.70 when \(\lambda=5.0\). This demonstrates that global flow matching cannot effectively supervise regions occupying ~2.3% of the image without targeted reweighting.
• Joint self-attention outperforms explicit cross-attention: Native joint self-attention outperforms sequential and parallel cross-attention variants across LSE-C (6.70 vs 6.51), WER (0.19 vs 0.24), and audio quality PQ (6.49 vs 6.25). Adding explicit linear projection layers to cross-attention collapses LSE-C to 1.59, proving that unconstrained parameter projection disrupts fine-grained cross-modal alignment.
• Unified editing eliminates cascading errors: In joint video-audio editing, JoVA outperforms two-stage pipelines using ground-truth audio, establishing the highest LSE-C (5.88) and lowest LSE-D (1.66) while preventing identity distortion and background degradation.

Highlights & Insights

• Architectural Minimalism in Multimodal Diffusion: Demonstrates that multimodal audio-video alignment does not require complex multi-branch routing networks; duplicating visual weights and sharing native transformer self-attention achieves superior cross-modal coherence.
• Channel-Wise Mask Conditioning: Elegant spatial conditioning bypasses the sequence length explosion of video tokens, cleanly unifying image-to-video synthesis and video editing into a single pipeline.
• Semantic-Aware Dynamic Supervision: Selectively enabling localized mouth losses for talking heads while suppressing them on ambient scenes guarantees pinpoint lip alignment without impacting natural environment generation.

Limitations & Future Work

• High-Velocity Extremity Artifacts: The authors note that rapid, fine-grained hand movements occasionally exhibit structural inconsistencies and finger flickering in complex action sequences.
• Text Degradation in Complex Backgrounds: During instruction-guided video editing, scenes featuring dense textual typography (e.g., storefront signs) can experience spelling drift in background regions.
• Limited General-Scene Editing Corpus: Joint audio-video editing currently concentrates on talking-head videos due to the extreme difficulty of gathering paired audio-video editing data for natural environments. Future research will explore scaling synthetic pipelines for ambient audio-video editing.

Related Work & Insights

• vs UniVerse-1 / Ovi / LTX-2: Existing joint frameworks process video and audio through separate streams connected via specialized cross-attention layers. JoVA collapses these into a single shared transformer sequence, avoiding prior distortion and achieving superior cross-modal synchronization.
• vs Cascaded Video Editing Pipelines (OmniVideo / Ditto + Wav2Lip): Cascaded systems suffer from severe identity drift, boundary seams, and require clean external driving audio. JoVA unifies video and audio denoising in a single pass, delivering end-to-end multimodal editing with higher instruction fidelity.

Rating

• Novelty: ⭐⭐⭐⭐☆ Elegant unification of joint generation and editing via native joint self-attention and channel-wise mask conditioning.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Curated 3M multi-domain dataset, established two dedicated benchmarks, and conducted exhaustive comparisons and architectural ablations.
• Writing Quality: ⭐⭐⭐⭐⭐ Well-organized narrative, crystal-clear motivation, and meticulous tabular validation.
• Value: ⭐⭐⭐⭐⭐ Provides a foundational blueprint for next-generation unified multimodal audio-visual creation systems.