MAVIN: Multi-Shot Audio-Visual Generation with Customized Narrative Control¶

Conference: ECCV 2026
Paper: CVF Open Access
Area: Video Generation
Keywords: Audio-Visual Generation, Multi-Shot Video Generation, Boundary-Aware Attention, ID-Aware Propagation, Diffusion Transformer

TL;DR¶

To tackle temporal misalignment, limited controllability over vocal timbre and appearance, and incomplete audio scripting in cinematic filmmaking, MAVIN introduces a multi-agent scripting pipeline, boundary-aware attention, and ID-aware propagation, enabling the first end-to-end multi-shot audio-visual generation framework with precise transition boundaries, dialogue timestamps, and multi-subject identity consistency.

Background & Motivation¶

Recent diffusion transformer (DiT) architectures have significantly advanced the synthesis of high-fidelity, single-shot video clips. In the pursuit of bringing generative models closer to professional cinematic workflows, the community has increasingly explored long-horizon narrative planning, multi-shot storyboard synthesis, and audio-visual synchronization. However, contemporary clip-level audio-visual models are fundamentally constrained by a single global prompt condition, making them ill-suited for complex cinematic narratives that demand multiple distinct camera shots, seamless transitions, structured character dialogue, and synchronized sound effects.

Existing efforts to bridge this gap struggle with three core tensions. First, in terms of temporal alignment, standard cross-attention allows all text tokens to globally attend across the entire latent sequence, causing severe cross-shot semantic leakage and rendering it incapable of enforcing exact start and end timestamps for dialogue and visual events. Second, in terms of controllability, while recent personalization methods preserve visual character appearances across shots, joint audio conditioning—such as binding a specific vocal timbre to a given character—remains largely untouched, causing identity blending and vocal crosstalk in multi-character scenes. Third, existing narrative scripting agents focus almost exclusively on visual descriptions, lacking timestamped speech intervals and acoustic cues required for joint audio-visual rendering.

MAVIN addresses these limitations by decoupling narrative semantics across temporal boundaries and isolating multi-subject identity representations. The core idea is to structure free-form prompts into hierarchical captions (global storyline, shot-level events, and character appearance/timbre), enforce strict temporal routing via boundary-aware attention over asymmetric video-audio latent partitions, and decouple reference timbre and appearance via ID-aware propagation to achieve end-to-end synchronized multi-shot audio-visual generation.

Method¶

Overall Architecture¶

Built upon the dual-tower diffusion transformer backbone of OVI, MAVIN jointly predicts the velocity fields for video and audio latents via flow matching. Given free-form user prompts alongside optional reference images and audio clips for multiple characters, the system operates across three coordinated stages: (1) a lightweight multi-agent scripting pipeline converts unstructured inputs into hierarchical captions; (2) continuous video frames and event-driven audio signals are encoded into asymmetric latent spaces via 3D video and audio VAEs; (3) stacked DiT blocks process the joint latents using boundary-aware text-cross-attention and identity-aware masked propagation, outputting temporally aligned, multi-shot audio-visual sequences via ODE solving and VAE decoding.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
    A["User Prompt + Optional References"] --> B["Multi-Agent Scripting Pipeline<br/>Structure Parsing → ID Aligning → Narrative Refining"]
    B --> C["Hierarchical Captions<br/>Global Context + Shot Intervals + Character Appearance & Timbre"]
    C --> D["Asymmetric Latent Partition<br/>Continuous Video Shots + Sparse Audio Dialogue Intervals"]
    D --> E["Boundary-Aware Attention<br/>Dynamic Routing Matrix M Restricting Attention Windows"]
    E --> F["ID-Aware Propagation<br/>Pretrained Timbre Encoder + ID Embeddings + Attention Masks"]
    F --> G["Joint Velocity Prediction & VAE Decoding<br/>Flow Matching ODE → Synchronized Multi-Shot Video & Audio"]

Key Designs¶

1. Multi-Agent Scripting Pipeline: Structuring Free-Form Prompts into Hierarchical Captions To bridge the gap between abstract user prompts and the rigorous temporal conditioning required by diffusion models, MAVIN employs three sequential LLM agents. The Structure Parser analyzes the input prompt against predefined templates to extract global attributes, decompose shot transition boundaries \(\{\mathcal{T}^\mathrm{s}_i\}_{i=1}^S\), identify discrete characters \(\mathcal{R}\), and assign dialogue intervals \(\{\mathcal{T}^\mathrm{r}_j\}_{j\in\mathcal{R}}\). Next, the Identity Aligner grounds each character with the corresponding user-provided reference images and audio. Finally, the Narrative Refiner enriches visual action events, camera language, and exact spoken lines. This produces a unified hierarchical caption \(\mathcal{C} = \{\mathcal{C}^\mathrm{global}, \{\mathcal{C}^\mathrm{shot}_i\}_{i=1}^S, \{\mathcal{C}^\mathrm{role}_j\}_{j\in\mathcal{R}}\}\) that decouples macro-storyline, micro-shot dynamics, and entity-level attributes.

2. Boundary-Aware Attention: Asymmetric Latent Partitioning and Token Routing Standard cross-attention distributes semantic tokens across all latent frames, inevitably blurring shot cuts and misaligning speech timestamps. MAVIN addresses this by observing that video is temporally continuous while speech is event-sparse: video latents \(\mathbf{Z}^\mathrm{v}\) are partitioned into shot intervals \(\mathcal{T}^\mathrm{s}_i\), whereas audio latents \(\mathbf{Z}^\mathrm{a}\) are partitioned into active dialogue intervals \(\mathcal{T}^\mathrm{r}_j\). Inside the DiT text-cross-attention layer, standard attention is reformulated with a dynamic binary visibility routing matrix \(\mathbf{M} \in \{0, 1\}^{N \times L}\):

\[\mathbf{Z}^r = \mathrm{softmax}\left(\frac{\mathbf{Q}\mathbf{K}_{\mathcal{C}}^\top}{\sqrt{d}} \odot \mathbf{M}\right)\mathbf{V}_{\mathcal{C}}\]

For a video latent token \(k\) at timestamp \(\tau(k)\), global tokens are always visible; shot tokens \(\mathcal{C}^\mathrm{shot}_i\) are visible if and only if \(\tau(k) \in \mathcal{T}^\mathrm{s}_i\); and role visual tokens \(\mathcal{C}^\mathrm{role}_j\) are visible only when character \(j\) is present in shot \(i\) during that interval. Similarly, audio tokens only attend to character audio descriptions \(\mathbf{d}^\mathrm{a}_j\) when \(\tau(k) \in \mathcal{T}^\mathrm{r}_j\). This completely eliminates semantic bleed-over across cut transitions and forces dialogue to begin and end precisely within designated timeframes.

3. ID-Aware Propagation: Decoupled Timbre Embeddings and Layer-Specific Identity Masks Personalizing multiple characters across shots requires injecting reference appearances and voices without causing identity crosstalk. Directly injecting raw reference audio conflates linguistic content, prosody, and emotion with timbre. MAVIN overcomes this by processing reference audio through a Pretrained Timbre Encoder (PTE) to extract a clean timbre embedding \(\mathbf{e}_j\), which is projected, broadcast, and modulated alongside VAE audio tokens and learnable role ID embeddings \(\mathbf{p}_j^\mathrm{a}\). Reference images are similarly encoded and modulated with visual ID embeddings \(\mathbf{p}_j^\mathrm{v}\). To prevent reference tokens from corrupting global scene dynamics, MAVIN enforces an Identity-Aware Mask across four key layers: video self-attention restricts video tokens to only interact with visual anchors present in the current shot; audio self-attention confines audio tokens to active speaker anchors; text-attention binds role prompts strictly to their corresponding anchors; and cross-attention permits interactions only between the visual and vocal anchors of the exact same character (\(\hat{\mathbf{Z}}^\mathrm{a,ref}_j \leftrightarrow \hat{\mathbf{Z}}^\mathrm{v,ref}_j\)).

Loss & Training¶

The framework is trained using a three-stage progressive alignment schedule to prevent gradient conflicts among diverse conditions: Stage 1 optimizes text-driven multi-shot video and audio generation; Stage 2 introduces single-modality identity anchors to tune visual and audio reference conditioning paths; Stage 3 executes full joint training across all modalities and conditions. The model is optimized using flow matching, regressing the velocity fields of video and audio with a mean squared error objective against independent Gaussian priors.

Key Experimental Results¶

Main Results¶

Quantitative evaluations are conducted on the 1,000-sample MAVINSet benchmark (unseen multi-shot cinematic scenarios, evaluated without reference images/audio for a strict and fair baseline comparison). MAVIN is compared against both cascaded multi-shot frameworks (T2V models coupled with Hunyuan-Foley for V2A) and state-of-the-art joint T2AV models (JavisDiT, UniVerse-1, OVI, LTX-2).

Category	Method	Video FVD ↓	Audio FAD ↓	Lip Sync Sync ↑	Speech Alignment TAMS ↑	Shot Transition STA ↑	Character Consistency CISC ↑
Cascaded (T2V + V2A)	VideoGen-of-Thought	418.7	13.2	2.278	-	-	0.6195
Cascaded (T2V + V2A)	MovieAgent	395.3	15.9	1.913	-	-	0.4872
Cascaded (T2V + V2A)	EchoShot	276.3	12.3	2.095	-	0.6002	0.6104
Cascaded (T2V + V2A)	CineTrans	254.2	11.9	1.892	-	0.5820	0.6073
Joint (T2AV)	JavisDiT	512.6	21.3	3.734	0.1012	-	-
Joint (T2AV)	UniVerse-1	356.9	11.2	3.851	0.2892	-	-
Joint (T2AV)	OVI (Backbone)	318.4	8.3	4.231	0.4432	0.4992	0.6121
Joint (T2AV)	LTX-2	289.7	8.6	4.365	0.5343	0.5733	0.6015
Ours	MAVIN	231.6	6.8	6.032	0.8104	0.9897	0.6319

Note: For cascaded models, downstream V2A models cannot synthesize intelligible speech due to missing fine-grained lip motion in T2V generation, making WER and TAMS inapplicable (-).

Ablation Study¶

Ablations are performed under the customized generation setting with explicit visual and acoustic references to quantify the necessity of each component.

Configuration	Video FVD ↓	Audio FAD ↓	Lip Sync Sync ↑	Speech Alignment TAMS ↑	Shot Transition STA ↑	Character Consistency CISC ↑	Note
Full Model (MAVIN)	241.9	6.9	6.057	0.793	0.9886	0.6397	Complete framework
w/o BTR (Boundary Routing)	268.4	7.6	5.830	0.582	0.5580	0.5520	Temporal boundaries leak; severe STA/TAMS degradation
w/o IM (Identity Mask)	256.9	7.3	5.970	0.787	0.8010	0.5710	Cross-character crosstalk degrades character identity
w/o PTE (Timbre Encoder)	242.5	7.0	6.010	0.792	0.9460	0.6240	Raw audio reference entangles prosody/semantics
w/o MAP (Scripting Pipeline)	261.1	7.5	5.910	0.569	0.6230	0.5900	Free-form prompts fail to provide decoupled boundaries

Key Findings¶

Crucial Role of Boundary-Aware Routing: Removing BTR causes the Shot Transition Accuracy (STA) to collapse from 0.9886 to 0.5580 and the Time Alignment Metric for Speech (TAMS) to drop from 0.793 to 0.582, proving that implicit attention fails to maintain temporal hard cuts and speech boundaries.
Preventing Identity Bleeding: The Identity-Aware Mask is essential for multi-subject scenes; removing it causes significant character drift across shots (CISC drops from 0.6397 to 0.5710).
Overwhelming Human Preference: In subjective user studies on Amazon Mechanical Turk, MAVIN achieves 66.4% and 64.6% user preference in Audio Narrative Adherence (ANA) and Video Narrative Adherence (VNA), outperforming all baseline models by an order of magnitude.

Highlights & Insights¶

Asymmetric Latent Boundary Modeling: The design elegantly honors the distinct physical natures of video (continuous visual flow partitioned by cinematic shot boundaries) and dialogue (sparse, event-driven audio partitioned by speaker intervals).
Disentangled Timbre Personalization: Integrating a dedicated pretrained timbre encoder decouples a character's vocal identity from the semantic content of the reference speech, resolving the persistent problem of acoustic contamination during personalization.
Bridging Natural Language and Diffusion Control: The multi-agent scripting pipeline acts as an automated director, translating casual user prompts into structured mathematical boundary constraints for the diffusion backbone.

Limitations & Future Work¶

Duration and Capacity Constraints: The current implementation is capped at 15 seconds (1–6 shots) and up to 3 customized characters, requiring external concatenation for feature-length storytelling.
Sensitivity to Script Timestamps: Downstream generation quality is tightly bound to the quality of the multi-agent script; unrealistic timing generated by the LLM can lead to compressed or unnaturally fast speech synthesis.
Future Directions: Exploring larger foundational DiT backbones to extend multi-shot generation into multi-minute horizons, and developing visual interactive storyboard GUIs for real-time director tweaking.

vs. OVI / LTX-2: End-to-end joint audio-video models achieve solid single-clip synchronization, but lack hard-cut transition mechanisms and timestamped speech control, leading to severe character and background drift across shots (STA < 0.60); MAVIN introduces boundary-aware routing to boost STA to 0.9897.
vs. MovieAgent / EchoShot / CineTrans: Existing multi-shot systems follow a cascaded pipeline (T2V followed by V2A); because the video generation stage is audio-unaware, lip motions are inconsistent and downstream audio models fail to produce intelligible speech; MAVIN performs joint diffusion flow matching, achieving state-of-the-art lip sync (SyncNet score of 6.032).

Rating¶

Novelty: ⭐⭐⭐⭐⭐ [Pioneering framework resolving both temporal boundary alignment and multi-subject audio-visual customization in multi-shot generation]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive validation on the curated 800K MAVINSet and 1K verified benchmark across 13 objective metrics and AMT user studies]
Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, mathematically rigorous formulation of attention routing, and well-structured visualizations]
Value: ⭐⭐⭐⭐⭐ [Highly impactful for the generative filmmaking community, bridging the gap between text-to-video toys and professional directorial tools]