Unison: Harmonizing Motion, Speech, and Sound for Human-Centric Audio-Video Generation

Conference: ECCV2026
Paper: Official ECCV Page
Area: Video Generation
Keywords: joint audio-video generation, speech-sound disentanglement, semantic gating, cross-modal synchronization, flow matching

TL;DR

Unison generates speech and sound effects through separate streams, coordinates their acoustic information with semantic gating, and learns mutual audio-video guidance through progressive cross-modal forcing, improving audio quality and overall temporal alignment on a curated held-out test set without consistently surpassing LTX-2 in lip synchronization or visual aesthetics.

Background & Motivation

Human-centric audio-video generation requires more than producing images and adding a soundtrack. When someone sings while playing a guitar, their lips should follow the lyrics, their fingers should correspond to plucking sounds, and background rain should remain audible beneath the singing. Joint models such as Ovi and UniAVGen already exchange video and audio features, while work such as Harmony refines this interaction to the frame level. Nevertheless, speech and environmental sounds often compete within the same acoustic representation. Speech has structured phonetic patterns and is especially prominent in human-centric scenes, encouraging models to prioritize intelligible words while weakening short impacts, accompaniment, and background sounds.

Adding audio-video cross-attention alone does not resolve a second problem: when both modalities are trained at the same noise level, neither may supply reliable timing cues. The network can access the other modality without necessarily having to use it to associate lips and fingers with acoustic onsets. Two distinct failures therefore matter: competition between speech and sound effects within the soundtrack, and temporal correspondence between the soundtrack and visible motion. Treating both as a single synchronization problem obscures their different causes.

Unison retains a pretrained video model and concentrates its changes on audio representations and joint training. Speech and sound effects receive separate generative trajectories and supervision, followed by semantically controlled information exchange. Training progressively introduces asymmetric audio-video corruption, allowing the cleaner modality to become a useful condition for the other. Core Idea: give speech and sound effects distinct modeling responsibilities, then use controlled acoustic interaction and asymmetric-noise training to learn acoustic layering and motion-audio correspondence separately.

Method

Overall Architecture

The main inputs are a caption describing the scene, actions, and environmental sounds, and a transcription specifying spoken words or lyrics. The paper evaluates both image-conditioned TI2AV and T2AV without an image condition. The video branch uses Wan2.2-5B with 29 layers; the audio branch builds on MMAudio, incorporates Zipformer for speech, and has 23 layers. The outputs are human-centric video and coordinated speech and sound effects, not skeletal motion or action labels.

The branches exchange information through frame-level bidirectional cross-attention. Local alignment uses a 3-frame window with stride 1, retaining only the middle frame's representation during restoration to preserve nearby temporal context. Within the audio branch, speech and sound effects follow two separate trajectories. Their interaction must be distinguished from audio-video interaction: the former addresses acoustic interference, whereas the latter supplies conditions linking motion and sound.

Solid arrows below represent the main generation-time information flow; dotted arrows indicate training supervision or objectives. Progressive cross-modal forcing is a training mechanism, not a synchronization repair applied after video generation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Scene caption and transcription<br/>Audio noise"] --> B["Dual-Stream Decomposition<br/>and Supervision"]
    B --> C["Bidirectional Audio<br/>Cross-Attention"]
    C --> D["Semantic-Conditioned Gating"]
    V["Video noise and conditions<br/>Wan2.2 video branch"] <--> E["Frame-level bidirectional<br/>audio-video interaction"]
    D <--> E
    E --> O["Decode video and audio"]
    T["Training audio and video<br/>Mel-Roformer separation"] -.-> B
    E -.->|Branch prediction losses| F["Progressive Cross-Modal Forcing"]
    T -.->|Targets and noise scheduling| F

Key Designs

1. Dual-Stream Decomposition and Supervision: environmental sound is not merely a speech residual

The foundation of the Semantic-Guided Harmonization Strategy (SGHS) is to maintain speech and sound-effect latents as two temporally aligned sequences. Mel-Roformer separates the training audio, which is then encoded into distinct target latents; the two streams receive independent flow-matching supervision. Producing the correct words and synthesizing the accompanying guitar and rain thus have separate learning targets, rather than relying on one mixed-audio loss to assign suitable importance to both. These targets come from source-separated recordings, not perfectly clean manually recorded sources, so separation quality also constrains supervision quality.

Decoupling does not mean complete isolation. If speech and sound effects were generated independently throughout, they might each be clear without sounding as though they belonged to the same scene. Merging them too early, however, could reintroduce speech dominance. The following modules therefore preserve stream identity while exchanging information inside the network. The caption supplies scene-level acoustic cues, while the transcription specifies linguistic content; the semantic inputs also serve distinct purposes.

2. Bidirectional Audio Cross-Attention: share context without erasing source identity

Bidirectional Audio Cross-Attention (Bi-ACA) allows speech features to query sound-effect features and vice versa. The resulting sequences are concatenated along the sequence dimension, processed globally through a shared Transformer, and split back into separate trajectories at the block exit. This repeated interaction, merging, and splitting lets singing access accompaniment context and lets accompaniment use rhythmic cues from singing, without repeatedly identifying sources inside an already mixed audio representation.

Both sequences reuse the same temporal indices for Rotary Positional Embeddings (RoPE), so corresponding positions refer to the same time. A modality-specific learnable bias prevents shared attention from confusing the same timestamp with the same type of sound. These operations take place inside the audio branch and complement external frame-level audio-video cross-attention: even with identical visual conditioning, speech and sound effects can first supply acoustic priors to one another before establishing their relationships with video.

3. Semantic-Conditioned Gating: regulate cross-stream injection, not output volume

Unrestricted cross-attention can inject excessive environmental information into the speech trajectory and damage phonetic clarity. Semantic-Conditioned Gating (SCG) average-pools transcription and caption features separately, then uses an MLP and sigmoid to predict two gates. The following relation is readable in the source text and shows how the gates depend on semantic conditions:

\[ [g_{sp}, g_{sfx}] = \sigma(\mathrm{MLP}([c_s; c_a])). \]

Here, \(c_s\) and \(c_a\) are pooled transcription and caption features; \(g_{sp}\) and \(g_{sfx}\) control cross-attention updates entering the speech and sound-effect trajectories, respectively. The gates multiply cross-stream residual updates, not the final waveform amplitudes. Original stream features remain available, and the gates need not sum to 1 as mixing proportions would. A smaller speech gate therefore means less interference from sound-effect features, not necessarily quieter speech.

The intended behavior is content-dependent sharing: dense narration should preserve linguistic clarity, whereas a musical performance may benefit from richer cross-stream exchange. Figure 8 shows gate trends across network depth, denoising progress, and scene categories, but does not provide every implementation detail needed to reproduce the gating behavior. It supports a content-adaptive interpretation without establishing a one-to-one causal mapping between a particular gate value and a perceptual outcome.

4. Progressive Cross-Modal Forcing: turn the more reliable modality into a condition

Video and audio receive separately sampled training timesteps, allowing relatively clean audio to accompany noisy video or the reverse. Bidirectional cross-attention still performs information exchange; forcing does not introduce a separate teacher model. Instead, it increases the prediction-loss weight of the noisier branch, encouraging that branch to use the more reliable structure in the other modality. Audio-leading samples emphasize video prediction, while video-leading samples emphasize audio prediction. These roles can change by sample rather than permanently restricting guidance to audio-driven video.

Fully independent sampling from the start can produce large noise disparities and frequent changes in the leading modality, exposing the system to unstable conditions before cross-modal relationships have formed. The authors therefore begin with synchronous warmup, progressively increase the probability of independent sampling while limiting the timestep gap to 0.25, and finally move to full independence. Section 3.3 describes the last phase as using unconstrained decoupled updates. Direction-aware loss reweighting starts in the second phase, with guidance strength 0.5. Progressive here refers to the training curriculum, not to audio gradually catching up with motion during playback.

An implementation ambiguity deserves attention: the flow-matching interpolation in Section 3.1 moves from noise toward data, whereas Section 3.3 adopts the convention that smaller timesteps correspond to cleaner states. The mapping between these conventions is not fully explained. Operators are also missing from cached Equations (8) and (9), so a reconstructed weighted objective is not presented as the authors' exact equation. What is clear is that the three branches use flow-matching prediction losses, with video loss and the two audio losses combined according to the guidance direction. Reproduction still requires checking timestep mapping, the independent-sampling probability curve, and constraints in the final phase.

A Worked Example

Consider someone playing a guitar in the rain while singing specified lyrics, as an explanatory example rather than a new experiment. The caption describes the person's actions, guitar, and rainfall; the transcription supplies the lyrics. Two audio trajectories model singing and the accompaniment plus rain. Bi-ACA exchanges acoustic context, and SCG controls how much information is injected so that rendering rain does not undermine lyrical clarity. Local audio-video interaction links lips and fingers to these acoustic cues.

For a training sample with cleaner audio, singing boundaries and plucking sounds provide cues for video prediction. With cleaner video, mouth and hand motion instead support audio prediction. At inference time the model samples jointly; users do not need to supply Mel-Roformer-separated ground truth. The paper also demonstrates A2V and V2A, generating one modality while conditioning on an existing audio or video input. These directions are primarily qualitative demonstrations in the main paper and should not be treated as equivalent to the quantitative main-table evaluation.

Loss & Training

There are two major training stages and three curriculum phases within joint training; these are different levels of scheduling. Audio pretraining uses 4 H100 GPUs, batch size 96, and learning rate 0.0001, with 1,000 steps of linear warmup and decay by a factor of 0.1 at steps 240,000 and 270,000. Joint training uses 16 H100 GPUs, batch size 32, learning rate 0.00002, bf16, and ZeRO-2. Only the audio branch and fusion modules, including bidirectional cross-attention and layer normalization, are optimized; the video backbone remains frozen.

Synchronous warmup, incremental decoupling, and full independence occupy proportions 0.3, 0.4, and 0.3 of joint training. The audiovisual corpus contains approximately 2 million clips totaling over 3,000 hours, while the audio-only corpus contains 50 million segments exceeding 130,000 hours, including public datasets and internal speech data. Face detection and SyncNet filtering remove lip-audio mismatches and off-screen voice-overs. Quality improvements therefore also depend on data filtering and cannot be attributed solely to architectural modules.

Inference uses a 50-step flow-matching sampler with classifier-free guidance scale 6.0 and produces 25 FPS video. The main paper does not report a complete combination of output resolution, duration, total training steps, and inference latency. A 5B video backbone therefore cannot be directly translated into total system size or an end-to-end efficiency advantage.

Key Experimental Results

Main Results

The authors use a curated set of 1,000 held-out samples with Gemini-generated annotations, evaluating text-to-audio-video (T2AV) and text-and-image-to-audio-video (TI2AV). The main paper does not specify the sample count for each task or provide enough information to verify equal compute budgets. The table below supports within-task comparisons only and preserves the original numerical scales from Table 1.

PQ is the Audiobox-Aesthetics perceptual-quality score, with higher values preferred. WER is word error rate obtained by separating vocals and transcribing them with Whisper-large-v3, with lower values preferred. AV is ImageBind audio-video semantic similarity; LSE-C is SyncNet lip-synchronization confidence. DS measures audio-video temporal offset using Synchformer, with lower values preferred. The main paper does not sufficiently specify DS aggregation and units, so the values are not converted to milliseconds here.

Task	Model	PQ ↑	WER ↓	AV ↑	LSE-C ↑	DS ↓
TI2AV	Ovi	6.25	0.43	0.87	2.81	0.12
TI2AV	UniAVGen	6.18	0.33	0.81	2.89	0.15
TI2AV	MOVA	6.28	0.29	0.88	3.24	0.13
TI2AV	LTX-2	6.30	0.25	0.89	3.45	0.10
TI2AV	Unison	6.34	0.22	0.91	3.30	0.08
T2AV	Ovi	6.08	0.18	0.83	3.47	0.08
T2AV	LTX-2	6.12	0.11	0.81	3.62	0.07
T2AV	Unison	6.17	0.09	0.86	3.55	0.06

Against LTX-2 on TI2AV, Unison improves PQ by 0.04, reduces WER by 0.03, and reduces DS by 0.02, but LSE-C falls from 3.45 to 3.30. The original table also favors LTX-2 on LSE-D, where lower is better (7.62 versus 7.88), and visual aesthetics VA, where higher is better (4.15 versus 4.02). The advantages are therefore concentrated in audio quality, semantic consistency, and overall temporal offset, not every synchronization or visual-quality metric.

Ablation Study

The following table combines component ablations from Table 2 with training-strategy ablations from Table 3. They are separate experiments, not a sequential component-addition procedure. VA is the LAION aesthetic predictor score, with higher values preferred; the full model uses progressive forcing (PF). The main paper does not separately detail the composition of the ablation evaluation subset.

Experiment	Config	VA ↑	PQ ↑	LSE-C ↑	DS ↓
Components, Table 2	Without SGHS	3.99	6.12	3.08	0.15
Components, Table 2	Without Bi-ACA	4.00	6.20	3.18	0.11
Components, Table 2	Without SCG	4.01	6.21	3.22	0.10
Components, Table 2	Without cross-modal forcing	3.91	6.24	3.02	0.19
Strategy, Table 3	Always synchronous: SyncOnly	3.90	6.10	3.12	0.17
Strategy, Table 3	Always independent: IndepOnly	3.95	6.18	3.28	0.14
Shared reference	Full model PF	4.02	6.34	3.30	0.08

Removing the whole SGHS reduces PQ by 0.22, the largest audio-side decrease. Removing cross-modal forcing increases DS from 0.08 to 0.19, the strongest synchronization degradation among the component ablations. SGHS encompasses multiple audio-side designs, so its removal and the submodule removals should not be interpreted as four disjoint contributions.

Key Findings

• Switching from always-independent sampling to the progressive curriculum reduces DS from 0.14 to 0.08. The benefit therefore concerns how asymmetric noise is introduced, not merely whether the two modalities receive different noise levels.
• Removing audio modules leaves VA between 3.99 and 4.01, whereas removing forcing reduces it to 3.91. This supports distinct roles for acoustic modeling and audio-video training, but does not directly measure changes inside the frozen video backbone.
• The user study includes 40 samples and 25 participants, giving 1,000 sample-participant evaluations. Lower mean ranks are better. The key results from Table 4 follow; Gemini is an additional model evaluation, not a human-voting column.

Method	Lip synchronization ↓	Speech-sound harmony ↓	Motion-audio alignment ↓	Gemini score ↓
LTX-2	1.74	1.95	1.89	2.05
Unison	1.86	1.55	1.92	1.68

The user-study prose claims that Unison has the best motion-audio alignment, but Table 4 favors LTX-2 at 1.89 over Unison at 1.92. This note preserves the table values rather than forcing agreement with the prose, and does not interpret small differences without confidence intervals as statistically significant advantages.

Highlights & Insights

• Assign source responsibilities before allowing interaction: separation-based supervision gives environmental sounds their own learning targets, while Bi-ACA and SCG prevent independent generation from becoming an incoherent mixture. This intervenes more directly in representation structure than adding a coordination loss only to the final mixed soundtrack.
• Training distributions can shape cross-modal dependence: the cleaner modality supplies more reliable context, and the noisier branch receives greater learning emphasis. The transferable idea is to design differences in conditioning reliability and a curriculum, not to treat cross-attention itself as the new contribution.
• Synchronization needs multiple measurements: lip confidence, overall temporal offset, and semantic similarity can rank models differently. These results show particularly clearly that better acoustic layering does not automatically imply the best lip-audio synchronization.

Limitations & Future Work

• Boundaries disclosed by the authors: there is no standalone limitations section. The authors disclose internal speech training data and promise to release code and models after acceptance, but the supplied paper gives no actual release URL; availability cannot be assumed.
• This note's interpretation: incomplete reproducibility details: noise-timestep conventions, whether the final phase retains a timestep-gap bound, gating details, and total training budget require clarification. The 5B figure refers only to the video backbone and does not establish that the complete system is nearly four times smaller than LTX-2.
• This note's interpretation: limited evaluation coverage: the test set is author-curated and Gemini-annotated, without sufficient task-split or language-distribution details, or results for long videos, multiple people, and complex occlusion. A2V and V2A demonstrations also lack a corresponding full quantitative table.
• This note's interpretation: supervision and metrics share dependencies: both training-source separation and WER evaluation depend on Mel-Roformer, while face and SyncNet filtering favors visible on-screen speakers. Further work should isolate separation errors and assess off-screen speech and overlapping speakers, while reporting uncertainty in human evaluations.

Related Work & Insights

• vs MMAudio: Unison inherits its audio-modeling foundation but adds Zipformer-based speech capability and separately supervised speech and sound-effect streams. It is not simply a video-to-audio model appended after rendering a finished video.
• vs Harmony / UniAVGen: frame-level audio-video cross-attention is an existing foundation. Unison additionally changes acoustic representations and cross-modal noise training; gains should be interpreted through ablations rather than attributed entirely to the fusion architecture.
• vs Diffusion Forcing: the original principle assigns different noise levels to sequence elements. Unison extends this asymmetry across audio and video and adds direction-aware reweighting and a progressive curriculum.
• vs LTX-2: Unison performs better here on audio quality, WER, and DS, while LTX-2 leads on visual aesthetics and some lip-synchronization metrics. A useful next question is how to improve acoustic layering and fine-grained lip alignment together, rather than optimizing only an aggregate score.

Rating

• Novelty: 4/5. Combines source separation, semantic gating, and progressive cross-modal forcing into a targeted human-centric audio-video framework.
• Experimental Thoroughness: 3/5. Includes main comparisons, component and curriculum ablations, and a user study, but test-set and resource disclosures are limited and some claims conflict with tables.
• Writing Quality: 3/5. Clearly distinguishes two types of mismatch, but timestep conventions and user-study conclusions need clarification.
• Value: 4/5. Offers transferable ideas for coordinating speech, environmental sounds, and motion, although engineering efficiency and complex-scene generalization remain uncertain from the main paper.