CoSyncDiT: Cognitive Synchronous Diffusion Transformer for Movie Dubbing

Conference: ECCV2026
Paper: ECCV Paper
Full Text: Conference PDF
Code: https://github.com/GalaxyCong/CoSyncDiT
Area: Audio & Speech
Keywords: movie dubbing, visual voice cloning, flow matching, implicit alignment, semantic regularization

Identity check: manifest ID 3458, the cached official URL, and the PDF title agree; the first author is Gaoxiang Cong. There is no subtitle change in this cache. The output task is speech rather than image generation, so the note belongs in Audio & Speech. The cache does not provide a verifiable arXiv ID for this paper.

TL;DR

CoSyncDiT organizes voice-style learning, lip-rhythm calibration, and text alignment sequentially inside a flow-matching Transformer, then regularizes alignment with frame-level contrastive learning and CTC, achieving 47.24% speaker similarity, 7.53% WER, and 0.319 Sync-KL on CinePile-Dub zero-shot Setting 2 (original Table 5).

Background & Motivation

Movie dubbing is not simply guessing speech from a silent video: it receives a silent clip, a prescribed script, and a reference voice, and must generate speech that preserves the reference identity while matching the visible lips. Methods such as StyleDubber and ProDubber predict phoneme durations and expand phonemes into acoustic frames. This helps pronunciation but discretizes continuous speech rhythm into integer duration boundaries. Some contrastive approaches still require external forced aligners to supply those intervals.

AlignDiT instead learns implicit alignment, but unified cross-attention over prefix reference audio, target video, and their text asks the model to determine both speaker identity and the word corresponding to each frame simultaneously. The CTC paths in Figure 2 show that reference-audio regions can activate target text, leaking across the reference/target boundary. Arbitrary reference segments and expressive cinematic prosody make this interference more difficult than in controlled recordings.

The paper therefore reframes visual conditioning as a question of when each condition should enter: establish voice style and its linguistic relationship first, calibrate rhythm with lips next, and retrieve the appropriate linguistic context last. Core Idea: apply acoustic style adapting, fine-grained visual calibrating, and time-aware context aligning in sequence, then supervise timing on contextual outputs and semantics on final hidden states to reduce reference-voice interference with target articulation.

Method

Overall Architecture

The inputs are reference audio, a target silent video, and a dubbing script; the modeled output is a mel-spectrogram of the target speech. The system learns a vector field from Gaussian noise to the target mel representation and generates it through ODE integration at inference time. It neither generates a new video nor constitutes a complete movie-background-sound reconstruction system.

Acoustic and textual features form a unified prior, followed by acoustic style adapting, fine-grained visual calibrating, and time-aware context aligning. Joint Semantic and Alignment Regularization (JSAR) supplies training-only supervision: a pretrained AV-HuBERT audio branch provides frame targets, and the script supplies CTC semantic targets. These three stages describe the architectural conditioning order, not a partition of the 32 ODE evaluations into three time intervals.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    input["Reference audio + script<br/>Noisy state"] --> style["Acoustic Style Adapting"]
    style --> visual["Fine-grained Visual Calibrating"]
    video["Silent video<br/>Lip encoding and upsampling"] --> visual
    visual --> context["Time-aware Context Aligning"]
    text["ConvNeXtV2<br/>Text features"] --> context
    context --> output["Vector field prediction<br/>ODE integration to mel"]
    context -.-> jsar["Joint Semantic and<br/>Alignment Regularization<br/>Training only"]
    teacher["Audio frame targets + script"] -.-> jsar

Key Designs

1. Acoustic Style Adapting: establish acoustic and linguistic priors before visual interference

During training, a binary temporal mask hides the target mel region and retains the acoustic context available as a reference. Text is expanded toward mel length through two paths: padding preserves the full linguistic content, while cross-attention supplies a coarse temporal prior. Both textual sequences and the masked mel form an acoustic-semantic prior, which a unified projection combines with the current noisy speech. Padding does not determine the true duration of each word; subsequent modules still learn the fine correspondence.

Multi-head self-attention then models long-range acoustic dependencies to capture the reference style and its relationship with text. Time-Adaptive Layer Normalization (Time-AdaLN) uses flow time to control scale, shift, and residual gating, followed by an MLP. Raw visual features are deliberately absent at this stage, keeping lip rhythm from overwriting an immature voice-style representation. The large speaker-similarity drop when this stage is removed in Table 6 provides the clearest evidence for its role.

2. Fine-grained Visual Calibrating: introduce lip motion as a controlled residual

A pretrained AV-HuBERT extracts lip-motion features from cropped mouth videos, and cascaded upsampling matches their temporal resolution to the target mel. Upsampling aligns the feature grids without producing forced phoneme-duration labels. Consequently, the visual branch can adjust rhythm at acoustic-frame resolution without first assigning every phoneme to a fixed integer interval.

The upsampled visual features enter the acoustic-style representation through a zero-initialized learnable residual gate. The paper describes this gate as layer-specific and conditioned on the timestep: visual corrections begin near zero, and training learns when and in which channels to use lip motion. The goal is not to discard video but to reserve it for rhythm calibration while preserving established voice style. Removing this branch barely changes SPKSIM but substantially worsens Sync-KL.

3. Time-aware Context Aligning: retrieve the script using calibrated audio-visual states

The lip-calibrated hidden states become cross-attention queries, while ConvNeXtV2 text representations supply both keys and values. Each query therefore carries the current acoustic state and lip rhythm, rather than representing only a noisy acoustic frame, and can retrieve the text appropriate to that moment. Cross-attention is positioned after these representations mature instead of uniformly mixing reference audio and target visual information prematurely across all layers.

Time-AdaLN and time-controlled residual gating also adapt text retrieval to the flow state: a highly noisy state need not use the same alignment strength as a nearly reconstructed speech state. The diagnostic in Figure 2 visualizes frame-level CTC emission probabilities along a non-blank Viterbi path, not an ordinary attention heatmap. It checks whether reference-side activations cross the reference/target boundary.

4. Joint Semantic and Alignment Regularization: constrain timing and pronunciation separately

A flow-matching reconstruction objective alone can produce acoustically plausible vector fields that remain temporally shifted. JSAR applies frame-level InfoNCE to intermediate cross-attention context outputs: these and pretrained AV-HuBERT audio-branch features are L2-normalized along the feature dimension, matching time indices form positive pairs, and other indices form negatives. Supervision comes from audio-frame representations rather than forced phoneme boundaries.

The following normalizes the notation of InfoNCE from the textual definition accompanying original Equation 4; the cached equation has corrupted character order. \(N\) is the flattened frame count, \(\hat z_i\) denotes a context-frame representation, \(f_i\) its audio target, and the temperature is \(\tau=0.07\).

\[ \mathcal{L}_{CL}=-\frac{1}{N}\sum_{i=1}^{N}\log\frac{\exp(\langle\hat z_i,f_i\rangle/\tau)}{\sum_{j=1}^{N}\exp(\langle\hat z_i,f_j\rangle/\tau)}. \]

Semantic regularization attaches CTC to the final flow hidden states, encouraging retention of the script while allowing different frame and text lengths. Its projection contains two temporal downsampling layers with Mish activations, mapping 1,024-dimensional features to a 2,547-dimensional output space. The supervision sites differ: contextual outputs enforce frame alignment, while final hidden states support intelligibility. These are not two names for the same contrastive loss.

A Worked Example

Consider the script “I trust you, every day” from Figure 3 under Setting 2: the reference voice comes from another clip of the same character, while the lips come from the current silent shot. The model first acquires voice style and linguistic priors from the reference rather than copying its pauses. The current shot's lip residual then calibrates speech rhythm, and context alignment retrieves the current script accordingly.

During training, same-time audio targets discourage a global shift in context, while CTC requires the final representation to retain the script. At inference time, target ground-truth audio is not needed as a JSAR teacher; the learned vector field produces the target mel through 32 function evaluations. This illustrates module cooperation, not an additional quantitative example reported by the paper.

Loss & Training

OT-CFM uses the straight path \(x_t=(1-t)x_0+tx_1\), where \(x_0\) is Gaussian noise, \(x_1\) is the target mel, and \(t\) is sampled uniformly from \([0,1]\). Following the MSE definition of original Equation 5, its core objective can be written in normalized notation as:

\[ \mathcal{L}_{fm}=\mathbb{E}_{t,x_0,x_1}\left[\left\|v_\theta(x_t,t,\mathcal{H}_m,\mathcal{X}_{lip},\mathcal{H}_{text})-(x_1-x_0)\right\|_2^2\right]. \]

Training combines flow matching, frame contrastive learning, and CTC, but the cached text does not specify the complete loss weights; equal weighting should not be assumed. Acoustic-semantic CFG uses jointly conditioned, acoustic-only, and unconditional vector fields. Equation 6 has missing characters in the cache and does not supply numerical guidance scales, so their precise roles are not inferred from the corrupted expression here.

The implementation uses 22 layers, a hidden dimension of 1,024, and 16 attention heads. The unified projection maps 712 to 1,024 dimensions; ConvPosition uses kernel size 31 and 16 groups. Mouth crops are 96×96, AV-HuBERT features are 1,024-dimensional, the text encoder contains 4 ConvNeXt V2 blocks of hidden dimension 512, and the final vector field is 100-dimensional.

AdamW uses \(\beta_1=0.9\), \(\beta_2=0.999\), weight decay 0.01, and \(\epsilon=10^{-8}\). A random contiguous 70%–100% span of mel frames is masked. Inference uses an Euler solver with 32 function evaluations. The cache does not report learning rate, batch size, training duration, or a separate end-to-end latency benchmark, so evaluation count alone does not establish real-time performance.

Key Experimental Results

Main Results

CelebV-Dub uses the official split of 79,933 training samples and 213 test samples. Setting 1 uses target ground-truth speech as the reference; Setting 2 uses another clip of the same speaker. CinePile-Dub is an out-of-domain movie evaluation: models train only on CelebV-Dub and encounter unseen voices and videos, rather than being fine-tuned on the movie test set (original Tables 1, 4, and 5).

WER is computed using Whisper-large-V3 recognition and is lower-is-better. SPKSIM measures WavLM-TDNN speaker-embedding similarity; EMOSIM measures cosine similarity between Emotion2Vec emotion embeddings, both higher-is-better. DNSMOS is a quality predictor, not human MOS. Sync-KL is a lower-is-better duration-divergence proxy that supports synchronization claims only when generated and ground-truth total durations are close.

Dataset / Setting	Method	SPKSIM (%) ↑	WER (%) ↓	EMOSIM (%) ↑	Sync-KL ↓	DNSMOS ↑	Source
CelebV-Dub / 1	AlignDiT	59.71	9.48	84.54	0.402	3.45	Table 2
CelebV-Dub / 1	CoSyncDiT	65.21	4.29	84.61	0.392	3.46	Table 2
CelebV-Dub / 2	AlignDiT	49.49	13.18	79.70	0.413	3.47	Table 3
CelebV-Dub / 2	CoSyncDiT	53.44	6.39	80.29	0.381	3.47	Table 3
CinePile-Dub / Zero-shot 2	AlignDiT	43.82	23.92	72.53	0.355	3.33	Table 5
CinePile-Dub / Zero-shot 2	CoSyncDiT	47.24	7.53	70.12	0.319	3.39	Table 5

These results do not establish universal dominance. InstructDub achieves 5.64% WER versus 6.39% for CoSyncDiT on CelebV-Dub Setting 2, and 4.61% versus 5.59% on CinePile-Dub zero-shot Setting 1 (Tables 3 and 4). Under zero-shot Setting 2, CoSyncDiT's 70.12% EMOSIM is below AlignDiT's 72.53%. The advantage is the overall trade-off, not superiority on every metric.

Ablation Study

All configurations below are evaluated on CelebV-Dub Setting 2; values come from original Table 6.

Config	SPKSIM (%) ↑	WER (%) ↓	Sync-KL ↓	Effect
Full model	53.44	6.39	0.381	Reference
Without Acoustic Style Adapting	19.64	6.84	0.385	Strong damage to voice identity
Without Fine-grained Visual Calibrating	53.25	6.40	0.419	Timing degrades more than pronunciation
Without Time-aware Context Aligning	52.75	7.39	0.446	Largest synchronization-divergence increase
Without all JSAR	51.30	8.72	0.431	Pronunciation and timing both degrade
Without semantic consistency	52.34	8.39	0.392	Primarily harms pronunciation
Without temporal consistency	51.37	6.58	0.425	Primarily harms synchronization

Key Findings

• Removing Acoustic Style Adapting lowers SPKSIM by 33.80 percentage points; removing Context Aligning raises Sync-KL by 0.065. This supports the division between establishing voice identity and calibrating lips/text, but individual ablations do not validate a cognitive mechanism.
• Table 7 adds independent visual synchronization metrics. In Setting 2, CelebV-Dub LSE-C/LSE-D are 6.54/7.85 versus AlignDiT's 6.44/7.92; CinePile-Dub scores are 5.13/9.26 versus 5.05/9.33. The direction is consistent, but the margins are limited.
• Table 8 samples 50 CinePile-Dub clips for human evaluation on a 1–5 scale with 0.5 increments. CoSyncDiT obtains MOS-N 4.04±0.09 and MOS-S 4.01±0.10, versus AlignDiT's 4.02±0.11 and 3.97±0.12. The intervals overlap, so higher means alone do not establish statistical significance.

Highlights & Insights

• The method makes multimodal fusion concrete through injection order and supervision location. Visual residual calibration and later text retrieval have distinct responsibilities, making failures easier to isolate through ablation than simply adding more cross-attention.
• JSAR places temporal supervision on contextual features and semantic supervision on final hidden states. Other conditional speech-generation tasks could borrow this error-specific placement, provided that the frame-level teacher is reliable.

Limitations & Future Work

• The paper has no dedicated limitations section. For reproducibility, the cache does not specify the CinePile-Dub test size, loss weights, learning rate, training budget, or number of human raters; architecture details alone do not fill these gaps.
• Setting 1 uses target ground-truth speech as the reference and is comparatively favorable. Setting 2 and out-of-domain evaluation are closer to deployment; the results do not establish cross-language dubbing, arbitrary overlapping dialogue, or background-audio generation.
• Sync-KL assumes comparable total duration, while the supplementary visual synchronization improvements are modest. Stratified analysis of severe occlusion, profile views, and different-content references would be more informative than treating overall averages as reliability on every shot.
• Voice cloning requires authorization for reference voices and identity safeguards. This is an application-level risk assessment, not a claim that the paper implements consent verification or watermarking.

Related Work & Insights

• vs AlignDiT: both avoid external forced phoneme alignment. CoSyncDiT addresses interference between reference audio and target text through ordered conditioning and dual-location regularization rather than homogeneous fusion throughout the architecture.
• vs StyleDubber / ProDubber / InstructDubber: these methods emphasize phoneme duration, prosody, or emotion control, and retain better WER in some settings. CoSyncDiT's main value is reducing rigid duration constraints while improving the joint identity/synchronization trade-off.
• vs F5-TTS / FlowDubber: flow matching alone is insufficient for movie dubbing. The former does not supply fine-grained visual synchronization, while the latter still relies on forced alignment and sequence expansion. The transferable contribution is condition organization and supervision, not merely adopting flow matching.

Rating

• Novelty: 4/5. Reorganizes conditioning and regularization around reference/target interference, without proposing new flow-matching theory.
• Experimental Thoroughness: 4/5. Covers two reference settings, out-of-domain movies, component ablations, and objective/subjective evaluation, but lacks budget and stratified robustness reporting.
• Writing Quality: 4/5. The problem, diagrams, and ablations connect clearly; corrupted cached equations and missing setup details limit reproducibility.
• Value: 4/5. A useful conditioning design for dubbing, with deployment value still dependent on authorization, speed, and complex-dialogue validation.