Skip to content

JAM-Flow: Joint Audio-Motion Synthesis with Flow Matching

Conference: ECCV2026
Paper: ECCV page
Area: Video Generation / Audio & Speech
Keywords: joint audio-motion generation, flow matching, talking head generation, zero-shot TTS, multimodal diffusion transformer

TL;DR

JAM-Flow treats LivePortrait's low-dimensional mouth expression keypoints and F5-TTS's mel-spectrograms as two mutually maskable modalities inside a single flow-matching framework, trained jointly through a dual-stream MM-DiT with half-fused joint attention, RoPE temporal alignment, and modality-asymmetric attention masks; one ~500M model covers talking head generation, TTS, motion-to-audio, and automated dubbing, producing a 20-second video in 45 seconds on a single RTX A6000.

Background & Motivation

Face video synthesis and speech synthesis have both advanced rapidly over the past few years, yet they remain two largely disjoint lines of work. On the talking head side, GAN-based methods (SadTalker, LivePortrait) rely on explicit motion representations and warping to reach near real-time speed, while diffusion-based methods (Hallo, AniPortrait, X-Portrait, OmniHuman) achieve higher fidelity at the cost of 7 to 30 minutes for a 20-second video, plus temporal flicker over long sequences. On the speech side, the line running from Tacotron through non-autoregressive FastSpeech and diffusion-based zero-shot cloning such as VoiceBox to F5-TTS, which pushes non-autoregressive TTS to 2.42% WER with flow matching, is equally mature. The problem is that both directions are built as separate pipelines, chained together only through an intermediate audio or video file.

Real speech, however, is a bidirectionally coupled process: the movement of lips, cheeks, and jaw is not merely a picture pasted onto the sound, it is an integral part of spoken language, and prosody is in turn carried by facial expression. In current practice, talking head models treat audio as a strictly unidirectional condition (audio-to-visual), while TTS systems are entirely blind to facial dynamics. Cascading the two systems costs more than stacked latency and accumulated error: such an architecture simply cannot express the reverse relation — generating speech that matches an already recorded mouth motion is out of reach for a conventional TTS system, because its conditioning has no visual branch.

The angle taken here is not to train a unified large video generation model from scratch, but to reuse two strong unimodal pretrained models and let the network focus on learning the coupling between them. LivePortrait supplies a compact implicit keypoint representation and efficient warping-based rendering, F5-TTS supplies conditional flow matching (CFM) generation over mel-spectrograms, and the two are welded together by an "inpainting-style" random masking supervision scheme. Core idea: treat facial motion (modeling only the 4 of LivePortrait's 21 expression dimensions that control the mouth) and mel-spectrograms as maskable completion tasks, and train a partially fused dual-stream MM-DiT with modality-asymmetric attention masks, so that a single model generates speech and lip motion conditioned on each other from any subset of inputs.

Method

Overall Architecture

The goal is to generate temporally aligned mel-spectrograms and facial motion from any subset of three conditioning signals: text, reference audio, and reference motion. The architecture is dual-stream: a Motion-DiT generates mouth keypoints \(\mathbf{e}_{\text{mouth}} \in \mathbb{R}^{T_{\text{frame}}\times 4\times 3}\) in LivePortrait's expression deformation space, and an Audio-DiT generates mel-spectrograms \(\mathbf{a}_{\text{mel}} \in \mathbb{R}^{T_{\text{mel}}\times d_{\text{mel}}}\). Tokens from the two streams exchange information only in a subset of transformer layers via joint attention; the remaining layers stay modality-specific. During training the inputs of both streams are randomly masked and the model completes the masked parts under a conditional flow matching objective. At inference, controlling which inputs stay unmasked switches the task — text alone is joint generation, text plus reference audio is zero-shot cloning, reference motion is motion-to-audio, and video plus text is automated dubbing. The outputs are handed to two off-the-shelf decoders: Vocos turns the mel-spectrogram into a waveform, and the mouth keypoints, together with fixed identity and pose features, go to the LivePortrait decoder that warps video frames.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["text / reference audio / reference motion"] --> B["Inpainting-style joint supervision<br/>random masking + two-stage training"]
    B --> C["Low-dimensional mouth keypoints<br/>only 4 mouth dims generated"]
    B --> D["Audio-DiT<br/>mel-spectrogram CFM branch"]
    C --> E["Half-fused joint attention<br/>dual-stream MM-DiT"]
    D --> E
    E --> F["RoPE temporal alignment<br/>modality-asymmetric masks"]
    F --> G["Vocos decoder / LivePortrait decoder"]

Key Designs

1. Inpainting-style joint supervision: turning both modalities into completion tasks so cross-modal dependence is forced by the objective itself

The naive route is to train two unimodal models separately and cascade them, but then cross-modal information can only travel one way through an inference-time interface, and the model never learns a joint distribution. This paper applies F5-TTS's "speech generation as mask-and-predict" formulation to both modalities at once: during training a segment of the mel-spectrogram and a segment of the motion sequence are randomly masked, and the masking pattern varies — sometimes only motion is hidden, sometimes only audio, sometimes both — so the model must complete the missing part from the remaining context alone. Both streams use the same conditional flow matching objective: given paired samples \(x_0\) (prior end) and \(x_1\) (data end), regress the target velocity \(x_1-x_0\) along the linear interpolation path.

\[\mathcal{L}_{\text{CFM}}(\theta)=\mathbb{E}_{t,\,x_0,\,x_1}\left[\left\|v_\theta(x_t,t)-(x_1-x_0)\right\|^2\right],\qquad x_t=(1-t)\,x_0+t\,x_1\]

(Equations (1)–(4) of the paper are corrupted in the cached rendering; this is reconstructed from the prose description of the linear interpolation path and target velocity, ⚠️ refer to the original paper.) The total loss is the sum of the audio and motion CFM losses.

The direct consequence is that completing the masked audio requires reading the unmasked motion, and vice versa — mutual conditioning is not an add-on module but a behavior forced by the training objective. More practically, since the masking pattern is random during training, the inference configurations "generate motion from full audio", "generate audio from motion only", and "generate everything from text only" all fall inside the same conditional distribution, so no task-specific architecture or output head is needed. The authors state this is the first framework to apply inpainting-style joint supervision to the simultaneous generation of two modalities.

2. Low-dimensional mouth keypoints: analyze the pretrained representation first, then cut the generation space down to 4 dimensions

This design comes from an observation about an existing representation rather than from network architecture. LivePortrait's motion encoder outputs a 3D expression deformation \(\mathbf{e}\in\mathbb{R}^{21\times 3}\) for 21 canonical keypoints. Visualizing this space dimension by dimension reveals that roughly 4 of the dimensions consistently govern the mouth region: perturbing only those 4 changes the lip shape, while freezing the rest leaves the rest of the frame untouched. The Motion-DiT therefore generates only \(\mathbf{e}_{\text{mouth}}\subset\mathbf{e}\) (4 dimensions × 3 coordinates), while the other 17 dimensions \(\mathbf{e}_{\text{rest}}\) are fed in as known conditions and combined with identity features and pose parameters for decoding.

Shrinking "what must be generated" from whole frames to this small subspace is the root reason the model runs near real time: a 25 fps, 20-second video requires generating \(500\times 4\times 3=6000\) scalars, whereas a pixel-space diffusion model must generate 500 high-resolution frames over the same duration. A secondary benefit is identity preservation — video frames are warped from source frames rather than re-synthesized, so appearance does not drift as it does in pixel diffusion. The discussion also notes that compact keypoint warping is not only cheaper but exhibits less flicker over long sequences than diffusion-based talking head methods such as Hallo and Hallo3.

3. Half-fused joint attention in a dual-stream MM-DiT: fuse cross-modal information without washing out either modality's prior

The common MM-DiT recipe gives each modality its own DiT branch and applies symmetric joint self-attention in every layer. Here the two token streams are concatenated for joint attention in only \(N_{\text{joint}}\) layers (half the total, 11 in the experiments), placed in the shallower part of the stack; the rest of the layers are computed per modality. This preserves the F5-TTS speech prior in the Audio-DiT and the motion prior in the Motion-DiT, so cross-modal information is injected only at shallow depth instead of blending both representations into one modality-agnostic space. The ablation shows the full-joint variant (22 layers) scores slightly better (LSE-C 4.81 vs. 4.64) but trains unstably and costs far more compute; half fusion gives comparable quality at much lower cost.

An equally important precondition is how the two branches are initialized. The Audio-DiT adopts pretrained F5-TTS directly; the Motion-DiT is first trained from scratch on CelebV-Dub following the standard talking head recipe (wav2vec2 features concatenated to the motion input) until convergence, and only then are the joint attention layers attached. The authors conclude that each modality must first be trained well on its own before cross-modal relations can be learned. Equally, the Audio-DiT must be unfrozen and fine-tuned during the joint stage: freezing it yields a slightly better WER (6.76% vs. 7.28%) but noticeably worse synchronization, because the Motion-DiT can only unilaterally accommodate a frozen audio distribution and never learns a genuine joint distribution; only joint adaptation through the shared attention layers produces synchronized, coherent output.

4. RoPE temporal alignment and modality-asymmetric attention masks: genuinely aligning two modalities with different lengths and different receptive fields

This is the most novel architectural design in the paper. Two difficulties compound. First, the two streams have different token counts (mel-spectrograms follow the audio frame rate, motion follows the video frame rate), so their positional index scales differ and queries and keys interact at incompatible rotation angles when joint attention is applied naively. Second, the two modalities need completely different receptive fields: facial motion depends almost entirely on adjacent frames, whereas speech prosody depends on the context of the whole utterance. Prior multimodal transformers used symmetric attention for both streams, which does not work here.

The paper makes a very light RoPE normalization: the rotation angle at position index \(p\) becomes

\[\phi(p,d)=\frac{p}{L}\cdot L_{\text{ref}}\cdot\theta_d,\qquad L_{\text{ref}}=\max(L_a,L_m)\]

where \(L\) is the sequence length of the current modality. Audio and motion tokens at the same timestep thus land at comparable angular scales despite very different sequence lengths. The authors call the adjustment straightforward yet indispensable — without it joint training does not converge at all, which is why the ablation table has no corresponding row.

The masks are then tailored per modality according to its functional role. For motion tokens, self-attention is restricted to a local window (matching the inductive bias that facial motion depends on temporally adjacent frames), motion-to-audio attention is likewise confined to the corresponding local window, and audio-to-audio attention is disabled during motion generation. Conversely, for audio tokens, global self-attention across the whole utterance is retained, cross-modal attention is allowed only to motion tokens at the same timestamp, and motion-to-motion self-attention is masked out entirely. The combined effect is that each modality attends in the way most natural to its own generative process, while the cross-modal channel is strictly limited to "the same instant". Removing the masks causes a sharp drop in lip-sync quality, and the authors observe that the model often fails to achieve coherent joint training at all, producing temporally drifting or unsynchronized motion; to their knowledge no prior work has designed such asymmetric, modality-specific masking for joint attention.

A Worked Example: from a silent driving video to a dub

The input is a 20-second, 25 fps silent talking video (the V2V setting) with a target line of text. The system first extracts 17 non-mouth expression keypoints per frame as \(\mathbf{e}_{\text{rest}}\), keeping them unmasked as a "clue" (following [79]). During the forward pass the motion branch runs CFM over those 500 frames, generating only \(4\times 3\) mouth deformation values per frame, \(500\times 4\times 3=6000\) scalars in total, while the audio branch runs CFM over the same timeline to generate the corresponding mel-spectrogram. Joint attention between the two streams happens only in the 11 shallow layers, where audio tokens may attend only to motion tokens at the same timestamp and motion tokens only to audio tokens inside their local window. Afterwards the mouth deformation, \(\mathbf{e}_{\text{rest}}\), identity features, and pose parameters go to the LivePortrait decoder to warp 500 frames, and the mel-spectrogram goes to Vocos for waveform decoding. The whole run takes about 45 seconds on a single RTX A6000.

Loss & Training

Training proceeds in two stages. Stage 1 prepares the unimodal pretrained models: the Audio-DiT takes F5-TTS directly, and the Motion-DiT is trained from scratch with wav2vec2 audio features as the condition, following the standard talking head recipe. Stage 2 attaches the joint attention layers and applies RoPE alignment, training both branches together so gradients flow across modalities through the shared attention layers; wav2vec2 features are dropped at this stage — text conditioning is injected through the Audio-DiT, and the Motion-DiT instead consumes \(\mathbf{e}_{\text{rest}}\) plus the audio branch's intermediate features \(f_{\text{audio}}\).

Both stages use the same CFM objective, and the total loss is the sum of the audio and motion losses, with random masking applied to both input streams during training (motion only, audio only, or both masked). No diffusion distillation is used, and the authors believe there is still room to make inference faster.

Key Experimental Results

Training uses CelebV-Dub (filtered from CelebV-HQ and CelebV-Text) with paired audio-visual data for the joint stage; evaluation is on the CelebV-Dub test split and HDTF. Among the metrics, FID and FVD measure frame quality and temporal consistency, LSE-C (higher is better) and LSE-D (lower is better) are SyncNet's lip-sync confidence and distance, WER is word error rate, and SIM-o and spkSIM measure speaker similarity to the reference voice.

Main Results

Table 1. Talking head generation on HDTF; inference measured on a single RTX A6000 for a 20-second video.

Method FID ↓ FVD ↓ LSE-C ↑ LSE-D ↓ Inference (20s) ↓ #Params
Ground Truth - - 8.70 6.597 - -
SadTalker 22.340 203.860 7.885 7.545 2.5 min (with GFPGAN) ~200M
DreamTalk 78.147 890.660 6.376 8.364 - -
AniPortrait 26.561 234.666 4.015 10.548 7 min ~1B
Hallo 20.545 173.497 7.750 7.659 23 min 1–3B
Hallo3 20.359 160.838 7.252 8.106 30 min (H100) 1–3B
Ours-Stage1 18.372 194.27 7.138 7.947 - -
Ours (I2V) 17.571 192.30 7.324 7.777 45 sec ~500M (joint)
Ours (V2V) 11.633 25.07 8.086 7.181 45 sec ~500M (joint)

Table 2. Text-to-speech on LibriSpeech-PC test-clean.

Method WER ↓ SIM-o ↑
CosyVoice 3.59% 0.66
FireRedTTS 2.69% 0.47
E2 TTS 2.95% 0.69
F5-TTS 2.42% 0.66
MegaTTS 3 2.31% 0.70
Ours 4.91% 0.64

Table 3. Automated video dubbing on CelebV-Dub.

Method LSE-C ↑ LSE-D ↓ WER ↓ spkSIM ↑
Ground Truth 6.73 7.44 4.15% -
Zero-Shot TTS 2.78 11.68 3.83% 0.316
HPMDubbing 6.36 7.80 24.06% 0.146
StyleDubber 3.78 10.40 9.48% 0.264
VoiceCraft-Dub 6.05 8.33 7.01% 0.333
Ours 3.43 10.56 6.39% 0.410

Table 4. Ablation on the number of joint attention layers, motion attention masking, and Audio-DiT fine-tuning.

# Joint Layers Attn. Mask Train Audio-DiT FID ↓ LSE-C ↑ LSE-D ↓ WER ↓ SIM-o ↑
22 (Full) ✗ ✗ 5.759 4.81 8.40 6.88% 0.62
11 (Half) ✗ ✗ 5.735 4.64 9.07 7.25% 0.62
11 (Half) ✗ ✓ 5.748 6.44 7.99 7.93% 0.61
11 (Half) ✓ ✗ 5.747 5.76 8.22 6.76% 0.63
11 (Half) ✓ ✓ 5.662 6.45 7.73 7.28% 0.64

In the ablation, LSE-C and LSE-D are computed from generated motion paired with ground-truth audio (generated audio makes SyncNet metrics unstable), while WER and SIM-o use both generated motion and audio.

Key Findings

  • Joint training gives a real gain over the unidirectional Stage-1 model. Ours-Stage1 is a unimodal motion branch conditioned on wav2vec2 features, trained in the conventional audio-driven way; its FID 18.372 / FVD 194.27 are close to I2V's 17.571 / 192.30, but under the V2V setting FID drops to 11.633, FVD collapses to 25.07, and LSE-C rises to 8.086. The same model improves on both synchronization and quality after switching to the joint framework, which is the most direct evidence for the value of inpainting-style joint supervision.
  • RoPE alignment is a make-or-break design. The authors state explicitly that without it joint training does not converge at all, which is why no ablation row exists; this is a precondition for training rather than a component that costs a point or two.
  • The attention masks matter more than the numbers suggest. In Table 4, enabling the masks (11 half, no Audio-DiT fine-tuning → with fine-tuning) lifts LSE-C from 4.64 to 6.45, but the authors stress that beyond raw scores, removing the masks means the model often fails to learn coherent joint behavior and produces temporally drifting output — the subjective degradation is far worse than the metric degradation.
  • Full joint attention is not worth it. The 22-layer variant scores slightly higher on LSE-C (4.81 vs. 4.64) than the half-fused one, but trains unstably and is much heavier, so half fusion is chosen as the performance/efficiency trade-off.
  • The Audio-DiT must be unfrozen, which creates a WER-versus-sync trade-off. Freezing it gives a better WER (6.76% vs. 7.28%) but drops LSE-C from 6.45 to 5.76; unfreezing lets both modalities co-adapt through the shared attention layers, giving clearly better synchronization and coherence.
  • The TTS gap traces back to supervision, not capacity. CelebV-Dub's transcripts are Whisper-generated pseudo-labels, and Whisper-base already reports 4.50% WER on LibriSpeech-PC, close to the ~4.9% observed here — indicating the word error rate is bounded by label noise rather than model capability. The audio track was also obtained via Spleeter source separation, which introduces artifacts absent from standard TTS corpora.
  • Joint training produces emergent behavior beyond explicit supervision. The authors observe that smiling motion tends to produce brighter, higher-pitched voices despite the absence of any emotion supervision, along with natural pauses and consistent tone shifts, suggesting the model captures cross-modal alignment beyond timestamp-level synchronization. Another oddity is asymmetric adaptation: when generating audio and motion simultaneously, the model tends to keep the visual motion nearly unchanged and adjust the audio more substantially to match the given motion, plausibly because realistic visual motion is harder to modify than audio; the practical effect is that visual identity is preserved.
  • It achieves the highest speaker similarity on automated dubbing. On a task it was not explicitly optimized for, the method reaches 6.39% WER versus VoiceCraft-Dub's 7.01% and the highest spkSIM of 0.410; but its LSE-C/LSE-D are on par with Zero-Shot TTS and clearly behind HPMDubbing and VoiceCraft-Dub — the authors warn that SyncNet-style metrics are unstable under codec variation and with TTS-generated audio and should not be over-interpreted. In the user study the method is preferred over VoiceCraft-Dub in 62.6% versus 37.4% of cases with no votes for the other baselines, and in the HDTF user study the V2V and I2V variants are ranked the top two methods in overall quality.

Highlights & Insights

  • Designing the generation space by analyzing an existing representation is the paper's finest move. It changes no network structure; instead it visualizes LivePortrait's expression deformation space, confirms that ~4 of the 21 dimensions control the mouth, and then cuts the Motion-DiT's target down to \(4\times3\) dimensions. The dimensionality of the generation space is the precondition for efficiency, and that judgment comes from observation rather than search — any work reusing a pretrained motion representation can follow the same procedure.
  • The modality-asymmetric masks write "how the two modalities should look at each other" into the inductive bias explicitly. Where earlier multimodal transformers assume symmetric attention, this paper argues that motion should only see its local window, speech should see the whole utterance, and cross-modal attention should only connect the same instant — three constraints each corresponding to a physical property of the respective generative process. The idea transfers to any pair of modalities with different sampling rates or temporal scales (video + audio, video + depth).
  • "Don't train a big model, only learn the coupling" turns out to be a viable alternative route. The Audio-DiT is initialized from F5-TTS and rendering is delegated to LivePortrait's warping decoder, so the paper only newly trains the motion branch plus a thin layer of joint attention and still reaches 45 seconds per 20-second clip. This suggests that in multimodal generation, spending compute on the interface may pay off better than piling it onto the backbone.
  • Inpainting-style joint supervision solves training signal and inference flexibility at once. Random masking makes "partially missing inputs" a common case inside the training distribution, so inference needs no task-specific architecture or branch switching — one model natively supports text-only generation, zero-shot cloning, motion-driven generation, and video dubbing. This "trade a training objective for free multi-task ability" pattern is worth reproducing in other conditional generation tasks.

Limitations & Future Work

  • The authors acknowledge limitations rooted mainly in data and compute: CelebV-Dub's transcripts are Whisper pseudo-labels and its audio comes from source separation, so the errors propagate directly into the supervision; LivePortrait confines the modelable motion to the facial region, leaving head pose and body out of scope. They suggest cleaner data and stronger video diffusion backbones as the way to extend the framework.
  • The evaluation itself warrants caution: the FVD gap in Table 1 (25.07 versus 160–890 for the baselines) needs careful attribution, because the V2V setting warps frames from the driving video so that nearly all appearance comes from the source frames. That is not the same source as diffusion methods synthesizing pixels from scratch (Hallo, Hallo3, AniPortrait), and the paper does not separate how much of the quality and consistency advantage comes from generative ability versus from generating almost nothing. Under I2V the FVD of 192.30 is actually slightly behind Hallo3's 160.838, and that comparison is closer to like-for-like.
  • There is no strict control experiment comparing "one joint model" against "separately trained TTS plus talking head models, cascaded". Ours-Stage1 in Table 1 is only a unidirectional, wav2vec2-conditioned motion branch, neither an independent TTS baseline nor a paired cascaded system, so the direct evidence for the central claim that joint training beats separate training remains thin.
  • The conclusions about Audio-DiT fine-tuning in Table 2 and Table 4 must be read against each other: the WER regression it causes (6.76% → 7.28%) stacks on an already high absolute WER driven by data noise, leaving a sizeable gap to dedicated TTS systems (2.31%–3.59%). The paper attributes this to label noise but does not train an F5-TTS control on the same Whisper pseudo-labeled data to quantify that contribution directly.
  • In the flagship joint generation scenario (text to audio + motion), speaker identity is randomly sampled and no quantitative evaluation of identity control or identity consistency is provided. Cross-modal evaluation also leans heavily on SyncNet-style metrics which the authors themselves call unstable on generated audio, meaning part of the main conclusion rests on an unreliable measure.
  • Concrete improvement directions: retrain on human transcripts or cleaner dubbing data to unlock WER; replace the warping decoder with a lightweight video diffusion backbone to extend motion beyond the face (at the cost of the current real-time speed); distill the two CFM streams to further cut sampling steps; and generalize inpainting-style joint supervision to other modality pairs such as depth + video and audio + video.
  • vs F5-TTS: F5-TTS is a unimodal CFM zero-shot TTS model and the initialization source for the Audio-DiT here. This paper keeps its mask-and-predict formalization but extends the masking mechanism from within audio to "one entire other modality", and unfreezes it for joint training. The price is a WER regression from 2.42% to 4.91%, in exchange for lip-sync and motion-conditioned generation in the same model — closer to a capability trade than a pure performance improvement.
  • vs MM-DiT (SD3 / Flux / CogVideoX): These works use one DiT branch per modality with full-layer symmetric joint attention for multimodal generation, but always in a "one side generates, the other conditions" setting or between inherently asymmetric modalities such as text and images. This paper moves joint attention to a subset of layers and adds modality-specific asymmetric masks, and is the first to use inpainting-style joint supervision to synthesize two modalities simultaneously — the two points that separate it from the existing MM-DiT line.
  • vs diffusion talking head methods (Hallo / Hallo3 / AniPortrait / OmniHuman): They generate video in pixel space with audio as a unidirectional condition, reaching high quality but requiring 7–30 minutes of inference and flickering over long sequences. This paper generates in a compact keypoint space and restores pixels by warping, so it is an order of magnitude faster with stable identity; the price is that motion is confined to what LivePortrait can express, so expressiveness on complex scenes and large expressions falls short of a diffusion backbone.
  • vs LivePortrait: LivePortrait is a video-driven reenactment framework that produces no audio itself; here it serves as the reused motion representation and rendering backend. The contribution runs the other way — the paper shows that its expression deformation space contains a 4-dimensional mouth subspace small enough to be generated in real time.
  • vs VoiceCraft-Dub / HPMDubbing and other automated dubbing work: Those methods focus on synthesizing aligned speech for a given video, typically with extra duration or style control modules. This paper does nothing dubbing-specific; the capability falls out of the natural inpainting configuration where motion stays unmasked and audio is masked. Its WER and spkSIM are competitive, but its lip-sync metrics lag, indicating that the usable boundary of this emergent capability still needs more reliable evaluation to pin down.

Rating

  • Novelty: ⭐⭐⭐⭐ [The first framework to apply inpainting-style joint supervision to simultaneous audio-motion generation; the modality-asymmetric attention masks and length-normalized RoPE alignment are original architectural contributions, though the backbone and decoders are reused]
  • Experimental Thoroughness: ⭐⭐⭐ [Covers talking head generation, TTS, and dubbing with an ablation, but lacks a strict joint-versus-cascaded control, and the dubbing task's key metrics rest on SyncNet measures the authors themselves distrust]
  • Writing Quality: ⭐⭐⭐⭐ [Motivation and design rationale are clear, and the discussion of metric failure and data noise is notably candid; some equations are corrupted in the cached/preprint rendering]
  • Value: ⭐⭐⭐⭐ [Offers a practical route of "reuse pretrained unimodal models plus a thin coupling layer", and the 45-seconds-per-20-seconds inference speed has direct relevance for real-time digital human applications]