VoCa: Unified Autoregressive Modeling for Talking Audio-Video Generation

Conference: ECCV 2026
Paper: ECCV Official
Full Text Cache: /Users/zy/workspace/paper_cache/ECCV2026/eccv-4195.txt
Area: Video Generation (also suitable for Audio & Speech / Multimodal VLM)
Keywords: Unified Autoregressive Modeling, Talking-Head Generation, Audio-Video Generation, Diffusion Distillation, Rolling-Window Denoising

TL;DR

VoCa establishes a unified autoregressive generation framework that leverages an LLM to map text transcripts into shared representations driving dedicated speech and video decoders simultaneously, utilizing a speech refiner and causal rolling-window denoising distillation to achieve strictly synchronized, streaming talking-head audio-video generation.

Background & Motivation

Conventional talking-head generation systems predominantly follow an audio-driven paradigm, animating facial geometry and motion conditioned on an existing audio track. In interactive scenarios where only textual transcripts are available, standard practice relies on a cascaded pipeline: a text-to-speech (TTS) engine synthesizes speech audio first, which is subsequently fed into a visual diffusion model to drive avatar animation. This sequential pipeline inevitably suffers from cumulative error propagation and substantial end-to-end latency. More critically, because the visual generator conditions strictly on acoustics rather than linguistic semantics, subtle misalignments between spoken intentions and co-speech bodily gestures frequently emerge. Recent end-to-end models attempt to tackle these drawbacks through twin-backbone diffusion designs, yet they require training separate audio branches from scratch at massive computational expense and still necessitate an external LLM for conversational reasoning.

The core tension underlying text-driven talking-head synthesis lies in the contradiction between high-level causal semantic reasoning and strict low-level audio-visual temporal synchronization. On one hand, discrete audio representations produced by autoregressive language models are conventionally optimized for waveform reconstruction rather than phonetic-visual grounding, leaving a noticeable semantic gap against the nuanced phonemic cues required for expressive lip motion. On the other hand, high-fidelity video diffusion transformers fundamentally rely on bidirectional attention across complete temporal sequences, rendering them structurally incompatible with the causal, step-by-step nature of streaming generation.

This paper addresses the dilemma by turning an autoregressive large language model into the shared temporal backbone for both modalities. The LLM jointly outputs discrete acoustic tokens and continuous hidden states, which are simultaneously consumed by dedicated speech and causal video decoders via specialized cross-modal refinement and windowed denoising. Core idea: construct a unified autoregressive co-generation framework where an LLM provides shared acoustic-semantic representations, a speech refiner distills self-supervised speech semantics to bridge the cross-modal gap, and a unified active window with rolling-window distillation adapts bidirectional diffusion into a streaming causal generator.

Method

Overall Architecture

VoCa accepts a textual transcript and a static reference portrait image as input, synthesizing synchronized speech waveforms and video frames in a unified autoregressive stream. The architecture comprises three integrated components: a 3B-parameter autoregressive Large Language Model (LLM) equipped with a Dual-FFN design, a neural audio decoder based on an improved RVQGAN (DAC), and a video decoder consisting of a dedicated speech refiner alongside a causal diffusion transformer.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Text Transcript + Reference Image"] --> B["LLM Shared Representation Generation<br/>Predict Discrete Audio Tokens & Hidden States"]
    B --> C["Unified Window Partitioning<br/>Manage Rolling Active Window & History Cache"]
    C --> D1["Speech Decoder Synthesis<br/>RVQGAN Time-Domain Waveform Decoding"]
    C --> D2["Speech Refiner Cross-Modal Bridging<br/>Temporal Interpolation + Residual CNN + HuBERT Distillation"]
    D2 --> E["Causal Diffusion Transformer<br/>Localized Audio Attention + Decoupled Identity Injection"]
    E --> F["Rolling-Window Denoising & DMD Distillation<br/>Chunked Progressive Denoising & Few-Step Synthesis"]
    D1 --> G["Output: Synchronized Speech Waveform & Talking-Head Video"]
    F --> G

During inference, the LLM autoregressively predicts discrete audio tokens and their corresponding multi-layer hidden states. These representations populate a fixed-length unified active window that advances chunk by chunk. The speech decoder decodes the audio tokens into time-domain waveforms. Concurrently, the speech refiner aligns, enhances, and downsamples the shared representations, conditioning the causal diffusion transformer. Utilizing both global and local key-value caches, the transformer denoises video latents within a synchronized active denoising window, streaming high-fidelity frames in lock-step with the audio.

Key Designs

1. Speech refiner cross-modal bridging: closing the semantic gap between acoustic tokens and facial dynamics

Discrete audio tokens predicted by the LLM are inherently pre-trained for acoustic waveform reconstruction; while rich in acoustic textures, they lack explicit phonetic and linguistic abstractions necessary for fine-grained lip dynamics. The speech refiner addresses this limitation through a multistage transformation. It first temporally interpolates the LLM representations onto the video frame rate, then feeds them through a stack of 1D convolutional residual blocks to accentuate rhythm and stress cues. These refined tokens are concatenated with the LLM's continuous hidden states—which supply utterance-level emotional and non-verbal semantic context—and downsampled via two temporal convolution layers (stride 2) to match the temporal shape of the video latents.

To instill robust phonemic understanding, an auxiliary distillation loss \(\mathcal{L}_{aux}\) is applied during the pretraining stage. Ground-truth audio is processed through a self-supervised HuBERT model, whose layer-wise representations are average-pooled to form the target feature \(h_{target}\). The refiner's output \(h_{refiner}\) is supervised directly:

\[\mathcal{L}_{aux} = \| h_{refiner} - h_{target} \|_2^2\]

Combined with the video latent Flow Matching loss \(\mathcal{L}_{flow}\), this auxiliary constraint forces the refiner to capture fine-grained phonetic landmarks while retaining utterance-level stylistic conditioning.

2. Localized audio and decoupled identity attention: enforcing strict causal alignment and identity preservation

The video generation backbone is adapted from the Wan 3D diffusion transformer operating in VAE latent space. To strictly eliminate future audio leakage and curb the quadratic computational overhead of full cross-attention, VoCa introduces localized audio attention: spatial tokens of any video frame attend exclusively to the speech refiner's conditioning vector at the exact matching time index. This one-to-one temporal mapping guarantees causal generation and establishes deterministic synchronization between acoustic cues and visual lip motions.

To ensure long-term appearance and facial fidelity, identity features are injected via a decoupled dual-pathway mechanism. A DINOv2 vision encoder extracts global appearance characteristics (such as clothing and general morphology), while an InsightFace model extracts high-precision facial geometry embeddings. Within each transformer block, two dedicated cross-attention layers independently condition the frame latents on DINOv2 and InsightFace features, summing their outputs. This dual formulation prevents facial feature blurring and maintains identity stability across dynamic head turns.

3. Unified window partitioning and rolling-window DMD distillation: transforming bidirectional diffusion into a causal streaming generator

To synchronize the diffusion process with the LLM's autoregressive rate, VoCa integrates a unified window partitioning mechanism with Distribution Matching Distillation (DMD). The latent video sequence is structured into non-overlapping chunks of \(K=3\) latent frames. The reference portrait is encoded into an initial clean chunk \(B_0\) (at timestep \(t=0\)), which together with the first generated motion chunk \(B_1\) forms an immutable global context cache. A rolling history cache maintains recent clean frames for local temporal continuity.

During rollout, the active denoising window spans up to \(M=5\) adjacent chunks assigned progressively higher noise levels. The newest chunk enters at pure Gaussian noise, intermediate chunks continue denoising from noisy caches, and the oldest chunk completes its final denoising step and transfers into the clean KV cache. The student generator \(G_\theta\) is trained via DMD distillation against a frozen real-score network \(s_{\mathrm{real}}\) and a trainable fake-score network \(s_{\mathrm{fake}}\):

\[\mathcal{L}_{DMD} = \frac{1}{2} \left\| x - \operatorname{sg}\left( x - w(\tau) \left[ s_{\mathrm{fake}}(x_\tau, \tau, c) - s_{\mathrm{real}}(x_\tau, \tau, c) \right] \right) \right\|_2^2\]

Clean context chunks are masked out so gradients only backpropagate through newly generated motion latents, enabling few-step causal video generation matched to the LLM's streaming emission rate.

Loss & Training

VoCa's training follows a structured three-stage curriculum: 1. Alignment Pretraining: The LLM and speech decoder remain frozen while the video decoder is trained via Flow Matching \(\mathcal{L}_{flow}\) and auxiliary HuBERT distillation \(\mathcal{L}_{aux}\): \(\mathcal{L}_{pre} = \mathcal{L}_{flow} + \lambda_1 \mathcal{L}_{aux}\) (\(\lambda_1 = 1.0\)). 2. Joint Finetuning: Freezing only the speech decoder, the LLM and video decoder are optimized jointly. The LLM applies a delay pattern and cross-entropy loss \(\mathcal{L}_{ce}\) over discrete audio tokens, while the video decoder employs teacher forcing with chunk-level causal masking: \(\mathcal{L}_{ar} = \mathcal{L}_{flow} + \lambda_2 \mathcal{L}_{ce}\) (\(\lambda_2 = 0.25\)). 3. Distribution Matching Distillation: The student video generator is distilled into a few-step rolling causal model using the DMD loss \(\mathcal{L}_{DMD}\), freezing the LLM, speech decoder, and real-score network.

The model is trained on 64 NVIDIA H800 GPUs with a global batch size of 64, supporting resolutions up to \(832 \times 480\) and video durations of 2–7 seconds. A conditioning dropout rate of 5% (independent and joint) is applied to support classifier-free guidance.

Key Experimental Results

Main Results

Quantitative evaluations across three standard talking-head benchmarks (HDTF, CelebV-HQ, RAVDESS; 300 sampled videos each) assess Identity Consistency (IDC), Lip Synchronization (Sync-C), Spatial Fidelity (FID), and Temporal Coherence (FVD).

Method	HDTF (IDC↑ / Sync-C↑ / FID↓ / FVD↓)	CelebV-HQ (IDC↑ / Sync-C↑ / FID↓ / FVD↓)	RAVDESS (IDC↑ / Sync-C↑ / FID↓ / FVD↓)
Hallo (2024)	0.981 / 7.312 / 35.830 / 37.060	0.934 / 4.417 / 48.410 / 122.510	0.948 / 3.536 / 27.700 / 418.980
AniPortrait (2024)	0.978 / 3.607 / 36.100 / 71.270	0.938 / 1.804 / 50.000 / 110.600	0.935 / 2.364 / 23.800 / 505.040
V-Express (2024)	0.972 / 7.744 / 40.990 / 122.270	0.878 / 4.029 / 61.330 / 180.110	0.939 / 4.444 / 29.100 / 601.920
Hallo3 (2024)	0.976 / 6.898 / 35.550 / 42.850	0.930 / 4.173 / 44.750 / 91.410	0.949 / 4.948 / 24.520 / 54.330
EchoMimicV3 (2025)	0.976 / 2.573 / 35.570 / 58.380	0.930 / 2.134 / 49.880 / 92.540	0.965 / 2.494 / 15.090 / 44.360
FantasyTalking (2025)	0.969 / 3.836 / 37.360 / 46.600	0.930 / 2.797 / 47.220 / 106.810	0.952 / 3.803 / 29.350 / 57.820
HunyuanAvatar (2025)	0.970 / 7.584 / 35.570 / 106.270	0.915 / 4.494 / 46.550 / 97.800	0.957 / 5.214 / 19.760 / 99.490
VoCa-1.3B (Ours)	0.981 / 6.082 / 33.620 / 39.630	0.927 / 3.620 / 48.040 / 95.870	0.970 / 4.585 / 14.070 / 55.190
VoCa-5B (Ours)	0.982 / 8.054 / 33.020 / 36.930	0.936 / 4.769 / 44.480 / 91.170	0.970 / 5.636 / 13.160 / 48.710

(Note: Data from Table 1 of the paper. VoCa-5B attains top rankings across all metrics, noticeably outperforming specialized non-autoregressive baselines in Sync-C).

Ablation Study

The paper systematically ablates the representation design (Table 2), auxiliary loss weight \(\lambda_1\) (Table 3), and refiner convolutional depth (Table 4):

Ablation Aspect	Variant	IDC↑	Sync-C↑	FID↓	FVD↓	Note
Representation (Table 2)	Audio Tokens Only	0.970	5.799	37.066	59.961	Lacks high-level semantics; stiff non-verbal dynamics
	Hidden States Only	0.973	3.773	36.124	56.256	Lacks fine acoustic details; severe lip misalignment (-2.309 Sync-C)
	Both Tokens & Hidden (Default)	0.981	6.082	33.620	39.630	Best overall trade-off between acoustic sync and expressive motion
Auxiliary Loss (Table 3)	\(\lambda_1 = 0\) (No Distillation)	0.975	4.782	34.808	45.082	Lacks explicit phonemic guidance; substantial drop in Sync-C (-1.300)
	\(\lambda_1 = 1\) (Default)	0.981	6.082	33.620	39.630	Optimal balance between audio-visual sync and perceptual fidelity
	\(\lambda_1 = 2\) (Excessive Weight)	0.963	6.003	36.916	64.509	Over-regularization degrades visual fidelity and temporal smoothness
Refiner Depth (Table 4)	0 layers	0.973	4.754	34.996	47.177	Raw token interpolation cannot resolve acoustic stress patterns
	2 layers	0.969	5.743	35.322	54.948	Limited temporal receptive field impairs FVD
	4 layers	0.976	5.919	34.915	46.671	Steady progression towards convergence
	6 layers (Default)	0.981	6.082	33.620	39.630	Peak synchronization and image quality
	8 layers	0.982	6.054	33.506	39.269	Performance saturates with added computational cost

Key Findings

• Synergy of Tokens and Hidden States: Discrete tokens anchor micro-lip dynamics and rhythm, whereas continuous hidden states drive natural head movement and co-speech gestures. Stripping discrete tokens degrades Sync-C from 6.082 to 3.773.
• Critical Role of HuBERT Distillation: Omitting auxiliary semantic distillation (\(\lambda_1 = 0\)) results in a sharp Sync-C drop to 4.782, validating that self-supervised phonetic distillation effectively bridges the reconstruction-focused nature of audio codebooks.
• Robustness in Close-Ups and Dynamic Scenes: VoCa generates realistic fine details (beard textures, teeth alignment) under tight close-ups and avoids background bleeding under extreme environmental effects (rain, lightning, fires). In user studies (scale 0–5), VoCa achieved 4.46, substantially leading EchoMimicV3 (4.02), Hallo3 (3.64), and FantasyTalking (3.45).

Highlights & Insights

• Cascadeless Unified Autoregressive Architecture: By employing an autoregressive LLM as the shared temporal generator, VoCa unifies speech and talking video synthesis without intermediate TTS latency or cross-model error accumulation.
• Strictly Localized Audio Attention: Constraining video tokens to cross-attend strictly to their contemporaneous audio conditioning vector preserves causal autoregression while drastically lowering computational memory compared to full temporal cross-attention.
• Adapted Rolling-Window DMD Distillation: Adapting sliding-window denoising from general video models with an image-anchored dual-chunk global context allows stable, drift-free streaming talking-head generation.

Limitations & Future Work

• Single-Speaker Scope: VoCa is specifically tailored to single-speaker talking heads and does not currently accommodate multi-speaker turn-taking, overlapping speech, or crowded conversational scenes.
• Substantial Compute Footprint: Training a 3B LLM alongside 1.3B/5B diffusion transformers across three stages requires 64 H800 GPUs, presenting non-trivial reproduction hurdles for smaller research teams.
• Extreme Pose Articulation: While half-body gestures are natural, rapid full-body rotational movements can occasionally exhibit subtle perspective distortions due to single-image conditioning.

Related Work & Insights

• vs OmniTalker / Ovi: OmniTalker and Ovi utilize parallel twin-backbone diffusion architectures trained from scratch for audio and video, incurring immense compute costs while still relying on an external LLM for dialogue. VoCa integrates a foundation speech LLM natively, natively inheriting voice cloning and contextual understanding.
• vs Hallo3 / EchoMimicV3 / FantasyTalking: These state-of-the-art approaches function as audio-driven, non-autoregressive bidirectional diffusion models requiring pre-recorded or pre-synthesized audio tracks. VoCa demonstrates that an autoregressive, streaming-capable architecture can match or surpass these specialist offline models in both visual quality and synchronization.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Pioneers a cascadeless, streaming autoregressive framework unifying speech and video generation from text via shared LLM representations]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Exhaustive multi-dataset benchmarking, detailed ablations on representations, loss weights, and architecture depth, plus user studies]
• Writing Quality: ⭐⭐⭐⭐⭐ [Exceptionally clear problem framing, coherent architectural exposition, and rigorous training recipe]
• Value: ⭐⭐⭐⭐⭐ [Provides a crucial architectural blueprint for interactive digital avatars, real-time conversational agents, and streaming video synthesis]

Related Papers

• [ECCV 2026] AVTok: 1D Unified Tokenization for Holistic Audio-Video Generation
• [ECCV 2026] MMControl: Unified Multi-Modal Control for Joint Audio-Video Generation
• [ICLR 2026] JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generation
• [ECCV 2026] AR-CoPO: Align Autoregressive Video Generation with Contrastive Policy Optimization
• [ECCV 2026] Learning Transferable Dynamics Priors from Action to World Modeling