Skip to content

Live Avatar: Streaming Real-time Audio-Driven Avatar Generation with Infinite Length

Conference: ECCV 2026
Paper: ECCV Official
Code: https://liveavatar.github.io/
Area: Human Understanding / Image Restoration
Keywords: Audio-driven Avatar, Streaming Video Generation, Infinite-Length Generation, Diffusion Distillation, Pipeline Parallelism

TL;DR

Live Avatar presents an algorithm-system co-designed framework featuring noisy KV cache (History Corrupt) and Timestep-forcing Pipeline Parallelism (TPP), enabling a 14B-parameter diffusion model to perform stable, real-time streaming generation at 45.2 FPS beyond 10,000 seconds on 5 H100 GPUs.

Background & Motivation

Audio-driven avatar generation synthesizes photorealistic human face videos guided by continuous audio inputs, acting as a foundational technology for interactive digital communication, virtual live streaming, and intelligent personal assistants. Recent developments in video diffusion transformers (DiTs), such as Wan-S2V, have established unprecedented benchmarks for visual fidelity and facial expressiveness. However, deploying these powerful generative backbones in live interactive systems requires satisfying three stringent conditions simultaneously: real-time throughput (\(\ge 24\) FPS), chunk-by-chunk low-latency streaming responsiveness, and infinite-length temporal stability without visual collapse.

Current diffusion-based paradigms suffer from two fundamental tensions. The first is the conflict between long-horizon consistency and autoregressive error accumulation: existing streaming diffusion frameworks iteratively refresh their KV cache with clean generated frames, allowing subtle high-frequency imperfections to compound frame-by-frame, causing catastrophic identity drift, face distortion, or color fading within minutes. Even recent techniques like Self-Forcing fail under minute-level continuous rollouts. The second is the trade-off between real-time throughput and synthesis fidelity: large-scale 14B models achieve superior visual quality but suffer severe sequential denoising latency bottlenecks (often delivering only 0.25 FPS), whereas existing real-time systems bypass multi-step sampling by relying on lightweight 0.2B models or non-iterative face-warping methods at the expense of visual expressiveness and background realism.

Live Avatar resolves this dilemma by leveraging the static-scene prior of avatar interaction, where human appearance remains stationary relative to the background while visual dynamics are concentrated in local facial movements. This physical prior leads to a counter-intuitive insight: autoregressive history need not preserve pristine texture details, but only low-frequency motion cues. Core idea: store noisy representations in the KV cache (History Corrupt) to act as a low-pass filter that arrests error accumulation, and exploit this timestep-forcing property to build Timestep-forcing Pipeline Parallelism (TPP), turning sequential diffusion into an asynchronous spatial pipeline that sustains 45.2 FPS infinite-length generation on a 14B model.

Method

Overall Architecture

The Live Avatar framework integrates a two-stage streaming adaptation algorithm with system-level pipeline parallelism. Given an incoming audio stream and a static identity sink frame, the model processes video latents in blocks of 3 frames using block-wise causal attention. The persistent sink frame anchors global identity and appearance, while the sliding KV cache transmits motion dynamics across consecutive blocks.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Audio Stream + Static Reference Sink"] --> B["Stage 1: Motion-Frame Scaffolding Pretraining<br/>Noise-injected context decouples dynamics"]
    B --> C["Stage 2: Self-Forcing DMD Distillation<br/>Few-step causal compression to 4 steps"]
    C --> D["Causal Streaming Inference Engine"]
    D --> E["Long-Horizon Stability Pillars<br/>History Corrupt + AAS + Rolling RoPE"]
    E --> F["Timestep-forcing Pipeline Parallelism (TPP)<br/>GPU0-3 fixed timesteps + GPU4 streaming VAE decode"]
    F --> G["45.2 FPS Real-Time High-Fidelity Avatar Stream"]

Key Designs

1. Motion-Frame as Scaffold: Efficient Two-Stage Distillation Acceleration Directly performing multi-step Self-Forcing rollouts on a 14B diffusion model incurs prohibitive memory overhead and slow convergence. The authors recognize that noisy KV caches in Stage 2 serve a functionally analogous role to preceding motion frames used for clip continuation. Live Avatar therefore retains the frozen FramePack encoder from Wan-S2V during Stage 1 Diffusion Forcing pretraining, injecting independent flow-matching noise into motion context frames to teach dynamics-identity decoupling cheaply. In Stage 2, the motion frame encoder is removed and smoothly substituted with the sliding KV cache. This scaffolding mechanism delivers a \(5\times\) convergence speedup, saturating in only 500 DMD iterations compared to 2,500 steps for baseline training.

2. History Corrupt: Decoupling Motion and Identity in the Frequency Domain Conventional autoregressive video generation performs an extra forward pass to compute a clean-cache refresh after each step. In practice, this clean cache acts as the primary transmission vector for compounding errors, as the network continuously attends to fine-grained generated artifacts. Live Avatar omits clean-cache refreshing entirely, deliberately storing noisy representations in the rolling KV cache. Gaussian noise acts as a low-pass filter in representation space, suppressing accumulated high-frequency artifacts while retaining coarse-grained temporal motion trajectories. Furthermore, at elevated noise levels, marginal distributions \(p_t(x_t)\) converge toward a compact Gaussian prior, preventing long-horizon generations from drifting into out-of-distribution regions.

3. Adaptive Attention Sink and Rolling RoPE: Suppressing Distribution and Positional Drift To eliminate structural degradation during infinite-duration rollouts, the framework incorporates two auxiliary mechanisms: - Adaptive Attention Sink (AAS): Real-world reference photographs reside slightly off the generative model's learned data manifold, introducing an inductive bias that gradually causes desaturation and graying over extended generation. AAS replaces the user reference image with the model's own first-generated latent block as the permanent attention sink, eliminating the domain gap between conditioning inputs and generated video manifolds. - Rolling RoPE: In long-horizon autoregressive rollout, the relative rotary positional offset between the static sink frame and active generation blocks continuously exceeds the pretraining window. Rolling RoPE dynamically aligns the temporal coordinates of the cached sink frame to maintain a fixed relative positional distance to active blocks, preserving persistent identity retrieval without constraining local dynamics.

4. Timestep-forcing Pipeline Parallelism (TPP): Unlocking Spatial Denoising Concurrency For a 4-step distilled student model, conventional sequence parallelism offers limited throughput gains due to the short chunk length (3 frame latents). Observing that matching the noise level of active generation blocks with their attended KV cache (timestep-forcing) suppresses temporal flicker, the authors assign each GPU a dedicated denoising transition \(t_i \to t_{i-1}\). Across 4 denoising GPUs, each worker maintains its local, private noisy KV cache without inter-GPU cache synchronization, communicating only compact intermediate latents to downstream GPUs. A fifth dedicated GPU asynchronously handles streamable 3D VAE decoding with feature caching. By converting the sequential diffusion chain into an asynchronous spatial pipeline, TPP raises generation throughput to 45.2 FPS.

Loss & Training

Stage 1 is optimized with standard flow-matching velocity regression: $\(\mathcal{L}_{\text{FM}} = \mathbb{E}_{x_0, t} \left\| v_\theta(x_t, t, c) - (x_1 - x_0) \right\|_2^2\)$ In Stage 2, the student generator is trained under Distribution Matching Distillation (DMD) with score gradients: $\(\nabla_\theta \mathcal{L}_{\text{DMD}} = \mathbb{E}_{t, z} \left[ s_{\text{real}}(x_t) - s_{\text{fake}, \phi}(x_t) \right] \frac{\partial \hat{x}}{\partial \theta}\)$ Both the real-data score function \(s_{\text{real}}\) and the fake-data score estimator \(s_{\text{fake}, \phi}\) are initialized from the 14B Wan-S2V checkpoint and alternately updated along intermediate states sampled along the 4-step generation trajectory.

Key Experimental Results

Main Results

Evaluation is conducted on GenBench, comprising GenBench-ShortVideo (100 samples, ~10 seconds) and GenBench-LongVideo (15 test samples exceeding 5 minutes, covering diverse real, animated, and anthropomorphic portraits). Evaluation metrics include perceptual quality (IQA), aesthetic appeal (ASE), lip synchronization (Sync-C / Sync-D), identity consistency (Dino-S), and generation frame rate (FPS).

Dataset Model ASE โ†‘ IQA โ†‘ Sync-C โ†‘ Sync-D โ†“ Dino-S โ†‘ FPS โ†‘
GenBench-ShortVideo Ditto [23] 3.31 4.24 4.09 10.76 0.99 21.80
Echomimic-V2 [32] 2.82 3.61 5.57 9.13 0.79 0.53
Hallo3 [6] 3.12 3.97 4.74 10.19 0.94 0.26
StableAvatar [40] 3.52 4.47 3.42 11.33 0.93 0.64
OmniAvatar [12] 3.53 4.49 6.77 8.22 0.95 0.16
WanS2V [13] (Teacher) 3.36 4.29 5.89 9.08 0.95 0.25
Live Avatar (Ours) 3.44 4.51 7.03 8.30 0.96 45.20
GenBench-LongVideo Ditto [23] 2.90 4.48 3.98 10.57 0.97 21.80
Hallo3 [6] 2.65 4.04 6.18 9.29 0.83 0.26
StableAvatar [40] 3.00 4.66 1.97 13.57 0.94 0.64
OmniAvatar [12] 2.36 2.86 8.00 7.59 0.66 0.16
WanS2V [13] 2.63 3.99 6.04 9.12 0.80 0.25
Live Avatar (Ours) 3.42 4.76 7.16 8.31 0.97 45.20

Ablation Study

1. Inference Efficiency and Parallel Scaling (Table 3 in paper)

Configuration # GPUs NFE FPS โ†‘ TTFF (s) โ†“ Notes
WanS2V Baseline 1 80 0.29 45.50 Standard 80-step sequential sampling
+ DMD 4-Step Distillation 1 5 3.66 4.56 4 denoising steps + 1 clean-cache pass
+ Sequence Parallelism (SP4) 4 5 4.50 3.94 Short block sequences limit parallel scaling
+ TPP (Ours) 4 4 10.16 4.73 Eliminates clean-cache pass (NFE 5โ†’4)
+ VAE Asynchronous Offload 5 4 20.88 2.89 Decouples compute-heavy decoding bottleneck
+ Kernel Optimizations (Full System) 5 4 45.20 1.21 FP8, FlashAttention-3, torch.compile, fused kernels

2. Long-Horizon Stability Mechanisms on GenBench-LongVideo (Table 4 in paper)

Configuration ASE โ†‘ IQA โ†‘ Sync-C โ†‘ Dino-S โ†‘ Observed Degradation Mode
Full Model (Ours) 3.42 4.76 7.16 0.97 Stable fidelity beyond thousands of seconds
w/o History Corrupt (Clean Cache) 2.90 3.88 7.14 0.81 Rapid compounding errors; facial collapse in minutes
w/o Rolling RoPE 3.38 4.82 7.29 0.86 Positional extrapolation failure; identity drift
w/o AAS (Persistent Input Image) 3.13 4.68 7.25 0.96 Manifold gap accumulation; progressive desaturation

Key Findings

  • Noisy Caching Resolves Long-Term Drift: Removing History Corrupt drops identity consistency (Dino-S) from 0.97 to 0.81 and perceptual quality (IQA) from 4.76 to 3.88 on long videos, corroborating that clean KV cache refreshing directly drives recursive artifact compounding.
  • Seamless Train-Short-Infer-Long Generalization: While trained exclusively on ~3-second clips (84 frames), Live Avatar extrapolates beyond 10,000 seconds without quality decay, maintaining nearly identical metrics between short and long benchmarks (Dino-S 0.96 vs 0.97, ASE 3.44 vs 3.42).
  • Timestep-Forcing Suppresses Temporal Flicker: Matching the noise level of current blocks with their attended KV cache improves temporal consistency (T.Flicker) from 0.876 (clean cache) and 0.891 (fixed noise) to 0.971 (Table 6), while establishing the foundational prerequisite for per-device pipeline isolation.

Highlights & Insights

  • The Counter-Intuitive Power of Lossy Caching: Conventional wisdom preserves pristine, high-fidelity representations in KV caches. By recognizing that interactive avatar scenes possess a strong static-scene prior, this work shows that injecting noise acts as a natural low-pass filter, decoupling high-frequency identity from dynamic motion.
  • Algorithm-System Synergy: Timestep-forcing serves dual roles: it removes temporal visual flicker algorithmically, and enables zero-communication local KV cache persistence per GPU, translating sequential sampling into high-throughput hardware pipelining.
  • Scaffolded Diffusion Distillation: Leveraging motion frames as a low-cost proxy for future KV caches accelerates DMD training on 14B models by \(5\times\), offering a general paradigm for distilling recurrent generative models.

Limitations & Future Work

  • Static Scene Dependency: The method relies on the assumption of static backgrounds and stationary camera poses common in talking avatars. Generalizing to dynamic panning, complex embodied movements, or multi-subject interactions remains an open challenge.
  • End-to-End Latency Overhead: Although generation throughput hits 45.2 FPS, initial frame generation requires a TTFF of 1.21 seconds, resulting in total round-trip latencies near 3 seconds when combined with network streaming.
  • vs OmniAvatar / Wan-S2V: While OmniAvatar and Wan-S2V produce expressive short clips, both degrade sharply on long videos (Dino-S falls to 0.66 and 0.80) and run at under 0.3 FPS. Live Avatar maintains stable identity while delivering a \(180\times\) speedup.
  • vs Ditto: Ditto achieves 21.8 FPS via 0.2B face warping, but exhibits poor audio-visual synchronization (Sync-C 3.98) and lacks full-frame diffusion synthesis capabilities.
  • vs Self-Forcing / CausVid: Prior autoregressive distillation models require clean-cache refresh passes that inflate NFE and compound long-term exposure bias. Live Avatar completely removes refresh passes, cutting inference NFE from 5 to 4 while securing unbounded temporal stability.

Rating

  • Novelty: โญโญโญโญโญ (Elegant low-pass filtering insight on noisy KV cache paired with timestep-forcing pipeline parallelism)
  • Experimental Thoroughness: โญโญโญโญโญ (Exemplary 5-minute GenBench benchmark, comprehensive system ablations, and 10,000-second stability stress tests)
  • Writing Quality: โญโญโญโญโญ (Clean technical exposition, precise failure-mode taxonomy, and transparent system profiling)
  • Value: โญโญโญโญโญ (First milestone bringing 14B-parameter video diffusion into practical real-time streaming, offering blueprint for interactive avatar systems)