InteractiveAvatar: Real-Time Streaming Video Generation for Consistent and Intent-Aware Avatars

Conference: ECCV 2026
Paper: ECCV paper page
Authors: Quanyue Song, Yishan He, Yanfei Zhang, Shihao Cheng, Zhixiang He, Zhizhi Guo, Chi Zhang, Xuelong Li, Caigui Jiang
Affiliations: Xi’an Jiaotong University; China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd.; Wuhan University; Institute of Artificial Intelligence (TeleAI), China Telecom
Area: Video Generation
Keywords: audio-driven avatars, streaming video generation, long-short visual memory, intent awareness, distribution matching distillation

The title and author list follow the conference PDF; Quanyue Song and Yishan He contributed equally as first authors. The task is interactive human video synthesis, not 3D avatar reconstruction.

TL;DR

InteractiveAvatar integrates long-short visual memory and intent-driven action control into an autoregressive diffusion avatar model, achieving 26.68 FPS under a 576p experimental setting with improved object persistence and text alignment, although the complete interaction pipeline still takes approximately 2.6 seconds to deliver its first frame.

Background & Motivation

Audio-driven avatars can already produce clear faces and reasonably accurate lip movements, but convincing speech does not imply meaningful interaction. When response audio alone drives body motion, the model largely reproduces correlations between speech and pose in its training data. Asked to check a watch, an avatar may answer the time without explicitly looking at the watch.

Meanwhile, autoregressive video generators typically attend to a reference image and a few recent chunks. As a conversation introduces a watch, a book, or a new pose, the current scene departs from the initial reference. Recent context is continually evicted, allowing object shape, color, or even presence to drift. Retaining all history would provide more conditioning information but conflicts with the computational budget of continuous streaming.

The paper separates behavior management from visual-history management: a language model maintains what to do and what state to preserve afterward, while the video model maintains recent motion and earlier visual content. Core Idea: preserve scene changes through semantically diverse long-short visual memory, then use action/stable-state cycling and conditioned-cache refreshes to make changing user intent affect the ongoing video.

Method

Overall Architecture

The inputs are an avatar reference image and user speech; the output is a continuous video synchronized with response audio and performing the requested actions. Automatic speech recognition first produces user text. The Reasoning-Reaction Module (RRM) combines that text with the current stable state to produce response and action conditions, while a generator equipped with Long-Short Visual Memory (LSVM) continues the video chunk by chunk.

The backbone is Wan 2.2 5B, with a causal 3D VAE for video-latent encoding and decoding. T5 encodes text for cross-attention; the architecture figure also shows Wav2vec audio conditioning and audio cross-attention. The reference image, adjacent chunks, and compressed history jointly support avatar continuity.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    INPUT["Reference image and user speech"] --> ASR["Speech recognition"]
    ASR --> STATE["Intent-Driven State Cycling"]
    STATE --> CACHE["Conditioned Cache Switching"]
    CACHE --> MEMORY["Long-Short Visual Memory"]
    MEMORY --> OUTPUT["Chunk denoising and decoding<br/>Output video"]
    OUTPUT -->|New frames update memory| MEMORY
    STATE -->|Stable state informs next turn| STATE

The three designs in the diagram control online inference; the generator acquires real-time capabilities through the four training stages described below. Cache switching does not regenerate the entire history. It refreshes conditioned KV representations of affected adjacent chunks when a new prompt arrives, preventing subsequent denoising from relying on stale instructions.

Key Designs

1. Intent-Driven State Cycling: separate an action from the state to maintain afterward

RRM produces more than a verbal reply. It gives the recognized user text and the previous stable-state description to a large language model, obtaining an action prompt, response audio, and a new stable-state prompt. For example, sitting on a sofa and using a phone requires both an action state describing the transition and a stable state describing the person already seated with the phone. This stable state supplies scene context for the next reasoning turn; it is not a replacement for the complete visual history.

While response audio is active, the video generator receives both the action prompt and the audio condition: the former guides body motion, while the latter drives lip synchronization. After the audio finishes, the audio condition becomes empty and the prompt switches to the stable state. Video generation continues without repeatedly executing sitting down or picking up an object. A new instruction starts the next action state. The transition follows the end of response audio, not a visually verified action-completion detector.

2. Conditioned Cache Switching: propagate new instructions into an established autoregressive context

KV caching avoids repeated computation, but the adjacent chunks' KV representations were computed under an earlier prompt. Merely changing the current text condition while keeping that cache can leave the new action constrained by stale context, delaying motions such as picking up and opening a book. The issue is not necessarily missing language understanding; the updated understanding has not fully reached the cached representations.

When the action prompt changes, Cache-Switching re-encodes the new text and recomputes affected conditioned KV tensors for previously adjacent chunks, replacing the corresponding cache entries. It preserves the historical role of generated frames while updating their conditioned representations. Its main benefit concerns action responsiveness after prompt changes, which cannot be measured through aggregate FPS alone. The ablation mainly demonstrates delayed action onset qualitatively and does not provide a per-configuration first-frame latency table.

3. Long-Short Visual Memory: retain dense recent context and semantically nonredundant distant frames

LSVM first converts video latent frames into memory tokens through a lightweight convolution-and-attention compression module. Its temporal compression ratio is 1, avoiding additional merging along the latent-frame time dimension and allowing individual latent-frame cache updates. This does not imply that the VAE itself has no temporal compression. A FIFO queue of capacity \(K\) holds recent history, while a long-term buffer of capacity \(N\) retains representative states. Their concatenation forms the history condition. During training, the short-term source spans 5 consecutive seconds, with earlier history providing long-term memory.

Both buffers are initialized by repeating the first-frame representation. Each new latent frame is compressed and added to the short-term queue; only the evicted latent frame becomes a long-term candidate. The system takes its corresponding image frames, extracts SigLIP2 features, and averages them into a semantic descriptor. Selection therefore compares semantic image content rather than pixelwise errors between compressed tokens.

A global redundancy score determines whether long-term memory accepts a candidate. The following equivalent notation follows the textual definition of original Eq. (10), with \(s_i\) denoting the semantic vector of long-term entry \(i\):

\[ \rho_i=\frac{1}{N-1}\sum_{j\ne i}\cos(s_i,s_j), \qquad R=\frac{1}{N}\sum_{i=1}^{N}\rho_i. \]

If replacing a slot with the candidate reduces redundancy to \(R'<R\), the candidate is retained and inserted chronologically; otherwise it is discarded. Dynamic Key-Frame Selection (DKFS) thus favors novel content instead of mechanically sampling at uniform intervals. A newly introduced watch may add semantic information, but this is not explicit object tracking or watch detection: retaining a small important object still depends on global semantic features and finite capacity.

A Worked Example

Consider the interaction in the paper's Figure 1: the user asks the avatar to put on a watch and report the time. RRM produces an action state for putting it on, looking at it, and answering. Relevant KV entries are refreshed when the prompt changes, and the generator produces motion and lip movements during the response audio. Once the reply ends, the stable state describes calmly facing forward while wearing the watch, instead of repeatedly checking it.

The user then asks for a cat to appear on the avatar's shoulder and requests information about cat breeds. The new action and reply begin. The previous stable state tells the language side that the avatar still wears a watch; recent visual memory supports the shoulder-motion transition, while frames leaving the short-term window undergo long-term selection. The two memory paths preserve scene descriptions and visual evidence, respectively, but neither guarantees permanent object persistence.

Loss & Training

Training does not simply convert a bidirectional diffusion model into an online system. It successively establishes audio-driven generation, memory-based reconstruction, causal initialization, and few-step autoregressive generation.

1. Stage 1 trains a bidirectional image-and-audio-conditioned video model on audiovisual data for 50K steps.
2. Stage 2 trains the memory compression module through video reconstruction for 30K steps. The recent 5 consecutive seconds supply short-term memory, and earlier history is randomly sampled for long-term memory.
3. Stage 3 performs ODE initialization, using block-causal attention to approximate the bidirectional teacher's ODE trajectories while optimizing memory compression, for 20K steps.
4. Stage 4 applies Self-Forcing DMD to distill the model with LSVM into a few-step autoregressive generator, for 20K steps.

Memory pretraining samples reconstruction targets at different temporal positions in a full segment. In the short-term range, a target-related subset is randomly preserved and other frames are noise-masked. In the long-term range, the temporally closest anchor is retained and the remaining frames are masked. Clean target copies supervise reconstruction of noisy targets conditioned on compressed history, encouraging both recent detail and persistent scene content to survive compression.

DMD aligns the distributions of student-generated samples and teacher-represented data at intermediate noise levels. Following the definition in original Eq. (2), its objective is:

\[ \mathcal{L}_{\mathrm{DMD}}=\mathbb{E}_t\left[D_{\mathrm{KL}}\left(p_{\theta,t}\,\|\,p_{\mathrm{data},t}\right)\right]. \]

Training alternates between the fake-score branch and the generator, using random intermediate states from student rollouts for multi-step distillation. All training uses 64 NVIDIA H100 GPUs and hybrid-sharding FSDP. The student learning rate is \(10^{-5}\) and the fake-score learning rate is \(2\times10^{-6}\). The main text does not specify the inference denoising-step count or long-term buffer capacity, so neither is assumed here.

Key Experimental Results

Main Results

The filtered training set contains approximately 3 million audiovisual clips: talking-head data, movie and television data primarily from OpenHumanVid, and a proprietary long-duration conversation subset. Testing uses 500 videos covering short-to-long and simple-to-complex interactions, with action changes at designated times.

Training and inference both use 576p, such as 1024×576 at 16:9. Each streaming chunk contains 3 latent frames, not 3 RGB frames. Non-real-time baselines construct long videos through batch continuation. Table 1 specifies H100 hardware, but deployment also places DiT and VAE on different GPUs for pipeline parallelism; the FPS results should therefore not be presented as verified single-GPU end-to-end measurements.

The following selection comes from original Table 1. Lower FVD is better; all other listed metrics favor higher values. OBJ measures object consistency with Gemini, ID measures identity preservation with DINOv3, and TV measures text-video alignment with VideoCLIP-XLv2. Their original score scales are retained rather than relabeled as percentages.

Method	FVD	OBJ	ID	TV	Speed (FPS)
OmniAvatar	831.9	82.8	4.38	24.57	0.17
HYAvatar	632.6	78.9	4.46	25.61	0.09
WanS2V	793.5	82.6	4.49	25.14	0.26
LiveAvatar	672.7	76.9	4.53	25.78	21.94
InteractiveAvatar	701.4	85.2	4.51	25.93	26.68

The complete interaction pipeline takes approximately 2.6 seconds to deliver its first frame: about 450 ms for DiT, 50 ms for lightweight VAE decoding, 1600 ms for the LLM audio response, and 450 ms for other overhead. These approximate components sum to about 2550 ms. First-frame responsiveness and sustained generation FPS measure different properties.

Ablation Study

The following results are from original Table 2, covering visual memory, RRM, and distillation. Removing DKFS replaces dynamic selection with random sampling; removing StateCycling uses a fixed action prompt; removing RRM falls back to a default prompt.

Config	OBJ	ID	TV	Speed (FPS)
w/o LongMem	82.6	4.43	25.91	28.92
w/o DKFS	83.1	4.45	25.85	26.83
w/o LSVM	78.4	4.38	25.87	30.04
w/o StateCycling	84.1	4.46	25.42	26.68
w/o CacheSwitching	84.5	4.47	25.76	26.75
w/o RRM	83.8	4.46	24.89	26.75
w/o DMD	Not reported	Not reported	Not reported	1.27
Full model	85.2	4.51	25.93	26.68

Key Findings

• Relative to LiveAvatar, OBJ improves by 8.3 points and speed by 4.74 FPS. However, ID falls from 4.53 to 4.51 and FVD rises from 672.7 to 701.4, so the method does not dominate every metric.
• Removing LSVM reduces OBJ by 6.8 points while increasing speed by 3.36 FPS, exposing the real computational cost of maintaining visual history.
• Removing RRM reduces TV from 25.93 to 24.89. Removing state cycling also causes repeated actions, consistent with its role in controlling action semantics and interaction timing.
• Without distillation, speed is only 1.27 FPS and stable long-video generation fails. The other metrics in that row are unreported, not zero.

Highlights & Insights

• Visual memory and linguistic state have distinct roles. Text remembers that a watch has been put on, while compressed visual history supplies its appearance; both support continuous interaction.
• A cache is not merely a speed optimization: it retains the influence of earlier conditions. Interactive autoregressive systems should co-design condition changes and cache refreshes.
• DKFS seeks semantic diversity within fixed capacity rather than uniform temporal coverage. This can inform other long-video conditioning caches, provided small-object retention is tested against global-feature blind spots.

Limitations & Future Work

• The authors note that finite memory and key-frame selection discard earlier latents. Long videos or large motions departing from the reference can still degrade, and objects can gradually disappear.
• Objects absent from the first frame can appear abruptly when requested. Prompt design only partly mitigates this behavior, which also depends on the base generator.
• From an evaluation perspective, the main text does not fully detail scoring protocols for metrics such as OBJ, uncertainty intervals, or a standardized long-video duration distribution. Indefinite streaming operation is not evidence of indefinite quality preservation.
• From an interaction perspective, the LLM audio response accounts for a substantial part of the approximately 2.6-second first-frame delay. Improving video FPS alone cannot eliminate waiting. Switching states at audio completion also motivates testing visual action-completion feedback.

Related Work & Insights

• vs LiveAvatar: both pursue streaming avatars. InteractiveAvatar is faster and more object-consistent in the reported setting, but does not simultaneously win on identity score and FVD.
• vs OmniAvatar / WanS2V: InteractiveAvatar explicitly produces action instructions and manages a stable state rather than relying only on speech-motion correlations. Long-video comparisons also involve batch continuation for non-real-time models.
• vs Self-Forcing / CausVid: few-step causal generation and distribution matching distillation supply the real-time foundation. This paper additionally addresses visual-memory updates and changing action conditions during conversation.

Rating

• Novelty: 4/5. Visual memory, linguistic state, and conditioned-cache refreshes form a combination tailored to streaming interaction.
• Experimental Thoroughness: 3/5. Main comparisons and component ablations are substantial, but evaluation details, duration distributions, and latency ablations remain limited.
• Writing Quality: 4/5. The method is clearly organized, while some implementation hyperparameters and scoring scales need further specification.
• Value: 4/5. The system demonstrates a practical route from lip synchronization toward sustained avatar interaction with explicit action semantics.