Towards Flexible, Natural, Efficient Interaction for Conversational Talking Face Generation¶
Conference: ECCV 2026
Paper: ECCV Official
Area: Human Understanding
Keywords: Conversational Talking Face, Non-verbal Feedback, Multi-person Interaction, Disentangled Motion Refinement, Real-time Rendering
TL;DR¶
InterTalk establishes a motion-driven conversational talking face framework that integrates a Responsive Context Encoder, disentangled motion refinement, and an iterative generation strategy, eliminating participant count restrictions and delivering high-fidelity, real-time (30+ FPS) multi-party dialogue animations.
Background & Motivation¶
Conversational talking face generation aims to synthesize interactive video streams where virtual characters speak, listen, and dynamically respond to each other with convincing non-verbal cues. Unlike conventional single-speaker talking head synthesis, conversational scenarios impose strict requirements on multi-round behavioral coordination: speakers must maintain sharp audio-lip synchrony, listeners should exhibit timely, context-sensitive non-verbal feedback such as head nods or subtle smiles, and conversational roles must switch seamlessly. Deploying such systems in practical applications—such as companion digital avatars, teleconferencing, and interactive live-streaming—requires satisfying three fundamental criteria simultaneously: flexibility (supporting multi-turn dialogues with an arbitrary number of participants), naturalness (accurate lip-sync and lifelike non-verbal responsiveness), and efficiency (low-latency real-time generation during continuous streaming).
Existing paradigms struggle to reconcile these three dimensions. Explicit motion-modeling approaches offer high computational throughput, but early attempts (e.g., DIM) rely on separate speaking and listening models that require manual, hard-coded role switching, hindering fluent multi-turn conversations. Subsequent models (e.g., INFP) introduce dual-track audio inputs but remain confined to single-person setups without modeling visual feedback from partners, resulting in isolated movements; furthermore, limited conversational training data constrains their lip-sync precision. On the other end of the spectrum, end-to-end video diffusion backbones (such as MultiTalk built upon Wan 2.1) produce visually appealing two-person conversations, but their immense computational demands (generating at sub-1 FPS) and strict binding constraints make them impractical for real-time applications or arbitrary group conversations.
This paper tackles these challenges by explicitly fusing multi-modal, multi-participant context into a unified interactive latent space and disentangling facial motion into specialized sub-regions driven by lightweight implicit keypoints. Core idea: construct InterTalk, an efficient motion-driven framework that leverages a Responsive Context Encoder to aggregate multi-track audio and motion feedback, applies disentangled motion refinement with single-speaker sync priors, and employs an iterative generation strategy to achieve responsive, natural interactions among arbitrary participants at 30+ FPS.
Method¶
Overall Architecture¶
The InterTalk framework comprises three coordinated components: the Responsive Context Encoder (RCE), the Interactive Motion Generator (IMG), and the Rendering Pipeline. The RCE ingests audio streams and prior motion feedback from all participants in the scene, using cross-modal alignment and adaptive attention pooling to construct a shared global interactive representation. Conditioned on this context and the self-audio stream, the IMG produces coarse facial motions and decouples them into lip, eye, and head-pose subspaces for targeted refinement. Finally, the rendering pipeline extracts 3D implicit keypoints, warps identity features via an appearance decoder, and composites individual facial animations back into the shared scene using a spatial stitching module.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multi-participant Multi-track Audio & Motion Inputs"] --> B["Responsive Context Encoder: Cross-modal Fusion & Adaptive Pooling"]
B --> C["Interactive Motion Generator: Causal Temporal Dynamics & Coarse Prediction"]
C --> D["Disentangled Motion Refinement: Specialized Lip-sync & Gaze Alignment"]
D --> E["Iterative Generation Strategy: Progressive Mutual Feedback Refinement"]
E --> F["Implicit Keypoint Warping & Spatial Multi-person Stitching"]
F --> G["Coherent Multi-participant Conversational Video"]
Key Designs¶
1. Responsive Context Encoder: Unified Multi-Participant Cross-Modal Dynamics Previous conversational systems rely almost exclusively on audio or isolated acoustic tracks, completely neglecting real-time non-verbal motion feedback from interlocutors. The RCE addresses this by ingesting both audio \(A_i\) and motion \(M_i\) from each participant \(i\), where \(M_i\) is explicitly decomposed into lip \(M_i^{\text{lip}}\), eye \(M_i^{\text{eye}}\), and head pose \(M_i^{\text{pose}}\). Speech is encoded via Wav2Vec 2.0 into \(F_{\text{a}}^i\), while the motion components are merged into \(F_{\text{m}}^i\). A cross-attention layer aligns audio-visual semantics, followed by a bi-directional LSTM covering a 3-second temporal window to capture short-term conversational context: $\(F_{\text{RCE}}^i = \text{biLSTM}\left(\text{CrossAttn}(F_{\text{a}}^i, F_{\text{m}}^i)\right)\)$ To flexibly scale to an arbitrary number of participants \(N\), the RCE computes a mean reference feature \(\bar{F} = \frac{1}{N}\sum_{i=1}^N F_{\text{RCE}}^i\) and derives dynamic attention weights via scaled dot-product similarity: \(\alpha_i = \text{Softmax}\left(\frac{{F_{\text{RCE}}^i}^\top \bar{F}}{\sqrt{d}}\right)\), yielding a consolidated interactive context feature \(F_{\text{RCE}} = \sum_{i=1}^N \alpha_i F_{\text{RCE}}^i\). This structure allows the system to flexibly adapt whether there are two participants or an entire group meeting.
2. Disentangled Motion Refinement: Specialized Subspaces with Single-Speaker Priors Directly predicting entangled full-face motion often degrades lip articulation and yields unnatural blinks, especially given the scarcity of high-resolution conversational datasets. The IMG decomposes motion generation into sequential stages: a fused Transformer encoder integrates \(F_{\text{RCE}}\) and the self-audio feature \(F_{\text{a}}\), a causally masked temporal alignment attention prevents future-frame leakage, a Transformer decoder tracks conversational transitions, and a feed-forward projection separates coarse motions \(M_{\text{coarse}}\) into lip, eye, and head-pose streams. Each stream undergoes dedicated refinement: $\(M_{\text{refine}}^{\text{eye}} = \text{BlinkEnhancer}(M_{\text{coarse}}^{\text{eye}}, z, f), \quad M_{\text{refine}}^{\text{lip}} = \text{SyncEnhancer}(M_{\text{coarse}}^{\text{lip}}, A_{\text{self}}), \quad M_{\text{refine}}^{\text{pose}} = \text{PoseInitializer}(M_{\text{coarse}}^{\text{pose}}, x_{\text{self}}, y_{\text{self}})\)$ Crucially, the SyncEnhancer is trained independently on a massive single-speaker talking-face dataset to predict fine-grained mouth deformations, injecting strong phoneme-viseme alignment priors without interfering with global head dynamics. The BlinkEnhancer introduces physiological blink frequency and noise regulation, while the PoseInitializer adjusts initial head orientations based on relative spatial coordinates \((x_{\text{self}}, y_{\text{self}})\) to ensure natural mutual gaze.
3. Iterative Generation Strategy: Low-Overhead Closed-Loop Feedback During live multi-person inference, generating participant motions sequentially introduces an unavoidable lag: earlier characters cannot access the contemporaneous feedback of characters generated later in the queue. InterTalk overcomes this with a progressive iterative generation strategy. In cycle \(j=0\), interactive motions and features are initialized to zero. Across successive cycles \(j \in \{1, \dots, K\}\) (typically \(K=2\sim 3\)), the IMG updates each participant's motion using the full set of motion representations \(M_{(j-1)}^{\neg i}\) generated in the preceding cycle: $\(M_{ij} = \text{IMG}\left(A_i, \text{RCE}\left(M_{(j-1)}^{\neg i}, A_{(j-1)}^{\neg i}\right)\right)\)$ Because the IMG operates strictly on compact motion representations rather than raw pixel tensors, executing 2 to 3 iterations introduces negligible computational overhead (less than 5% of overall pipeline runtime), while effectively eliminating response latency and synchronizing listener reactions with speaker pauses.
4. 3D Face-based Data Augmentation & Spatial Multi-Person Stitching To resolve the critical bottleneck of limited multi-person conversational video, the authors collected an in-the-wild dataset spanning 143 identities across 205 multi-speaker clips, applying an audio-visual separation network to isolate individual tracks. Furthermore, to exploit existing 3D conversation corpora (e.g., DualTalk) that only supply FLAME mesh coefficients, InterTalk introduces a 3D-to-2D motion converter. Given FLAME parameters \((\alpha_{\text{exp}}, \alpha_{\text{jaw}}, \alpha_{\text{pose}})\), vertex offsets \(\Delta = \text{FLAME}(\alpha_{\text{exp}}, \alpha_{\text{jaw}}, \alpha_{\text{other}}) - T\) are masked into localized eye and lip regions and projected into 2D implicit keypoint motion features. At render time, canonical keypoints \(K_{\text{ori}}\) are perturbed by driven transformations \(K_{\text{d}}\) to warp identity embeddings, and a spatial stitching module refines local boundaries with blending masks to composite participants into coherent group scenes.
Loss & Training¶
Training proceeds in two stages. In the primary stage, the RCE and IMG motion backbone are trained jointly using an \(L_2\) motion reconstruction loss \(\mathcal{L}_{\text{rec}}\) and a velocity smoothness loss \(\mathcal{L}_{\text{vel}}\): $\(\mathcal{L}_{\text{motion}} = \lambda_{\text{rec}} \mathcal{L}_{\text{rec}} + \lambda_{\text{vel}} \mathcal{L}_{\text{vel}}\)$ In the secondary stage, the motion backbone is frozen while the lightweight SyncEnhancer is fine-tuned on the high-definition HDTF dataset using a pre-trained SyncNet loss \(\mathcal{L}_{\text{sync}}\) to enforce crisp lip-audio synchronization. The rendering module inherits pre-trained weights from LivePortrait. Training is conducted on 8 NVIDIA RTX 4090 GPUs for 3 days using the Adam optimizer with a learning rate of \(1 \times 10^{-4}\).
Key Experimental Results¶
Main Results¶
The framework is evaluated across single-speaker talking head synthesis (HDTF), responsive listener head generation (ViCo), and multi-person conversational synthesis.
On the single-speaker HDTF benchmark, InterTalk achieves superior image fidelity (FID / NIQE), lip synchrony (Sync-C / Sync-D), and temporal consistency (FVD):
| Dataset | Metric | Ours | Prev. SOTA (Sonic / EchoMimic) | Gain |
|---|---|---|---|---|
| HDTF | Sync-C ↑ | 8.30 | 8.26 (Sonic) | +0.04 |
| HDTF | Sync-D ↓ | 6.91 | 6.88 (Sonic) | +0.03 (comparable) |
| HDTF | FID ↓ | 23.07 | 28.13 (EchoMimic) / 34.41 (Sonic) | -5.06 (sharper realism) |
| HDTF | NIQE ↓ | 13.29 | 13.42 (EchoMimic) / 13.88 (Sonic) | -0.13 |
| HDTF | FVD ↓ | 178.73 | 204.55 (Sonic) / 284.38 (EchoMimic) | -25.82 (smoother motion) |
On the responsive listener benchmark ViCo, InterTalk demonstrates state-of-the-art motion distribution alignment (FD), expressiveness (SID), and pose accuracy:
| Dataset | Evaluation Slice | Ours | DIM (ECCV'24) | INFP (CVPR'25) | ARIG (2025) |
|---|---|---|---|---|---|
| ViCo | FD (exp / pose) ↓ | 18.33 / 0.04 | 23.88 / 0.06 | 18.63 / 0.07 | 18.39 / 0.05 |
| ViCo | SID (exp / pose) ↑ | 4.85 / 4.07 | 3.71 / 2.35 | 4.78 / 3.92 | 4.82 / 3.94 |
| ViCo | RPCC (exp / pose) ↓ | 0.05 / 0.01 | 0.06 / 0.03 | - / - | 0.05 / 0.01 |
| ViCo | MSE (exp / pose) ↓ | 0.48 / 0.01 | 0.70 / 0.02 | 0.51 / 0.01 | - / - |
On the newly introduced multi-person conversational dataset, InterTalk is benchmarked against MultiTalk (built on the Wan 2.1 video diffusion backbone):
| Method | Paradigm | Sync-C ↑ | FD ↓ | RPCC ↓ | SID ↑ | FPS ↑ |
|---|---|---|---|---|---|---|
| MultiTalk | Video Diffusion (Wan 2.1) | 7.94 | 41.74 | 0.24 | 5.14 | 0.61 |
| InterTalk (Ours) | Explicit Motion-Driven | 8.30 | 22.67 | 0.09 | 4.28 | 31.23 |
Ablation Study¶
A comprehensive ablation study on the multi-person dataset validates each core architectural decision:
| Config | Sync-C ↑ | FD ↓ | RPCC ↓ | SID ↑ | FPS ↑ | Note |
|---|---|---|---|---|---|---|
| Full model | 8.30 | 22.67 | 0.09 | 4.28 | 31.23 | Best overall trade-off in realism, responsiveness, and speed |
| w/o Motion Feedback in RCE | 8.13 | 29.72 | 0.15 | 2.71 | 31.85 | Drastic drop in response diversity (SID drops from 4.28 to 2.71) |
| w/o Iterative Generation Strategy | 8.25 | 24.26 | 0.11 | 4.12 | 34.67 | Noticeable increase in FD (22.67 to 24.26) due to feedback lag |
| w/o Disentanglement on Motion | 7.58 | 25.17 | 0.11 | 3.85 | 31.56 | Severe decline in lip-sync accuracy (Sync-C plunges to 7.58) |
| w/o 3D Face Data Augmentation | 7.01 | 38.67 | 0.20 | 1.95 | 31.23 | Absence of 3D motion supervision causes near collapse (FD jumps to 38.67) |
Key Findings¶
- 3D Data Augmentation is Indispensable: Removing the 3D FLAME-based data augmentation causes the motion distribution distance (FD) to surge from 22.67 to 38.67 and slashes motion diversity (SID) by over 54% (from 4.28 to 1.95). This proves that 2D conversational video alone lacks sufficient geometric coverage, and 3D parameter conversion provides essential motion regularization.
- Motion Disentanglement Rescues Lip Synchrony: Disabling motion refinement reduces Sync-C from 8.30 to 7.58. Leveraging a dedicated SyncEnhancer trained on single-speaker speech allows the system to achieve state-of-the-art lip articulation without compromising nuanced non-verbal listener gestures.
- Iterative Inference Reaps Feedback Gains at Minimal Cost: As demonstrated in iteration analysis, increasing iteration steps progressively minimizes FD and increases SID, saturating around \(K=2\sim 3\). Crucially, inference speed only drops marginally from 34.67 to 31.23 FPS, remaining well above the 30 FPS real-time threshold.
Highlights & Insights¶
- Decoupled Motion Modeling vs. Heavy Video Diffusion: Rather than forcing large-scale video diffusion models (which bottleneck generation at 0.61 FPS) to solve appearance, synchronization, and interaction simultaneously, InterTalk resolves interaction dynamics in a compact latent motion space and offloads rendering to efficient implicit keypoints, achieving a 50× speedup (31.23 FPS).
- Cross-Domain 3D-to-2D Knowledge Transfer: The conversion of 3D FLAME coefficients into 2D implicit keypoint motion masks serves as a highly reusable blueprint for bridging 3D animation datasets with 2D portrait synthesis pipelines.
- Participant-Agnostic Adaptive Architecture: Through the RCE's adaptive reference pooling, the system gracefully scales from single-avatar human-agent interaction to arbitrary multi-party group dialogues without structural retraining or role-count limitations.
Limitations & Future Work¶
- Author-Acknowledged Limitations: Large out-of-plane head rotations (exceeding 60 degrees) and heavy hand-to-face occlusions can induce local warping artifacts in 2D implicit keypoint rendering. In extended unvoiced listening sequences, subtle periodic gestures may occasionally exhibit minor repetition.
- Reviewer-Identified Limitations: The rendering and stitching pipeline relies on static background framing and does not account for full-body movements, variable camera viewpoints, or physical walking within 3D environments. Acoustic interaction also lacks explicit semantic-emotional steering (e.g., sadness, anger, sarcasm).
- Future Directions: Upgrading the neural rendering backend to dynamic 3D Gaussian Splatting (3DGS) would offer robust multi-view consistency under large rotations; combining the motion generator with LLM-based cognitive emotion planners could enable richer, personality-driven conversational avatars.
Related Work & Insights¶
- vs MultiTalk (2025): MultiTalk uses Wan 2.1 diffusion, delivering rich visual details but suffering from heavy latency (0.61 FPS) and strict two-person constraints; InterTalk operates at 31.2 FPS and natively scales to arbitrary participant counts via adaptive pooling.
- vs INFP (CVPR 2025): INFP pioneers dual-audio conversational synthesis but lacks partner motion feedback and is restricted to single-person portrait rendering; InterTalk introduces bidirectional visual-acoustic context via RCE and supports true multi-person co-presence.
- vs DualTalk (CVPR 2025): DualTalk models 3D conversational meshes without delivering photorealistic 2D video; InterTalk leverages DualTalk's 3D motion corpus via FLAME-to-keypoint conversion to supervise photorealistic 2D rendering.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ Introduces the first real-time, participant-agnostic conversational talking face framework with bilateral visual-acoustic feedback.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across single-speaker, listener-response, and multi-person conversational setups with extensive ablations and a 23-subject user study.
- Writing Quality: ⭐⭐⭐⭐⭐ Coherent narrative structure, clear mathematical formulations, and well-organized experimental tables.
- Value: ⭐⭐⭐⭐⭐ Provides an immediately deployable architecture for interactive digital companions, multi-agent virtual environments, and low-latency group telepresence.