Conversational Human Audio-visual Talking Dialogue Generation¶

Conference: ECCV2026

Paper: https://eccv.ecva.net/virtual/2026/poster/4941
PDF: https://media.eventhosts.cc/Conferences/ECCV2026/pdfs/8120.pdf
Project: CHAT

Area: Human Understanding

Keywords: Dyadic dialogue generation, emotional speech, facial reaction generation, video diffusion, synthetic data

TL;DR¶

CHAT turns an anonymous scenario prompt into dialogue scripts, emotional dual-track speech with brief responses, and mutually responsive face videos with smooth transitions, achieving FID 17.33 and LSE-C 6.89 against related-task baselines while showing that synthetic dialogues can improve facial reaction model pretraining.

Background & Motivation¶

Natural dialogue is not two monologues played in alternation: a speaker's tone influences the listener's expression, while the listener may nod, blink, or offer a brief response before the speaker finishes. Digital humans need to learn these relationships, but collecting paired audio-visual data is constrained by privacy, demographic coverage, annotation costs, and multimodal synchronization. Simply adding more single-speaker videos does not directly supply examples of mutual influence.

Existing techniques already solve several parts of this problem. Hallo3 and SadTalker animate talking faces from supplied audio, ReactDiff generates facial reactions from a partner's audio-visual behavior, and dyadic head generation methods generally assume two audio tracks or scripts are already available. These input settings differ from generating a complete dyadic conversation from an anonymous scenario alone. Running two single-person models side by side does not automatically connect speech content, emotion, and nonverbal reactions.

The paper therefore defines Dyadic Interactive Audio-visual Dialogue Generation (DIADG), reusing language, speech, and face models for basic capabilities while focusing new modeling on interaction. Core idea: establish a shared dialogue timeline and emotion conditions, fill the listener's silent intervals, and explicitly condition each participant's facial behavior on the partner's audio-visual behavior rather than treating independent videos as a conversation.

Method¶

Overall Architecture¶

The input describes a scenario and conversational style without specifying real identities. The output comprises multiple dyadic audio-visual dialogues, each with separate video and audio tracks for both participants. CHAT has three top-level modules: Textual Dialogue Generation (TDG), Dyadic Audio Dialogue Generation (DADG), and Interactive Facial Behaviour Generation (IFBG). Its main refinement components are IAR within DADG and IFBR within IFBG.

TDG first determines identity descriptions and scripts. IAR refines the wording, annotates emotions and timing, and synthesizes two audio tracks. IFBG then samples synthetic faces, generates initial speaking segments from speech and emotion, and completes the videos through Silent Facial Completion and Responsive and Temporal Refinement. The latter two designs together cover IFBR: one fills the listening intervals, while the other handles responses to the partner and connections between segments.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Anonymous scenario prompt"] --> B["Textual Dialogue Generation"]
    B --> C["Interactive Audio Refinement"]
    C -->|Speech and emotion drive initial speaking segments| D["Silent Facial Completion"]
    B -->|Identity descriptions and synthetic faces| D
    D --> E["Responsive and Temporal Refinement"]
    C -->|Partner speech and emotion| E
    E --> F["Paired face videos<br/>and two audio tracks"]
    C --> F

Key Designs¶

1. Textual Dialogue Generation: establish one shared conversation rather than two separate monologues

TDG uses a text large language model to generate multiple scripts from the scenario, each accompanied by identity descriptions for both participants. These descriptions constrain how the characters speak and support subsequent synthetic-face sampling, preventing content and identity from being entirely independent. Each conversation is limited to 5–10 turns, an engineering choice intended to avoid coherence degradation in longer dialogues. A turn can contain multiple sentences, so turn count must not be equated with sentence count.

The script is then organized on separate participant timelines. When one person speaks, an equally long silent segment is reserved for the other, rather than compressing all of that person's lines into an uninterrupted monologue. The paper illustrates this arrangement through alternating odd- and even-numbered sentences; its purpose is to maintain a shared timeline. A silent segment is not a final instruction to do nothing: it explicitly reserves space for brief responses and listener expressions.

2. Interactive Audio Refinement: pass emotion, brief responses, and timing jointly into speech synthesis

IAR first prompts a language model to make the script more conversational and produce four types of control information: brief response words, sentence-level emotion descriptors, start and end times for sentences and responses, and a sound environment. Emotion descriptors include not only categories but also continuous prosodic attributes such as rate, pitch, energy, and pauses. Brief responses such as “hmm” and “okay” can be inserted into silent intervals with a predefined probability, so the two tracks need not consist exclusively of alternating complete sentences. The paper does not specify that probability's value, so it should not be treated as a known hyperparameter.

Each sentence or response word passes through a content Transformer encoder, while its emotion passes through an emotion Transformer encoder. The equal-dimensional representations are added elementwise and supplied to an emotional speech synthesizer alongside timestamps and sound-environment conditions. The important connection is not simply an extra emotion label: the same emotion conditions continue into face generation. Speech features are concatenated with emotion representations to drive the talking-face model, reducing mismatches such as an excited voice paired with an indifferent face. The dialogue remains primarily turn-based; this does not establish a solution to arbitrary interruptions or sustained overlapping speech.

3. Silent Facial Completion: connect listener intervals to surrounding speaking states

The initial face videos cover only speaking segments. An identity generation or sampling component derives synthetic faces from the textual identity descriptions, providing a consistent appearance throughout the conversation. The talking-face model generates corresponding segments from each participant's own speech and emotion. Freezing the image during gaps would produce a stiff listener and potentially abrupt transitions back to speech.

SFBG fills these silent intervals using the SilentDiff video diffusion model, conditioned on the current person's identity image and the preceding and succeeding speaking segments. These contextual constraints help maintain continuity in behaviors such as nodding and blinking, rather than merely interpolating frames. This stage primarily addresses how the person transitions between adjacent states; it does not yet sufficiently ensure that the motion responds to the other participant. That requirement belongs to the next design. Because future speaking segments are already used, the method performs offline sequence synthesis rather than strictly causal, real-time listener generation.

4. Responsive and Temporal Refinement: guide diffusion with partner behavior, then repair segment boundaries

RFBG refines both speaking and silent segments. When generating a person's current segment, it reads the partner's facial behavior, audio, and emotion from the preceding, current, and succeeding segments. Visual conditions are extracted at multiple spatial scales and fused with the target person's identity through learnable cross-attention at each scale. The desired information is the response cue provided by the partner's expressions and movements, not a copy of the partner's face; identity conditioning constrains the output to remain the target person.

Features at all scales are upsampled to a common spatial resolution and progressively introduced during denoising. Early steps receive low-resolution global motion, while later steps add high-resolution expression details. The scale schedule described in the paper can be written as:

\[ l(t)=\max\left(1,\left\lceil L\frac{t}{T}\right\rceil\right),\qquad \text{active scales}=\{l(t),\ldots,L\}. \]

Here \(L\) is the number of scales, and larger scale indices indicate lower resolution; \(t\) decreases from the noisy endpoint \(T\) toward the clean endpoint. Initially, conditioning therefore mainly uses scale \(L\), with finer scales becoming available later. The partner's audio and emotion representations are injected through cross-attention at the first denoising step. This aligns conditional resolution with the diffusion process of establishing an overall response before filling in facial details, rather than assigning every condition the same role throughout sampling.

Segmentwise RFBG processing can still leave discontinuities. TCR therefore applies symmetric Gaussian-weighted blending to the tail and head of adjacent segments. The paper's example uses 100 frames per segment and \(W=10\) frames on each side of a boundary. The blending weight is 0.5 at the boundary and decays outward, with scale parameter \(\sigma=W/3\). Both participants receive the same transformation over the same temporal window to avoid disrupting their established correspondence by smoothing only one participant. This addresses local stitching rather than regenerating the whole conversation, and it should not be interpreted as a formal guarantee of preserved interaction semantics.

A Worked Example¶

Consider the illustrative prompt “two anonymous colleagues discuss project progress.” TDG first generates the characters and an alternating script. While the first colleague speaks, an equally long silent interval is reserved on the second colleague's timeline. IAR can insert a contextually appropriate brief response and assign emotions and timing to both participants.

Speech and emotion then drive their initial speaking segments, while SFBG fills the second colleague's listening behavior from the surrounding speaking states. RFBG reads the first colleague's voice and expressions over neighboring segments to make the second colleague's behavior responsive; the same procedure applies to the other participant. TCR finally blends only segment edges, producing two aligned audio-visual streams. This illustrates the mechanism, not an additional experimental example reported in the paper.

Loss & Training¶

TDG and DADG use Gemini models, and the pretrained TTS reference in IAR is XTTS. SFBG and RFBG are pretrained with AdamW on HDTF and REACT 2024, respectively. RFBG uses 50 denoising steps. The talking-face model, SFBG, and RFBG are subsequently trained jointly, while the language models and TTS remain frozen.

“End-to-end” therefore refers only to the trainable face-related modules, not to updating the entire language–speech–video system under one objective. The available paper does not provide the complete joint loss, component weights, learning rate, or training budget; this note does not substitute a conventional diffusion loss for missing implementation details. The authors report that CHAT-AVD-50k contains 50,000 dialogue pairs, 100,000 clips, 1388.9 hours, and 100,000 synthetic facial identities, with turn-level emotion, scenario, and identity metadata. The conclusion still describes release with provenance metadata and bias auditing as future work.

Key Experimental Results¶

Main Results¶

Evaluation uses 1000 generated conversations of 5–10 turns each, but quantitative comparisons are restricted to their first 10 seconds to accommodate related baselines' length limits. Hallo3, SadTalker, and EDTalk receive CHAT-generated audio and identities; DIM and ReactDiff receive CHAT-generated speaker videos. These are not complete end-to-end competitors under identical task inputs. The table compares related capabilities rather than establishing a comprehensive ranking of dyadic generation methods.

FID/FVD measure image or video distribution differences, with lower values preferred; LSE-C measures audio-visual synchronization confidence, with higher values preferred. FRCorr measures facial reaction appropriateness and FRDiv measures reaction diversity, both higher-is-better. The table retains the original \(\times10^{-2}\) scale. The paper does not expand the computation of these two metrics inherited from REACT, so they should not be described as turn accuracy or semantic consistency rates.

Method	FID ↓	FVD ↓	LSE-C ↑	FRCorr ↑	FRDiv ↑
Hallo3	20.56	362.23	4.61	N/A	N/A
SadTalker	22.53	385.51	6.85	N/A	N/A
DIM	36.52	460.35	6.82	31.02	12.55
EDTalk	18.74	619.92	5.62	N/A	N/A
ReactDiff	21.36	386.22	5.23	48.05	15.31
CHAT	17.33	365.03	6.89	50.02	15.34

CHAT improves FID over the listed baselines, but its FVD is slightly worse than Hallo3's. FRDiv increases by only 0.03 table units over ReactDiff, which should not be presented as a large improvement. In the 60-participant user study using a 5-point scale, CHAT's interactivity score is 4.8, compared with 3.6 for DIM and 3.2 for ReactDiff. These are mean subjective scores, not win rates.

Ablation Study¶

The following subset of Table 3 also reports means over 1000 samples. Emo-Acc is speech emotion classification accuracy. Removing IFBR or RFBG leaves the audio branch unchanged; the paper marks the corresponding audio metrics as inapplicable, not zero.

Config	Emo-Acc ↑	FID ↓	FVD ↓	FRCorr ↑	FRDiv ↑
Full model	78.4%	17.33	365.03	50.02	15.34
Without IAR	41.2%	21.34	456.78	32.45	7.89
Without emotion representations	38.3%	23.45	423.56	28.34	5.67
Without IFBR	—	38.67	623.45	28.56	6.78
Without RFBG	—	20.34	412.56	25.78	4.56
Without TCR	—	18.56	421.34	48.67	14.23

Removing emotion representations lowers Emo-Acc by 40.1 percentage points, suggesting that they provide more than optional voice coloring. Without RFBG, FRCorr falls from 50.02 to 25.78 and FRDiv from 15.34 to 4.56, directly supporting the role of partner-conditioned refinement. Removing TCR increases FVD from 365.03 to 421.34, showing that boundary processing also affects video quality.

Key Findings¶

Synthetic-data utility is tested separately through downstream Interactive Head Generation: pretrain on CHAT-AVD-50k, fine-tune on the REACT 2024 training split, and evaluate under its official protocol. FRDist is a dynamic time warping distance to the ground-truth reaction, with lower values preferred. FRCorr again uses the \(\times10^{-2}\) scale.

Model	Pretraining data	FRCorr ↑	FRDist ↓
PerFRDiff	None, REACT 2024 only	37.21	94.72
PerFRDiff	CHAT-AVD-50k	40.11	89.45
ReactDiff	None, REACT 2024 only	24.19	86.70
ReactDiff	CHAT-AVD-50k	26.12	83.87

Both models benefit, but CHAT's IFBR is itself pretrained on REACT 2024. The evidence therefore supports scale and diversity augmentation within a related distribution, not strictly independent cross-domain transfer.
Although the downstream and main tables share model names, their task inputs, training configurations, and test protocols differ. ReactDiff's FRCorr values across those tables are not directly comparable.
Current automatic metrics primarily cover facial behavior and audio-visual properties. Complete turn organization, long-range semantic coherence, and interruption timing do not yet receive equally strong objective validation.

Highlights & Insights¶

A shared timeline turns silence into a behavioral interval to model rather than a gap in the data. Brief vocal responses and listener expressions can consequently occupy the same interaction coordinates.
Aligning coarse-to-fine partner visual conditions with diffusion progress is more targeted than simply adding cross-attention. Target identity also participates in multiscale fusion, keeping response modeling connected to identity consistency.
Data utility is tested through pretraining two facial reaction models rather than judged solely by visual appeal. The authors explicitly acknowledge the related-distribution overlap, keeping the interpretation within the evidence.

Limitations & Future Work¶

Repeated language, speech, and video diffusion calls make generation expensive. The authors use scalability mainly to mean demographic coverage, not low latency or low per-sample cost. The paper does not report a reproducible complete runtime and hardware budget.
The system currently generates English dialogue and primarily follows turn-taking with brief responses. Together with its dependence on future segments, this means it should not be treated as a real-time, free-form conversation system.
Related-task baselines depend on intermediate inputs supplied by CHAT, and some closer dyadic generation methods are excluded because code is unavailable. Evaluation of only the first 10 seconds also cannot establish long-conversation quality.
The joint loss, the implementation of emotion conditioning in the pretrained TTS, and FRCorr/FRDiv evaluation details are incompletely specified. Provenance metadata, bias auditing, and independent-corpus evaluation remain proposed follow-up work; anonymous prompts and synthetic faces alone do not establish those validations.

vs Hallo3 / SadTalker / EDTalk: These methods address face animation or emotion control from supplied audio. CHAT generates dialogue and audio upstream, then adds listener segments and mutual responses. Broader task coverage comes with accumulated costs and errors across generation stages.
vs DIM / ReactDiff: Facial reaction generation treats the partner's audio-visual behavior as given. CHAT first synthesizes those conditions and then constructs complete dyadic dialogue. Its main experiment uses related-task models to compare component capabilities, while the downstream experiment tests the training value of its synthetic data.
vs DualTalk / INFP / TAVID / JAM-Flow: The paper associates these methods with predefined audio or script conditions, whereas CHAT starts further upstream from an anonymous scenario prompt. This task boundary matters, but it does not mean that same-protocol experiments have established superiority over every such method.

Rating¶

Novelty: 4/5. The contribution lies in organizing scenario-to-dialogue generation and explicit audio/facial interaction refinement rather than reinventing foundation generators.
Experimental Thoroughness: 3/5. Main comparisons, detailed ablations, a user study, and downstream validation are present, but same-task baselines, long-dialogue evaluation, and independent transfer remain limited.
Writing Quality: 3/5. Module roles and limitations are clear, but training and some metric definitions are incomplete, and turns must be carefully distinguished from sentences.
Value: 4/5. Useful for paired interaction data synthesis and facial reaction training, currently best understood as an offline data-generation pipeline.