MindFlow: Harmonizing Cognitive Semantics and Acoustic Dynamics for Facial Animation Generation in Dyadic Conversations¶

Conference: ECCV2026
arXiv: 2606.27779
Code: https://harryxd2018.github.io/MindFlow/ (Project Page)
Area: Digital Humans / Image Generation
Keywords: Facial Animation, Dyadic Conversation, Ventral-Dorsal Dual-Pathway, Flow Matching, Selective Acoustic Injection

TL;DR¶

Inspired by the neuroscience model of the ventral-dorsal dual-pathway, MindFlow decouples facial animation generation in dyadic conversations into two collaborative pathways: cognitive semantic understanding (Ventral module) and reflexive motion generation (Dorsal module). Using the Chunk-State method, it extracts fine-grained emotional states from audio chunks to guide a flow matching network in generating high-fidelity facial animations.

Background & Motivation¶

Generating realistic facial animations for dyadic conversations is a long-standing goal in computer vision and graphics, with wide applications in virtual education, social companionship, and entertainment. However, existing methods consistently face a fundamental dilemma: the generated facial animations are either "hollow" (moving but lacking semantic support in expressions) or "stiff" (poor temporal alignment and incorrect response timing). The core reason for this dilemma is that facial movement in conversations is inherently paradoxical: on one hand, it requires high-level semantic understanding to ensure expressions are "reasonable" (e.g., smiling when hearing a joke); on the other hand, it requires instantaneous reflection of acoustic signals to guarantee synchronization (lips and audio must align perfectly).

From a neuroscience perspective, the human brain solves this contradiction perfectly. The famous Ventral-Dorsal dual-pathway model points out that natural conversation relies on two parallel neural pathways: a slow, contemplative Ventral pathway responsible for processing complex semantic and emotional information, and a fast, reflexive Dorsal pathway responsible for mapping acoustic signals directly to motor areas for pronunciation. However, existing computational methods fail to simulate this functional duality. Traditional audio-only driven methods (such as Audio2Photoreal, DualTalk) only mimic the Dorsal pathway's function, directly mapping motion from acoustic signals and ignoring high-level semantic understanding, which leads to visually active but semantically "hollow" animations. Recently, methods introducing LLMs (such as CustomListener, SocialAvatars) attempt to use text modality for semantic understanding, but they adopt a Sentence-Action paradigm—reasoning about behaviors based on sentence-level text. This approach faces two fundamental flaws: first, textual transcription inevitably loses paralinguistic information like intonation and stress; second, sentence-level granularity is too coarse to capture instantaneous emotional changes and precise temporal synchronization within an utterance.

The key insight of this paper is: since the human brain naturally processes these two types of requirements via two independent parallel pathways, computational models should also follow this biological design principle—there is no need to load two conflicting goals of semantic reasoning and motion control onto a single network. Core Idea: Inspired by the ventral-dorsal dual-pathway model in neuroscience, this paper decouples conversational facial animation generation into two independent, collaborative modules: the Ventral module continuously extracts fine-grained emotional states from the audio stream (cognitive semantic understanding), while the Dorsal module generates high-fidelity facial motion based on these emotional states and acoustic cues via conditional autoregressive flow matching (control of reflexive motion). The two modules achieve precise alignment using the Chunk-State method with audio chunks (rather than sentences) as the basic reasoning unit.

Method¶

Overall Architecture¶

The core architecture of MindFlow consists of two asynchronously running modules, corresponding to the biological Ventral and Dorsal pathways. The entire system operates in a streaming (causal) manner: at any time \(t\), it generates the current frame based solely on the historical audio and motion information up to \(t\)—consistent with the biological constraint of being unable to predict the future in human conversations. The inputs are continuous raw audio streams \(A_a^{\leq t}\) and \(A_b^{\leq t}\) of two interlocutors A and B, and the output is a 51-dimensional ARKit blendshape expression coefficient and a 3D Euler angle head pose for interlocutor A. The Ventral module asynchronously updates the emotional state \(S_a\) at the granularity of a 1.5-second audio chunk, while the Dorsal module generates facial motion in real-time at 25 FPS, reusing the latest emotional state until the next state arrives.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Dyadic Raw Audio Streams<br/>A_a, A_b"] --> B["Ventral Module<br/>(Asynchronous, every 1.5s)"]
    B --> C["Chunk-State Inference<br/>MLLM + Chain-of-State"]
    C --> D["Emotional State S_a"]
    D --> E["Dorsal Module<br/>(25 FPS Real-time)"]
    A --> F["Wav2Vec2 Acoustic Features"]
    F --> G["Selective Acoustic Injector"]
    G --> H["Autoregressive Transformer<br/>6× STM + SAI"]
    D --> I["Stylistic Temporal Modulator"]
    I --> H
    H --> J["Flow Matching Head<br/>5-step Euler Solver"]
    J --> K["Facial Animation<br/>51D ARKit + Head Pose"]

Key Designs¶

1. Chunk-State Method: Audio Chunks as Inference Units to Replace Sentence-Level Text Reasoning

Existing Sentence-Action methods first transcribe conversations into text and then use an LLM to reason about specific actions based on the full sentence. This introduces two issues: text transcription loses paralinguistic cues like speaking rate, pitch, and emotion; and sentence-level granularity is too coarse to guide precise temporal alignment. The idea of the Chunk-State method is to let the Ventral module avoid outputting specific motion commands (which is the Dorsal module's task) and instead directly take raw audio chunks as input, outputting a high-level, continuously evolving chain of emotional states (such as neutral \(\rightarrow\) slightly amused \(\rightarrow\) happy). A 1.5-second window is ideal: short enough to capture subtle emotional variations within an utterance, yet long enough (containing complete intonation patterns) to preserve paralinguistic cues. The Ventral module only performs "scene evaluation" at the cognitive level without pre-calculating spatial details; the specific expressions are autonomously generated by the Dorsal module upon receiving the emotional prior. This cognitive-motor decoupling design principle directly echoes the nature of the biological dual-pathway model, where the Ventral pathway is not responsible for fine motor execution.

2. Streaming Chain-of-State Mechanism: Enabling MLLM to Remember Its Inference Trajectory

Directly feeding audio chunks sequentially to the MLLM would disrupt conversation context (each chunk processed independently, forgetting historical emotions). Simply concatenating historical audio as input also leaves the MLLM lacking perception of prior inference states, causing emotion predictions to oscillate between adjacent chunks. The core of Chain-of-State is to incorporate the MLLM's inference history into the current reasoning: at step \(k\), the input to the Ventral module includes not only the current dyadic audio chunk, but also all historical audio chunks and all preceding emotional states \((A_a^{\leq wk}, A_b^{\leq wk}, S_a^{<k})\). The MLLM reasons about the current emotion within the full audio context and its own emotional trajectory, appending the new prediction to the context for the next round. This autoregressive emotional state chain effectively eliminates oscillation from memoryless reasoning—predictions of adjacent chunks transition smoothly, and the evolution of emotional states is continuously sent to the Dorsal module as a semantic prior.

3. Selective Acoustic Injector: Learning Dynamic Gating of Dyadic Audio via Attention Mechanism

There is an inherent paradox in dyadic conversation: when target A is speaking, the system should focus on A's own audio to guarantee lip-sync; when A is listening, it should pay attention to B's audio to drive appropriate listening reactions. Existing methods usually concatenate dyadic audio features and pass them through an MLP before cross-attention (early-mix), which dilutes information from different sources prematurely and makes it difficult for the network to distinguish between "speaking" and "listening" modes. The Selective Acoustic Injector avoids premature mixing by temporally interleaving A and B's audio features into a single acoustic context \(A_{ctx} = \text{Interleave}(A_a, A_b)\), and then using the motion history \(H_m\) as a query to perform cross-attention over this context. The attention mechanism naturally acts as a dynamic gate: when generating A's motion, attention spontaneously focuses on A's audio trajectory to ensure lip-sync; when it is the interlocutor's turn to speak, attention automatically switches to B's trajectory to drive listening behaviors. This gating behavior is not supervised by explicit speaker labels; it emerges implicitly from the generative target of flow matching. The attention heatmaps clearly demonstrate this unsupervised speaking-listening mode switching.

4. Conditional Autoregressive Flow Matching: Efficient Non-Deterministic Motion Generation

Facial motions are inherently diverse—the same audio can correspond to multiple plausible expression variations. Traditional deterministic regression (MSE loss) tends to collapse into blurry, averaged expressions. The Dorsal module adopts flow matching as a generative alternative, where an MLP parameterizes a velocity field \(v_\theta\) to learn the vector field mapping from a standard Gaussian distribution to the data distribution, with the training objective defined as \(\mathcal{L}_{flow} = \mathbb{E}[\|v_\theta(Z_\tau, \tau|C) - (Z_1 - Z_0)\|^2]\). A key advantage of flow matching is learning straight transport paths; during inference, a 5-step Euler solver is sufficient to approximate the solution, which is significantly faster than the iterative denoising steps required by diffusion models. Under the autoregressive framework, the Dorsal module sequentially predicts the head angular velocity and expression condition encoding of each frame. During training, Gaussian noise perturbations (random \(\sigma \in [0.01, 0.05]\)) are added to historical motions to prevent overfitting and history shortcuts. The head pose is deliberately designed to predict angular velocity rather than absolute angles, which effectively avoids pose drift issues in autoregressive inference.

Loss & Training¶

The only training signal for the Dorsal module is the flow matching loss \(\mathcal{L}_{flow}\), containing no explicit lip-sync loss or emotion classification loss. Training follows a two-stage strategy: first, pre-training for 90k steps on HDTF + VICOX to establish basic speaking and conversational behavior patterns, followed by fine-tuning for 30k steps on MEAD + VICO to enhance the quality of emotional expression. The audio encoder Wav2Vec2 is frozen throughout. Using the Adam optimizer with a learning rate of \(1\times10^{-5}\), cosine scheduling with a 1% warmup, and a batch size of 64. During inference, the Ventral module processes each chunk in approximately 1.38s, while the Dorsal module runs in real-time at 25 FPS. The system requires a total of 59 GB VRAM and can perform continuous inference on 2-minute sequences.

Key Experimental Results¶

Main Results¶

State	Dataset	Metric	Ours	Prev. SOTA	Gain
Speaking	HDTF	SyncD ↓	0.333	0.341 (A2P)	-2.3%
Speaking	HDTF	SyncC ↑	0.520	0.519 (A2P)	+0.2%
Speaking	HDTF	FD (Exp) ↓	15.76	17.64 (A2P)	-10.7%
Listening	VICO	FD (Exp) ↓	13.86	14.24 (A2P)	-2.7%
Listening	VICO	MSE (Exp) ↓	0.30	0.34 (A2P)	-11.8%

Ablation Study¶

Configuration	Key Metric	Description
Full model	FD (Exp) 13.86	Full model
w/ Random emotional state	FD (Exp) 15.21	Without semantic guidance, FD increases by 9.7%
Sentence-Action instead of Chunk-State	FD (Exp) 14.39	Sentence-level reasoning leads to temporal misalignment
w/o Selective Acoustic Injector (Ours)	SyncD 0.350	Removing injector decreases lip-sync
w/o SAI (A2P Architecture)	SyncD 0.341	Injector is cross-architecture general, and works effectively on other frameworks

Key Findings¶

Ablation studies clearly corroborate the necessity of the dual-pathway design: without Ventral semantic guidance, expression FD rises from 13.86 to 15.21, demonstrating that acoustic-motion mapping alone is insufficient to generate semantically reasonable expressions.
The Selective Acoustic Injector also brings improvements to the A2P architecture (SyncD 0.341 \(\rightarrow\) 0.331), proving its cross-architecture generalizability beyond the bespoke design of MindFlow.
Experiments on sampling steps reveal a counter-intuitive result: quality actually degrades when exceeding 5 steps, which is attributed to longer integration paths accumulating vector field prediction errors and introducing high-frequency jitter.
Training head pose to predict angular velocity instead of absolute angles successfully solves the issue where the model takes the shortcut of "copying historical poses" during autoregressive inference, avoiding cumulative drift.

Highlights & Insights¶

Dual-Pathway Decoupling + Granularity Breakthrough: Mapping the biological Ventral-Dorsal dual-pathway model directly to a computational framework allows each module to excel in its specialty—Ventral for scene evaluation and Dorsal for motion execution. This yields superior performance compared to loading two conflicting goals onto a single model. Moreover, replacing Sentence-Action with Chunk-State represents a crucial breakthrough in granularity: a 1.5-second audio chunk preserves paralinguistic cues while providing fine temporal resolution.
Implicit Attention Gating: The Selective Acoustic Injector does not rely on an explicit speaker detection branch. Instead, the attention switching pattern of "watching oneself when speaking, and watching the interlocutor when listening" naturally emerges from the flow matching objective, being both elegant and effective.
Real-time Advantages of Flow Matching: The 5-step flow matching solver under the autoregressive framework ensures high efficiency for non-deterministic generation, achieving a balance between expression diversity and computational overhead at an inference speed of 25 FPS.

Limitations & Future Work¶

Sole Reliance on Audio Modality: The authors explicitly acknowledge that the current framework only utilizes auditory input, neglecting visual cues such as the partner's facial expressions, eye contact, and body language. Future work could incorporate multimodal perception to achieve a more comprehensive contextual understanding.
High VRAM Consumption: Full inference requires 59 GB of VRAM, with the main bottleneck stemming from the parallel execution of the Ventral MLLM and the Dorsal module, which limits deployment on consumer-grade hardware.
Unknown Cross-Lingual/Cross-Cultural Generalization: The experiments were evaluated solely on English datasets, but emotional expressions vary significantly across different cultures and languages. While the Chunk-State method’s reliance on raw audio theoretically possesses cross-lingual transfer capabilities, this remains to be experimentally verified.
Special Treatment Still Required for Head Pose: Although angular velocity prediction mitigates the drift problem, the pose is not directly generated using flow matching like expressions are, indicating that there is still room for improvement regarding long-term structural stability.

vs. Purely Audio-Driven Methods (Audio2Photoreal, DualTalk): These methods only simulate the Dorsal pathway, neglecting semantic understanding. MindFlow decreases the listening state FD by 11.8% compared to the best audio-only method, demonstrating the necessity of semantic priors.
vs. Sentence-Action Methods (CustomListener, SocialAvatars): These methods understand textual semantics through LLMs but lose paralinguistic details and have coarse temporal granularity. MindFlow’s Chunk-State method reasons directly in the audio domain, offering higher precision and more accurate response timing.
vs. Unified Conversation Frameworks (UniTalker, OmniResponse): Most of these frameworks focus on comprehensive response generation during the speaking phase, paying less attention to continuous non-verbal feedback during the listening phase. MindFlow's dual-stream asynchronous architecture naturally covers the full information flow of both speaking and listening stages.

Rating¶

Novelty: ⭐⭐⭐⭐☆ The biologically-inspired dual-pathway decoupling and the Chunk-State method represent novel designs, constituting systematic improvements rather than disruptive breakthroughs.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Includes comparisons across dual metrics, multi-component ablations, user studies, and sampling step analysis, with in-depth discussions on ablation results (including cross-architecture injector validation).
Writing Quality: ⭐⭐⭐⭐⭐ The logical pipeline from neuroscience principles to method design and experimental validation is clear and complete, with a concise and powerful description of the motivation.
Value: ⭐⭐⭐⭐☆ Conversational facial animation is a core challenge in digital humans. MindFlow provides a reproducible dual-pathway paradigm that is highly instructive for subsequent research, though its VRAM overhead limits industrial deployment.