Wav2Sem: Plug-and-Play Audio Semantic Decoupling for 3D Speech-Driven Facial Animation

Conference: CVPR 2025
arXiv: 2505.23290
Code: https://github.com/wslh852/Wav2Sem.git
Area: Human Understanding
Keywords: Speech-driven facial animation, near-homophonic syllable decoupling, semantic features, plug-and-play module, self-supervised audio models

TL;DR

This work proposes Wav2Sem, a plug-and-play audio semantic decoupling module. By extracting global semantic features from complete audio sequences and fusing them with existing self-supervised audio models (HuBERT/Wav2Vec 2.0), it addresses the coupling issue of near-homophonic syllables in the feature space. This significantly mitigates the "averaging effect" in lip-shape generation, achieving consistent performance improvements across six facial animation models with different architectures.

Background & Motivation

Background: 3D speech-driven facial animation is an important technology in virtual reality, film production, and gaming. The high correlation between audio and lip shape makes lip-syncing a core evaluation metric. Current mainstream methods (e.g., FaceFormer, CodeTalker, FaceDiffuser) widely use pre-trained self-supervised audio models (e.g., HuBERT, Wav2Vec 2.0) as encoders, benefiting from their excellent feature generalization capabilities.

Limitations of Prior Work: Natural language contains numerous near-homophonic syllables—pronunciations that sound similar but correspond to significantly different lip shapes. For example, the long vowel /iː/ in "sheep" features slightly parted lips, whereas the short vowel /ɪ/ in "ship" has more neutral, wider-opened lips. Self-supervised models like HuBERT and Wav2Vec 2.0 are mainly trained on unlabeled audio, focusing on phoneme-level feature modeling and lacking semantic representation. Consequently, near-homophonic syllables are heavily coupled (nearly indistinguishable) in the feature space, leading subsequent lip-shape generators to produce "averaged" lip movements, which severely hurts lip-syncing accuracy.

Key Challenge: Self-supervised audio models excel at capturing acoustic/phonemic features but struggle with semantic information. However, distinguishing near-homophones (e.g., "sheep" vs. "ship") requires contextual semantic understanding—just as humans resolve ambiguities in daily communication via context rather than pure phonemes.

Goal: Without modifying existing facial animation model architectures, this work aims to design a plug-and-play module that directly extracts global semantic features from audio signals, compensating for the missing semantic information to decouple the features of near-homophonic syllables.

Key Insight: Text and audio are two different expressions of semantic details, yet they convey the same core meaning. Therefore, a model can be trained to directly learn the corresponding textual semantic representation (BERT space) from audio, bypassing the need for additional text inputs during inference.

Core Idea: Utilize a TCN + Transformer to extract global semantic features from audio sequences, align them with the sentence-level semantic space of BERT, and then inject them into the output of existing self-supervised audio encoders via a simple additive fusion. This achieves plug-and-play decoupling of near-homophonic features.

Method

The design of Wav2Sem is extremely elegant: it pre-trains an audio-to-text semantic mapping module, freezes its parameters, and directly integrates it into the backend of any existing facial animation model's audio encoder.

Overall Architecture

Training Stage: On the large-scale speech transcription dataset LibriSpeech-960, given the input audio \(\mathbf{A}\), local features are extracted via TCN and global semantics are captured through a 12-layer Transformer. The output is mean-pooled to obtain sentence-level semantic features \(\mathbf{F}_s\), which are aligned with the CLS token \(\mathbf{F}_{CLS}\) generated by BERT for the corresponding text using an L1 loss.

Inference/Downstream Phase: The parameters of Wav2Sem are frozen. Semantic features \(\mathbf{F}_s\) are extracted from the input audio and fused with the phoneme-level features \(\mathbf{F}_p\) of the original self-supervised encoder (HuBERT/Wav2Vec 2.0) using two fully connected (FC) layers and element-wise addition to obtain the decoupled features \(\mathbf{F}_d\). These replace the original audio features and serve as input to the downstream facial animation model.

Key Designs

1. Semantic Space Definition (BERT Sentence-level Representation):

   • Function: Provides a well-defined target semantic space to serve as an anchor for audio semantic alignment.
   • Mechanism: Uses the CLS token (or the average of all tokens) encoded by a pre-trained BERT from the text \((x_1, x_2, ..., x_M)\) as the sentence-level semantic representation \(\mathbf{F}_{CLS}\). Two versions are provided: \(\text{Wav2Sem}_c\) (aligning to the CLS token) and \(\text{Wav2Sem}_m\) (aligning to the token average).
   • Design Motivation: Directly constructing a semantic space from audio is challenging due to varying acoustic features for the same semantic content. In contrast, text has clear semantic boundaries, and BERT provides mature sentence-level semantic representations. Word-level alignment requires precise temporal alignment and lacks context, making sentence-level alignment a more natural choice.

2. Wav2Sem Encoder (TCN + Transformer):

   • Function: Extracts global semantic features from raw audio signals.
   • Mechanism: A 7-layer TCN block (512 channels, 49Hz output, ~20ms stride) first converts audio into local features \(\mathbf{Z} = \text{TCN}(\mathbf{A})\). Then, a 12-layer Transformer (8 attention heads, MLP inner dimension 3072) captures long-range dependencies: \(\hat{\mathbf{Z}}^l = \mathbf{Z}^{l-1} + \text{MHSA}(\text{LN}(\mathbf{Z}^{l-1}))\), \(\mathbf{Z}^l = \hat{\mathbf{Z}}^l + \text{MLP}(\text{LN}(\hat{\mathbf{Z}}^l))\). Finally, average pooling is performed across all positions: \(\mathbf{F}_s = \frac{1}{N}\sum_{i=0}^{N}\mathbf{Z}_i\).
   • Design Motivation: TCN excels at capturing local temporal patterns (phoneme/syllable boundaries), whereas the Transformer is adept at modeling global contextual relationships. Combining the two progressively builds global semantic understanding from local acoustic features.

3. Semantic Fusion Module:

   • Function: Injects global semantic features into phoneme-level features to decouple near-homophonic syllables.
   • Mechanism: An extremely straightforward design—the semantic feature \(\mathbf{F}_s \in \mathbb{R}^{1 \times C}\) is broadcasted along the sequence dimension through an FC layer, element-wise added with phoneme-level features \(\mathbf{F}_p \in \mathbb{R}^{N' \times C}\), and transformed via another FC layer: \(\mathbf{F}_d = \text{FC}(\text{FC}(\mathbf{F}_s) + \mathbf{F}_p)\).
   • Design Motivation: Intentionally kept simple to minimize integration complexity (a core requirement of "plug-and-play"). Even with simple additive fusion, semantic information effectively introduces distinguishing contexts for near-homophonic syllables, mimicking how humans resolve ambiguity through context.

Loss & Training

Wav2Sem pre-training uses the L1 loss: \(\mathcal{L} = \|\mathbf{F}_{CLS} - \mathbf{F}_s\|_1\). It is trained for 200 epochs on LibriSpeech-960 using the Adam optimizer with a learning rate of \(10^{-4}\) and batch size of 1 on an RTX A6000 Ada. After pre-training, the parameters of Wav2Sem are frozen, and the original hyperparameters of downstream models are adopted during joint training/evaluation.

Key Experimental Results

Main Results

Model	Audio Encoder	VOCASET LVE (×10⁻⁵)↓	BIWI LVE (×10⁻⁴)↓
VOCA	DeepSpeech	4.9245	6.7158
VOCA + Wav2Sem_c	+Wav2Sem	4.8915	6.6821
FaceFormer	Wav2Vec 2.0	4.1090	4.9847
FaceFormer + Wav2Sem_c	+Wav2Sem	3.9891	4.9571
CodeTalker	Wav2Vec 2.0	3.9445	4.7914
CodeTalker + Wav2Sem_m	+Wav2Sem	3.8714	4.7847
UniTalker	Wav2Vec 2.0	3.5416	4.0213
UniTalker + Wav2Sem_c	+Wav2Sem	3.1476	3.9112
FaceDiffuse	HuBERT	3.7924	4.2985
FaceDiffuse + Wav2Sem_m	+Wav2Sem	3.7628	4.2816
LG-LDM	HuBERT	3.7925	4.9869
LG-LDM + Wav2Sem_c	+Wav2Sem	3.7863	4.9258

Ablation Study

Semantic Representation	VOCA MVE↓	VOCA LVE↓	FaceFormer LVE↓
w/o Wav2Sem	6.1571	4.9245	4.1090
+ BERT_m (Direct Text)	6.0614	4.9041	4.0872
+ BERT_c (Direct Text)	6.0468	4.8995	4.0241
+ Wav2Sem_m (Inferred from Audio)	6.0358	4.9032	4.0654
+ Wav2Sem_c (Inferred from Audio)	6.0015	4.8915	3.9891

Key Findings

• Wav2Sem consistently improves LVE/MVE/FDD metrics across all six baselines with distinct architectures, validating its generalizable plug-and-play nature.
• Wav2Sem inferred directly from audio even outperforms direct usage of BERT text features in some scenarios (such as with FaceFormer), indicating that the extracted audio semantic features are of high quality.
• \(\text{Wav2Sem}_c\) (CLS token) performs better on the larger BIWI dataset, whereas \(\text{Wav2Sem}_m\) (token average) is superior on the smaller VOCASET dataset—reflecting that the CLS token contains richer information but is more prone to overfitting.
• T-SNE visualization clearly illustrates the feature-decoupling effect of Wav2Sem on near-homophonic syllables (e.g., /pl/ vs. /bl/, /tɪkl/ vs. /pɪkl/).
• Word-level L2 distance measurements reveal that the feature distance of near-homophones for Wav2Vec 2.0 + Wav2Sem increases from 0.0397 to 0.0701 (an improvement of ~77%).

Highlights & Insights

• Precise problem motivation: The coupling of near-homophonic word features is an inherent flaw in self-supervised audio models. This work is the first to systematically identify and resolve this issue.
• Ultimate "plug-and-play" paradigm: No modifications are made to any downstream model structures; the fusion block comprises only two FC layers and element-wise addition, minimizing integration costs.
• Clever training strategy: By pre-training the semantic mapping on large-scale text-audio pairs and freezing it, semantic learning is completely decoupled from facial animation generation.
• Convincing experimental design: Validated as highly effective across six baselines with vastly different architectures (CNN, Transformer, VQ-VAE, TCN, Diffusion, Latent Diffusion).
• Visualizations of semantic decoupling (T-SNE, L2 distance) provide intuitive evidence for the core claims.

Limitations & Future Work

• The degree of improvement is relatively small on certain strong models (e.g., LG-LDM's LVE decreases from 3.7925 to 3.7863), suggesting diminishing marginal returns for already advanced baselines.
• Only validated on English; near-homophones in tonal languages like Chinese may present more complex challenges.
• The 12-layer Transformer in Wav2Sem adds some inference overhead, and the specific extra latency has not been reported.
• The sentence-level semantic feature is globally shared (identical across all frames in a sequence). Can temporally localized semantic injection be further designed?
• Since only L1 loss is used for alignment, contrastive learning or more advanced semantic alignment methods might yield better results.

Related Work & Insights

• Difference from multi-modal methods like EMAGE: EMAGE requires additional text input, while Wav2Sem reasons semantic information directly from the audio, making it more practical (no text transcription needed during inference).
• Insight: The phoneme-level limitation of self-supervised audio models may affect many downstream tasks beyond facial animation. The core concepts of Wav2Sem could be transferred to fields like speech translation and speech emotion analysis.
• Using the BERT sentence-level semantic space as the target space for cross-modal alignment proves to be a general and effective choice.

Rating

Dimension	Score (1-10)	Description
Novelty	7	The identified problem is valuable, though the methodology is relatively straightforward.
Experimental Thoroughness	9	Evaluated with six baselines, two datasets, ablation studies, and visualizations.
Writing Quality	7	Clearly structured, but details could be more concise.
Value	8	Highly practical due to its plug-and-play nature; open-source code is provided.

Related Papers

• [CVPR 2025] Teller: Real-Time Streaming Audio-Driven Portrait Animation with Autoregressive Motion Generation
• [CVPR 2026] Stand-In: A Lightweight and Plug-and-Play Identity Control for Video Generation
• [CVPR 2025] Semantic Satellite Communications for Synchronized Audiovisual Reconstruction
• [CVPR 2025] Generative Inbetweening through Frame-wise Conditions-Driven Video Generation
• [CVPR 2025] AnimateAnything: Consistent and Controllable Animation for Video Generation