title: >- [Paper Note] EmoteGPT: 3D Human Facial Expression from Natural Language Descriptions description: >- [ECCV 2026][Image Generation][3D Facial Expression] Regressing disentangled FLAME 3D facial expressions from explicit and implicit natural language using an MLLM with a dedicated token. tags: - ECCV 2026 - Image Generation - 3D Facial Expression - MLLM - FLAME Model date: 2026-09-19 content_hash: 5a8db51474472747

EmoteGPT: 3D Human Facial Expression from Natural Language Descriptions¶

Conference: ECCV 2026
Paper: ECCV Official
Project Page: https://genintel.github.io/EmoteGPT/
Area: Image Generation
Keywords: 3D Facial Expression, Multimodal Large Language Models, FLAME Model, Txt2Emote, Explicit and Implicit Text Control

TL;DR¶

EmoteGPT presents the Txt2Emote benchmark featuring explicit feature and implicit situational annotations for 3D facial expressions, and integrates a dedicated <Expr> token within a multimodal large language model to deterministically regress disentangled FLAME expression parameters in real time.

Background & Motivation¶

Precise control of 3D facial expressions from natural language descriptions is crucial for expressive avatars, digital human animation, and intuitive human-computer interaction. Traditional expression authoring relies on tedious manual tweaking of blendshape weights or keyframing by skilled technical animators, which is labor-intensive and inaccessible to non-expert users. While recent advances in text-to-3D head generation have explored Score Distillation Sampling (SDS) via 2D/3D diffusion priors, these methods inextricably entangle facial identity, head pose, and facial expressions. Moreover, diffusion-based approaches suffer from expensive iterative sampling, noisy artifacts, and over-saturation, failing to provide fine-grained, real-time, and cleanly disentangled expression manipulation.

A primary technical barrier preventing direct language-driven expression synthesis is the severe scarcity of datasets pairing diverse natural language descriptions with precise 3D facial expression parameters. Existing text-annotated face datasets—such as CelebA-Dialog, MMCelebA-HQ, and CelebAText—predominantly focus on static appearance attributes (e.g., hair color, age, gender) and only offer coarse expression tags like "smiling" or "mouth open", completely omitting nuanced facial muscle articulations. Furthermore, humans rarely describe emotions by cataloging anatomical facial muscle actions; instead, natural human communication frequently relies on indirect, situational contexts that naturally evoke emotions (e.g., "just won a major lottery" or "arguing heatedly during a traffic altercation"). Prior generative models struggle with these contextual prompts due to the lack of situational grounding.

To overcome these challenges, this paper reformulates text-driven facial expression generation as a regression problem within the compact, disentangled parameter space of the 3D Morphable Model (FLAME). The core idea is to establish the Txt2Emote benchmark combining both explicit facial cues and implicit situational context, and introduce a dedicated <Expr> expression grounding token within a multimodal LLM to decode language representations directly into disentangled FLAME parameters via multimodal supervision.

Method¶

Overall Architecture¶

EmoteGPT centers around a multimodal large language model backbone coupled with a lightweight expression decoder head to establish an end-to-end mapping from text to 3D facial expression parameters. Given input prompts—whether explicit descriptions of facial physical details, implicit situational scenarios, or face images—the model processes the multimodal inputs and autoregressively predicts an output sequence containing a specialized <Expr> token. The model then extracts the top-layer hidden state associated with the <Expr> token and routes it through a lightweight MLP expression decoder head \(\eta\). This head decodes the semantic vector into 50-dimensional FLAME expression parameters \(\hat{\psi}\) and 6D jaw rotation parameters \(\hat{\theta}_{\text{jaw}}\), which are combined with an identity-neutral head template to synthesize expressive 3D meshes and drive downstream avatar rendering pipelines like GaussianAvatars, HeadStudio, and DiffusionRig.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Natural Language Input<br/>Explicit Facial Cues / Implicit Scenarios"] --> B["Dual-Granularity Text & Multimodal Alignment"]
    B --> C["Dedicated Expression Token Semantic Grounding"]
    C --> D["Lightweight FLAME Expression Decoding & Geometric Supervision"]
    D --> E["Downstream Expressive 3D/2D Avatar Synthesis"]

Key Designs¶

1. Dual-Granularity Text & Multimodal Alignment: Bridging Linguistic Scarcity and Situational Abstraction

To resolve the lack of paired text-expression data and bridge the gap between high-level emotional context and anatomical muscle deformations, the authors construct Txt2Emote, comprising 30k expressive face samples with balanced emotion coverage from AffectNet. 3D face geometries and ground-truth expression parameters are extracted using the state-of-the-art EMOCAv2 tracker, while GPT-4o generates paired descriptions under two complementary paradigms: (1) explicit descriptions detailing observable facial actions (e.g., lip stretch, eyebrow lift, eye narrowing) and overarching emotional impression, and (2) implicit descriptions depicting everyday situations or dramatic events that naturally elicit the corresponding expression. During training, EmoteGPT is supervised not only on text-to-expression pairs, but also on large-scale image-to-expression pairs (from CelebA, AffectNet, FFHQ) and general instruction-following VQA data (LLaVA-1.5-mix665k), thereby aligning the LLM's latent conceptual space with physical facial geometry.

2. Dedicated Expression Token Semantic Grounding: Seamlessly Embedding Continuous Geometry in Discrete Language Space

Autoregressively generating continuous numerical floats via standard language tokens is computationally inefficient and prone to compounding drift and quantization error. Instead, EmoteGPT treats 3D facial expressions as a distinct output modality and expands the MLLM vocabulary with a dedicated token <Expr>. Prompted with an instruction template (e.g., "USER: {description}, can you give the FLAME expression parameters of this person? ASSISTANT: Sure, it is <Expr>."), the model emits the <Expr> token at the designated response position. The 4096-dimensional hidden state \(h_{\langle\text{Expr}\rangle} \in \mathbb{R}^{4096}\) is extracted as a dense representation of the expression semantics. Compared to mean pooling across all text tokens (which dilutes localized expression signals) or the sequence-terminating <EOS> token (which conflates grammatical halting with geometric content), the isolated <Expr> token serves as an uncorrupted semantic anchor dedicated exclusively to 3D expression parameters.

3. Lightweight FLAME Expression Decoding & Geometric Supervision: Disentangled Representation and Sub-Millimeter Facial Fidelity

To project the high-dimensional hidden vector into a controllable, identity-independent geometric space, EmoteGPT employs a lightweight MLP decoder \(\eta\) configured with hidden layers [5120, 5120, 50] and GeLU activations. The decoder outputs 50-dimensional FLAME expression parameters \(\hat{\psi} = \eta(h_{\langle\text{Expr}\rangle})\) and 6D jaw rotations \(\hat{\theta}_{\text{jaw}}\). During synthesis, identity shape coefficients \(\beta\) and head pose parameters are held strictly at zero, completely isolating expression from facial morphology. To preserve delicate muscular deformations, training optimizes both a parameter-space \(L_2\) regression loss and a region-weighted vertex-level geometric loss in 3D mesh space:

\[\mathcal{L}_{\text{mesh}} = \| w \odot (M(\psi) - M(\hat{\psi})) \|_1\]

where \(M(\cdot)\) denotes the differential FLAME parameter-to-mesh function, and \(w\) represents region-dependent vertex weights emphasizing the high-dynamics facial zones—such as the mouth, lips, eyes, and eyebrows—relative to static skull areas.

Loss & Training¶

EmoteGPT undergoes end-to-end joint optimization. LoRA is applied to fine-tune the Vicuna-v1.5 language backbone within LLaVA-1.5-7B, while the expression decoder head \(\eta\) is trained from scratch; the visual encoder and other components remain frozen.

The overall training loss combines cross-entropy for text generation, parameter regression error, and vertex geometric distance:

\[\mathcal{L} = \lambda_{\text{txt}} \mathcal{L}_{\text{CE}}(\mathbf{y}_{\text{txt}}, \hat{\mathbf{y}}_{\text{txt}}) + \lambda_{\text{expr}} \|\psi - \hat{\psi}\|_2^2 + \lambda_{\text{mesh}} \| w \odot (M(\psi) - M(\hat{\psi})) \|_1\]

Hyper-parameters are configured as \(\lambda_{\text{txt}} = 1.0\), \(\lambda_{\text{expr}} = 1.0\), and \(\lambda_{\text{mesh}} = 0.01\). Optimization runs on 8 NVIDIA A40 GPUs using DeepSpeed ZeRO with the AdamW optimizer (learning rate \(4\times 10^{-5}\), WarmupDecayLR with 100 warm-up steps, per-GPU batch size 8, and 4 gradient accumulation steps).

Key Experimental Results¶

Main Results¶

Quantitative evaluations are conducted on the 2.5k held-out test split of Txt2Emote across explicit and implicit text benchmarks. Semantic emotional fidelity is assessed via a pretrained 3D emotion recognition network measuring Valence Concordance Correlation Coefficient (V-CCC), Arousal CCC (A-CCC), and discrete Emotion Accuracy (E-ACC). Geometric precision is quantified via \(L_1\) point cloud distances (in millimeters) computed over the full head (\(L_1^{\text{Head}}\)), the inner face (\(L_1^{\text{Face}}\)), and the lip region (\(L_1^{\text{Lip}}\)).

Training Supervision	Evaluation Benchmark	Method	V-CCC ↑	A-CCC ↑	E-ACC ↑	\(L_1^{\text{Head}}\) ↓	\(L_1^{\text{Face}}\) ↓	\(L_1^{\text{Lip}}\) ↓
Unsupervised / Static	Explicit Prompts	FLAME (Neutral)	-	-	-	0.83	1.41	3.53
Explicit Only	Explicit Prompts	Describe3D	0.48	0.34	0.34	-	-	-
Explicit Only	Explicit Prompts	CLIP Baseline	0.61	0.57	0.51	0.66	1.06	2.46
Explicit Only	Explicit Prompts	Vicuna-1.5-7B	0.56	0.53	0.51	0.66	1.08	2.48
Explicit Only	Explicit Prompts	EmoteGPT (Ours)	0.61	0.50	0.51	0.61	1.00	2.36
Explicit + Implicit	Explicit Prompts	CLIP Baseline	0.64	0.56	0.52	0.65	1.07	2.50
Explicit + Implicit	Explicit Prompts	Vicuna-1.5-7B	0.60	0.55	0.52	0.65	1.08	2.50
Explicit + Implicit	Explicit Prompts	EmoteGPT (Ours)	0.68	0.62	0.59	0.60	0.98	2.29
Explicit Only	Implicit Prompts (Zero-shot)	CLIP Baseline	0.36	0.28	0.32	0.84	1.45	3.70
Explicit Only	Implicit Prompts (Zero-shot)	Vicuna-1.5-7B	0.47	0.34	0.38	0.80	1.35	3.37
Explicit Only	Implicit Prompts (Zero-shot)	EmoteGPT (Ours)	0.49	0.44	0.41	0.74	1.30	3.27
Explicit + Implicit	Implicit Prompts	CLIP Baseline	0.49	0.41	0.41	0.73	1.24	3.04
Explicit + Implicit	Implicit Prompts	Vicuna-1.5-7B	0.49	0.39	0.41	0.71	1.21	2.94
Explicit + Implicit	Implicit Prompts	EmoteGPT (Ours)	0.57	0.48	0.44	0.66	1.09	2.69

When evaluated against text-to-3D diffusion heads (HumanNorm and Portrait3D) on fitted FLAME meshes, EmoteGPT demonstrates superior geometric accuracy: on explicit text, EmoteGPT achieves inner face and lip errors of 1.01 and 2.37 (vs. Portrait3D's 1.51 and 3.64); on implicit prompts, EmoteGPT achieves 1.19 and 2.93 (vs. Portrait3D's 1.74 and 4.43). In a perceptual user study with 25 computer graphics practitioners across 20 prompts, EmoteGPT was favored in 75% of evaluations, compared to only 14% for Portrait3D.

Ablation Study¶

1. Impact of Training Data Modalities:

Text Data	VQA Instructions	Face Images	Evaluation Set	V-CCC ↑	A-CCC ↑	E-ACC ↑	\(L_1^{\text{Head}}\) ↓	\(L_1^{\text{Face}}\) ↓	\(L_1^{\text{Lip}}\) ↓
✓	✗	✗	Explicit	0.58	0.56	0.52	0.64	1.06	2.49
✓	✓	✗	Explicit	0.66	0.61	0.56	0.60	1.02	2.32
✓	✓	✓	Explicit	0.68	0.62	0.59	0.60	0.98	2.29
✓	✗	✗	Implicit	0.55	0.37	0.41	0.71	1.21	2.98
✓	✓	✗	Implicit	0.55	0.46	0.42	0.67	1.15	2.73
✓	✓	✓	Implicit	0.57	0.48	0.44	0.66	1.09	2.69

2. Architecture Design of Expression Representation:

Grounding Design	Explicit \(L_1^{\text{Head}}\) ↓	Explicit \(L_1^{\text{Face}}\) ↓	Explicit \(L_1^{\text{Lip}}\) ↓	Implicit \(L_1^{\text{Head}}\) ↓	Implicit \(L_1^{\text{Face}}\) ↓	Implicit \(L_1^{\text{Lip}}\) ↓
Mean Pooling	0.62	1.00	2.35	0.68	1.15	2.85
`<EOS>` Token	0.75	1.21	3.05	0.84	1.43	3.93
`<Expr>` Token (Ours)	0.60	0.98	2.29	0.66	1.09	2.69

Key Findings¶

Multimodal Visual Co-supervision Drives Situational Generalization: Incorporating face images alongside general VQA instructions boosts arousal correlation (A-CCC) on implicit prompts from 0.37 to 0.48, and substantially compresses lip reconstruction error from 2.98 to 2.69 mm. Grounding linguistic concepts in real face visual textures enhances the model's physical understanding of emotional intensity.
Dedicated Token Outperforms Global Text Pooling: The dedicated <Expr> token sharply reduces geometric errors relative to the <EOS> token (e.g., implicit lip error drops from 3.93 to 2.69 mm). Isolating expression extraction from sentence completion avoids interference from trailing syntax.
Robust Zero-shot Situational Extrapolation: Even when trained exclusively on explicit descriptions, EmoteGPT achieves higher emotion accuracy on implicit prompts (E-ACC 0.41) than an LLM baseline trained on both explicit and implicit texts (0.39), highlighting the strength of the underlying foundation model for cross-modal reasoning.

Highlights & Insights¶

Embedding Continuous 3D Morphable Parameters within Discrete MLLMs: EmoteGPT bridges continuous 3D geometric control and discrete language generation by mapping the hidden representation of <Expr> via a lightweight 3-layer MLP, avoiding computationally heavy iterative sampling while delivering single-pass deterministic inference.
Intuitive Situational Grounding via Txt2Emote: By pairing everyday situations with corresponding 3D facial muscle geometries, the framework eliminates the requirement that end-users possess anatomical or blendshape expertise to manipulate digital human expressions.
Modular Compatibility with 3D/2D Avatar Pipelines: By regressing standardized, identity-isolated FLAME parameters, EmoteGPT integrates seamlessly into modern 3D Gaussian Splatting and 2D diffusion editing frameworks (GaussianAvatars, HeadStudio, DiffusionRig) for identity-preserving, text-driven animation.

Limitations & Future Work¶

Expressive Ceiling of Linear Statistical Models: FLAME relies on linear blendshapes and PCA identity spaces, which struggle to capture fine dynamic skin creases, wrinkles, and non-linear localized muscle deformations across varied demographics.
Reconstruction Artifacts in Pseudo-Ground-Truth: Training targets depend on monocular 3D face estimation (EMOCAv2), which may introduce tracking jitter and depth ambiguities under extreme poses or heavy occlusions.
LLM-Induced Cultural and Stylistic Bias: The linguistic descriptions in Txt2Emote are synthesized via GPT-4o, which may reflect cultural biases in situational interpretation. Future efforts could expand to multi-view video captures of real human interactions with multicultural grounding.

vs Describe3D / CLIP-Face: Early CLIP-based approaches rely on global image-text cosine similarity optimization that often collapses into sub-optimal local minima; EmoteGPT utilizes deep causal language representations, advancing emotion accuracy from 0.34 to 0.59.
vs HumanNorm / Portrait3D: Diffusion-based 3D head generators entangle identity geometry, hair, and texture with expressions, requiring minutes of iterative sampling per prompt; EmoteGPT decouples expressions via FLAME parameters and operates in single-pass real time.
vs LISA / ChatPose: EmoteGPT adapts the specialized task-token paradigm from 2D segmentation and human skeletal pose estimation to continuous 3D facial deformation spaces, establishing a scalable interface for emotional embodied agents.

Rating¶

Novelty: ⭐⭐⭐⭐☆ Introduces the dedicated <Expr> token mechanism and the Txt2Emote benchmark with dual explicit/implicit annotations.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across geometric distances, emotion recognition metrics, diffusion baselines, user studies, and downstream avatar integrations.
Writing Quality: ⭐⭐⭐⭐⭐ Rigorous organization, transparent experimental design, and clear problem motivation.
Value: ⭐⭐⭐⭐⭐ Establishes a highly efficient bridge between MLLMs and 3D digital human facial control with immediate utility for virtual production and games.