SICAGE: Speaker-Independent Culture-Aware Gesture Generation using TED4C-L Dataset¶

Conference: ECCV 2026
arXiv: 2606.30001
Paper: Project Page
Code: https://arielgjaci.com/sicage (including dataset and source code)
Area: Human Understanding / Gesture Generation
Keywords: Gesture Generation, Culture-Awareness, Domain Generalization, Diffusion Models, Speaker-Independent

TL;DR¶

SICAGE formulates culture-aware co-speech gesture generation as a domain generalization problem: treating each speaker as a domain, it extracts speaker-independent cultural representations from audio-visual and textual modalities using Fishr regularization or adversarial learning. These representations then condition a real-time diffusion generator, ALaDiT, significantly improving gesture realism, diversity, beat synchronization, and cultural consistency on the self-built 106-hour, four-culture TED4C-L dataset.

Background & Motivation¶

Co-speech gestures are a crucial modality of human communication, working in tandem with speech to convey semantics and emotion. Existing data-driven gesture generation methods—ranging from rule-based systems to diffusion models—have progressed rapidly in motion quality and diversity, but almost all neglect the cultural dimension. Cultural norms profoundly influence the expression and interpretation of gestures, yet current models assume homogeneous gesture styles across all speakers, presenting a critical blind spot in cross-cultural human-computer interaction scenarios.

A more insidious issue lies in the evaluation protocols. In the few datasets annotated with cultural labels, the training and test sets often contain different segments from the same speakers (speaker-dependent split). Under this setting, models might merely memorize the gesture styles of individual speakers rather than truly learning group-level cultural patterns—an effect that appears "culture-aware" on the surface but is actually confounded by speaker identity. Therefore, a reliable evaluation of cultural generalization must be conducted on speaker-disjoint splits, requiring a dataset large enough to maintain statistical power after speaker-based partitioning.

Addressing these conflicts, this work reformulates cultural representation learning as a Domain Generalization (DG) problem: each speaker is treated as a domain, and the goal is to learn cultural embeddings that generalize across speakers—remaining discriminative for cultural categories while suppressing dependence on speaker identity. To this end, the authors construct TED4C-L (106 hours, 764 speakers, 4 cultures) and design a modular framework, SICAGE. Within this framework, Fishr regularization matches the gradient variances across speaker domains to achieve speaker invariance, and the ALaDiT diffusion generator synthesizes real-time gestures conditioned on cultural embeddings. Core Idea: Cultural gesture generation should not memorize individual speakers, but rather learn generalizable cultural representations from "cross-individual patterns shared among different speakers."

Method¶

Overall Architecture¶

SICAGE is a three-stage modular framework: it first extracts multimodal features (audio, text, pose) from the TED4C-L dataset, then learns speaker-independent cultural embeddings via domain generalization methods (Fishr or adversarial learning), and finally feeds these embeddings along with the original features into the ALaDiT diffusion generator to synthesize 4-second culture-aware gesture sequences. The three stages are decoupled, allowing both the cultural embedding module and the generator to be replaced with alternative implementations.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["TED4C-L Dataset<br/>764 Speakers × 4 Cultures"] --> B["Multimodal Feature Extraction<br/>Audio/Text/Pose"]
    B --> C["Cultural Embedding Learning<br/>Fishr Regularization"]
    B --> D["ALaDiT Diffusion Generator"]
    C --> D
    D --> E["4-Second Culture-Aware Gesture Sequence"]

Key Designs¶

1. Speaker-Independent Cultural Representation Learning: Treating Each Speaker as a Domain

Training a classifier directly on coarse culture labels causes the model to learn speaker-specific gesture features, lacking cross-speaker generalization capability. The core operation of SICAGE is treating each speaker as an independent domain, using domain generalization objectives to force cultural embeddings to be insensitive to speaker identity.

Specifically: a feed-forward network (FFN) is trained using audio features (wav2vec, mel-log, onset strength) and text features (LaBSE sentence embeddings) as inputs. Each modality passes through an independent branch with attention pooling, is concatenated, and is then projected by a fusion layer into a 512-dimensional cultural embedding, which is finally fed into a culture classification head. Note that pose features are intentionally excluded here—because the target pose is the output to be generated and is unavailable during inference.

Domain generalization is implemented in two ways: (1) Fishr Regularization: matches the within-domain gradient variances of each speaker, with the penalty term \(P(\theta) = \frac{1}{|\mathcal{S}|}\sum_{d\in\mathcal{S}} \|v^d - \bar{v}\|^2\), where \(v^d\) is the variance of sample gradients within domain \(d\). In each step, \(k=64\) speakers are sampled with 16 samples per domain. A warmup with \(\lambda_p=0\) is applied for the first 500 steps, after which it increases to 1000. (2) Adversarial Learning: adds a Gradient Reversal Layer (GRL) and a speaker classification head after the shared encoder, with the total loss formulated as \(\mathcal{L}_{tot} = \mathcal{L}_{cult} - \lambda_2 \cdot \mathcal{L}_{spk} + \lambda_s \mathcal{L}_{SupCon}\), incrementally increasing the adversarial weight. Both schemes additionally apply a supervised contrastive loss \(\mathcal{L}_{SupCon}\) (\(\tau=0.07\), \(\lambda_s=0.2\)) to pull same-culture samples closer and push different-culture samples apart, further reinforcing speaker invariance.

The advantage of Fishr lies in the fact that the second-order statistics of gradient variance are more stable than the min-max game of adversarial training, especially when the number of speakers is large (764 domains). It eliminates the need to maintain an independent classification head for each speaker, requiring only the matching of gradient variance distributions across domains, which yields higher training efficiency.

2. ALaDiT: Hierarchical Transformer Diffusion Generator Conditioned on Cultural Embeddings

ALaDiT is a real-time diffusion model capable of generating 4-second gestures within 14 ms. Its inputs comprise six modalities: mel-log spectrogram, onset strength, wav2vec embedding, sentence embedding, a 1-second seed pose (\(X_0^{seed}\), 5 tokens), and a cultural embedding (\(X^{cu}\)).

The core of the architecture is a 10-layer hierarchical Transformer, where each layer sequentially performs: (a) Self-Attention—modeling temporal dependencies on the concatenated sequence of the noisy target pose \(X_t^{fin}\) (20 tokens) and the seed pose; (b) Cross-Attention—taking the aligned low-level audio features \(X^{low}\) (mel-log + onset + wav2vec, downsampled via windowed attention pooling to set a one-to-one correspondence with the pose tokens) as key/value, allowing the pose to attend to speech rhythm and prosody; (c) AdaIN Condition Injection—injecting high-level semantic features \(X^{high}\) (the concatenated and projected sentence embedding + cultural embedding) into each layer of the Transformer via Adaptive Instance Normalization, regulating the semantic and cultural styles of gestures in a global manner.

This hierarchical conditioning design—"low-level features via cross-attention + high-level features via AdaIN"—is critical: the temporal alignment information of audio (beats, accents) requires frame-level, fine-grained interaction, which is best suited for cross-attention; whereas cultural style and semantic content are global attributes, making them ideal for overall adjustment at the feature distribution level via AdaIN. The final denoised poses are reconstructed into continuous motions of 9 upper-body 3D keypoints through a pre-trained VQVAE decoder.

3. Multimodal Alignment Loss: Ensuring Aligned Gestures with Audio, Text, and Culture

Training a diffusion model solely with a reconstruction loss \(\mathcal{L}_{rec}\) (Huber Loss) cannot explicitly guarantee that the generated gestures align with the conditioning features in semantic space—the model might learn statistically realistic motions that are irrelevant to the speech content. ALaDiT introduces three layers of alignment constraints:

Low-level Audio Alignment \(\mathcal{L}_{low} = \mathbb{E}[1 - \cos(z_o, z_l)]\): projects the generated pose and low-level audio features into a shared space and maximizes their cosine similarity, ensuring that gesture beats and prosody are synchronized with speech.
High-level Semantic Alignment \(\mathcal{L}_{high} = \mathbb{E}[1 - \cos(z_o, X^{high})]\): similarly matches the global semantic direction of the generated pose with the joint sentence and cultural embeddings.
Contrastive Loss \(\mathcal{L}_{cont}\): applies InfoNCE-style contrastive losses to both low-level and high-level branches, pulling matched pairs (pose-audio/semantic-pose from the same speech segment) closer and pushing unmatched pairs (from different samples) apart, preventing the model from collapsing to trivial solutions (such as all outputs pointing in the same direction).

Additionally, a cultural classification head is attached to the denoised pose tokens to impose a cross-entropy loss \(\mathcal{L}_{cult}\), directly supervising cultural consistency from the generation output level. This design conceptually forms a closed loop with the aforementioned "cultural classifier evaluation CE F1"—the generator is also optimizing the same objective internally.

A Complete Example: Cultural Gesture Generation for an Indian Speaker Speaking Hindi¶

Taking a 5-second clip from a Hindi TED talk as an example: (1) Audio features (wav2vec, mel-log, onset) are extracted, text is encoded into a 768-dimensional vector via LaBSE, and the first 1 second of pose is encoded into 5 discrete tokens via VQVAE to serve as a seed; (2) Audio and text features are fed forward through the Fishr FFN, outputting a 512-dimensional cultural embedding—this embedding aligns with the statistical pattern of Indian culture ("highly expressive, larger gesture amplitude") but excludes any individual style unique to the specific speaker; (3) ALaDiT starts from random noise \(X_T^{fin}\) and performs layer-by-layer denoising in the 10-layer Transformer: the cross-attention layer aligns pose tokens with the beat features of Hindi speech, while the AdaIN layer modulates the global motion style (biasing towards larger amplitudes and more arm waving) using the sentence semantics and cultural embedding; after 50 diffusion steps, it outputs 20 denoised tokens; (4) The VQVAE decoder reconstructs these into a 4-second continuous pose sequence—the final gestures yield distinct wrist movements at beat points and Hindi accents, with an overall amplitude greater than the Italian/Japanese styles. Moreover, generated results for different Hindi speakers statistically share these general cultural patterns, while retaining distinct individual pose details.

Loss & Training¶

The total loss of ALaDiT is defined as \(\mathcal{L} = \mathcal{L}_{rec} + 0.1 \cdot \mathcal{L}_{cult} + 0.1 \cdot \mathcal{L}_{low} + 0.1 \cdot \mathcal{L}_{high} + 0.01 \cdot \mathcal{L}_{cont}\), where \(\mathcal{L}_{rec}\) is the Huber Loss. The optimizer is AdamW (\(\beta_1=0.9\), \(\beta_2=0.999\)) with a learning rate of \(5\times 10^{-5}\) decaying by 0.5 every 100k steps, a batch size of 64, 50 diffusion steps, and EMA (\(\beta=0.999\)) for parameter tracking. Model selection is based on the checkpoint with the lowest FGD on the validation set (saved every 50k steps, evaluated up to 500k steps). VQVAE is trained independently for 300 epochs (Adam, learning rate \(10^{-4}\), decaying by 10 every 100 epochs) with a loss comprising reconstruction \(L_1\), commitment, velocity, acceleration, and temporal smoothness regularization. The Fishr cultural classifier is trained for 50 epochs (AdamW, \(lr=10^{-4}\), effective batch size 1024), warming up with \(\lambda_p=0\) for the first 500 steps and then linearly increasing to 1000. All models are trained and evaluated on speaker-disjoint train/val/test splits.

Key Experimental Results¶

Main Results¶

Model	FGD ↓	CE F1(%) ↑	BAS(%) ↑	SRGR(%) ↑	Diversity ↑
MDM/NC	15.58	38.57	22.52	51.62	107.62
MDM/FI	7.59	47.09	22.59	51.86	109.37
DSG+/NC	2.76	41.51	22.48	68.17	108.85
DSG+/FI+Align	2.52	39.80	22.67	65.17	111.13
ALaDiT/FI	1.03	44.61	22.63	68.09	110.27
ALaDiT/ADV	1.53	42.71	22.45	67.57	111.75
ALaDiT/NC	1.60	43.41	22.51	67.72	109.50

ALaDiT/FI significantly outperforms all variants and baselines on FGD (motion realism) with a score of 1.03 (with the best MDM at 7.59 and the best DSG+ at 2.52), illustrating that the combination of Fishr cultural embeddings, hierarchical attention, and multimodal alignment is highly effective in approximating the true motion distribution. The CE F1 reaches 44.61% (against a random baseline of 25%), proving that the generated gestures indeed encode cultural signals discriminable by a classifier. For DSG+, the FGD even deteriorates after injecting cultural embeddings (from 2.76 to 4.81), indicating that the effectiveness of cultural embeddings heavily depends on whether the generator architecture explicitly supports multimodal alignment.

Ablation Study¶

Configuration	FGD ↓	CE F1(%) ↑	SRGR(%) ↑	Description
ALaDiT/FI	1.03	44.61	68.09	Full Model
w/o Cultural Conditioning (NC)	1.60	43.41	67.72	Remove explicit cultural embedding, relying purely on implicit audio/text inference
w/o Domain Regularization (NoDG)	1.56	43.18	67.76	Same encoder architecture but without speaker domain regularization
w/o Alignment Loss (NoAlign)	1.36	43.37	68.17	Using Fishr embedding but removing ALaDiT's multimodal alignment losses
One-Hot Cultural Labels	1.63	43.73	67.63	Replace cultural embedding with one-hot labels projected via MLP
Adversarial Learning (ADV)	1.53	42.71	67.57	Replace Fishr with adversarial learning, other settings remain identical

The ablation study reveals a clear hierarchy of contributions: (1) Presence vs. Absence of Cultural Embeddings (NC \(\rightarrow\) FI, FGD decreases from 1.60 to 1.03) delivers the single largest gain, demonstrating that the improvement in motion realism from explicit cultural conditioning cannot be implicitly substituted by audio/text; (2) Necessity of Domain Regularization (NoDG \(\rightarrow\) FI, FGD 1.56 to 1.03): even with cultural embeddings, omitting speaker domain regularization causes the FGD to revert almost to the NC level (1.56 vs. 1.60), proving that having "discriminative embeddings for culture" is insufficient—speaker information must simultaneously be suppressed to make the embeddings truly generalizable; (3) Fishr vs. Adversarial Learning: FI outperforms ADV across all metrics, and the CE F1 of ADV (42.71) is even lower than that of NC (43.41), revealing that the min-max game of adversarial training is less stable than gradient variance matching under this multi-domain setting; (4) OneHot discrete cultural labels marginally improve diversity but underperform Fishr in both FGD and CE F1, indicating that cultural embeddings learned from continuous multimodal signals retain richer, more nuanced cultural information than hand-crafted grouped labels.

Key Findings¶

Fishr is the optimal domain generalization strategy for cultural representation learning: In the multi-speaker setting (764 domains), Fishr's gradient variance matching is more stable than the GRL in adversarial learning and avoids the need to maintain an independent classification head for each speaker. In the ablation study, FI consistently outperforms both ADV and NC across all generator architectures (ALaDiT / MDM / DSG+).
The efficacy of cultural embeddings heavily depends on the generator architecture: MDM and ALaDiT exhibit substantial drops in FGD (15.58 \(\rightarrow\) 7.59, 1.60 \(\rightarrow\) 1.03) after injecting FI embeddings, whereas DSG+'s behavior deteriorates instead. This indicates that cultural embeddings alone are not a silver bullet; they require the generator architecture to explicitly model multimodal alignment (cross-attention + AdaIN + contrastive loss) to be leveraged effectively.
Without alignment loss, the model still outperforms NC (NoAlign FGD=1.36 vs. NC=1.60), reflecting that the Fishr embeddings themselves carry signals beneficial for generation, while the alignment loss further unleashes their full potential (1.36 \(\rightarrow\) 1.03).
User Studies (N=20): FI significantly outperforms ADV in overall ratings (6.06 vs. 5.65, \(p=0.033\)) and significantly outperforms NC in cultural fit (6.16 vs. 5.81, \(p=0.038\)). This suggests that the differences in gestures brought by Fishr cultural embeddings are perceptible to human observers. FI is consistently rated best on Japanese and Italian samples, whereas NC is slightly but non-significantly higher on Turkish samples, indicating group heterogeneity in the perception of cultural effects.

Highlights & Insights¶

Formulating cultural gesture generation as domain generalization is conceptually neat: treating each speaker as a domain, and culture as a shared cross-domain label. This mapping allows transferring any tools from the domain generalization field (Fishr, adversarial learning, DG evaluation protocols) directly, and naturally demands speaker-disjoint splits, resolving the evaluation confounding issue long ignored in this field.
Gradient variance matching (Fishr) being more stable than adversarial training in multi-domain settings is a transferable, practical insight: when the number of domains is extremely large (hundreds) and samples per domain are scarce, maintaining an adversarial classification head for each domain is both costly and unstable. In contrast, Fishr matches only second-order statistics (gradient variance), decoupling training overhead from the domain count. This is suitable for any "individual-as-domain" scenarios (e.g., personalized gestures, multi-style TTS, cross-user behavior modeling).
Hierarchical conditioning design (cross-attention for temporal alignment + AdaIN for global style) presents a general paradigm for handling multimodal condition signals: fine-grained temporal signals (audio beats, text phonemes) go through cross-attention, while global style signals (culture, emotion, speaking style) utilize AdaIN. This division of labor prevents cross-attention from being overwhelmed by global information, while ensuring AdaIN does not lose fine-grained temporal details.
The value of the TED4C-L dataset goes beyond this paper: with 106 hours, four languages, and 764 speakers of multimodal data, coupled with a speaker-disjoint split design, it can support any style-aware or culture-aware gesture generation research requiring cross-speaker generalization evaluations.

Limitations & Future Work¶

Coarse-grained cultural labels: grouping cultures by country (India, Italy, Japan, Turkey) neglects vast intra-national cultural heterogeneity, which the authors acknowledge as a practical operational approximation. Future work could introduce finer-grained cultural annotations (regions, subcultures, social classes) or automatically discover cultural subgroups from data.
Only 9 upper-body keypoints: for tracking reliability in unconstrained TED videos where hands/fingers are often blurry or out-of-frame, the authors omitted finger and lower-body tracking. This limits gesture expressiveness, as finger movements are vital expressive carriers in many cultures. Future work could incorporate more complete body/hand keypoints when video quality permits.
Incomplete decoupling between cultural representations and language/prosody/semantics: the current approach learns embeddings that are "culture-discriminative + speaker-invariant" but does not guarantee causal disentanglement among cultural, linguistic, prosodic, and semantic signals. A "cultural embedding" of a Hindi speaker might partially encode the prosodic features of the Hindi language rather than pure cultural styles. Future work could explore causal representation learning methods for cleaner disentanglement.
Participants in the user study might not be familiar with all evaluated cultures: the authors noted uneven familiarity among subjects with different cultures, which might affect the reliability of perceptual ratings, particularly for Turkish samples where FI did not significantly outperform NC.
Perceptible gap remains between generated gestures and ground truth: in the user study, all metrics of the Real condition exceed those of the generative models, indicating that there is still room to improve the perceptual quality of gesture generation.

vs DiffuseStyleGesture+ (DSG+): DSG+ generates gestures conditioned on style features but lacks dedicated cultural modeling and speaker-invariance constraints. The core difference of SICAGE is that it formulates cultural representation learning as a domain generalization problem. Experiments also show that DSG+'s performance deteriorates after injecting cultural embeddings, indicating that the value of cultural embeddings heavily depends on explicit support for multimodal alignment from the generator architecture.
vs Culturally Annotated Datasets like MCGD: MCGD labels the cultural aspects of 263 speakers but is not publicly available and has only ~20 people per culture. In contrast, TED4C-L features around 190 speakers per culture, spans 106 hours, and uses a speaker-disjoint split. It substantially surpasses prior datasets in both scale and quality, establishing a more reliable benchmark for cultural gesture studies.
vs Motion Diffusion Model (MDM): MDM is a general framework for diffusion-based pose generation but lacks explicit designs for cultural conditions and multimodal alignment. Under identical diffusion steps and optimization settings, ALaDiT achieves a substantially lower FGD than MDM (1.03 vs. 7.59), validating the benefits of the hierarchical condition architecture and alignment losses.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Formulates culture-aware gesture generation strictly as a speaker-level domain generalization problem for the first time, offering a clear concept and unique perspective. The TED4C-L dataset fills the gap in large-scale multi-cultural gesture datasets.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive ablations (6 ALaDiT variants + 2 baselines × 3 conditions), complete chain of evidence including objective metrics (FGD/CE F1/BAS/SRGR/Diversity) + user studies (20 participants × 7 questions) + qualitative visualizations + per-culture analyses.
Writing Quality: ⭐⭐⭐⭐ The methodology is clear, experimental setups are detailed, and supplementary materials are rich (VQVAE architecture, data distribution, per-culture results), though the Introduction is slightly wordy.
Value: ⭐⭐⭐⭐⭐ Introducing domain generalization tools to co-speech gesture generation opens up a new research direction. The TED4C-L dataset holds independent ecological value as a benchmark. The speaker-invariance perspective is extendable to other style-aware generation tasks such as emotional or personalized gestures.