CtrlCoMo: Controllable Co-Speech Motion Generation with Gesture–Action Disentanglement

Conference: ECCV 2026
Paper: ECCV 2026
Area: Human Understanding
Keywords: motion generation, co-speech gesture, gesture-action disentanglement, multi-scale residual quantization, multimodal control

TL;DR

Addressing the interference between co-speech gestures and semantic actions, CtrlCoMo introduces Pyramid-VQ with layer-wise decaying CLIP semantic regularization to hierarchically separate low-frequency actions from high-frequency gestures, enabling text-driven action synthesis and explicit AdaLN-based gesture intensity modulation.

Background & Motivation

Synthesizing human motion synchronized with speech audio is a fundamental pillar for creating expressive, realistic virtual avatars and embodied agents. Existing co-speech motion generation models have made notable strides in producing audio-aligned, rhythmically coherent gestures. However, in real-world scenarios, humans rarely engage solely in beat-synchronized gesturing; rather, they perform concrete, context-specific actions while talking—such as typing on a keyboard, pouring a cup of tea, handling objects, or adjusting glasses. Contemporary co-speech datasets (e.g., TalkSHOW, BEAT) are almost exclusively gathered from TED talks or broadcast news where full-body movement degenerates into standing conversational gestures. Conversely, classic text-to-motion datasets (e.g., HumanML3D, KIT-ML) feature semantically diverse activities but entirely lack synchronized speech audio.

The primary technical bottleneck in generating composite co-speech actions lies in the severe spatial and temporal interference between spontaneous gestures and intentional actions. When a person speaks while playing an instrument or preparing food, speech-driven gestures naturally overlap and interfere with the ongoing mechanical action. This entwinement confuses the correspondence between linguistic action semantics and acoustic rhythm, severely degrading text controllability. Standard Residual Vector Quantization (RVQ) architectures stack multiple quantization layers to encode residuals, but lack explicit temporal or functional constraints, causing action semantics and rhythmic gestures to mix unpredictably across codebook levels.

This paper is built on the key observation that co-speech gestures predominantly comprise temporally localized, high-frequency movements, whereas semantic actions exhibit low-frequency, globally coherent temporal patterns across the entire skeleton. Core idea: develop a multi-scale quantization architecture termed Pyramid-VQ that enforces temporal resolution constraints across quantization depths alongside a layer-wise decaying CLIP semantic regularizer to disentangle global actions into shallow layers and localized gestures into deeper layers, complemented by AdaLN-Zero modulation for explicit, continuous gesture intensity control.

Method

Overall Architecture

The CtrlCoMo pipeline consists of two principal components: a motion tokenizer based on Pyramid-VQ-VAE that discretizes and disentangles continuous motions into multi-scale token representations, and a cross-modal hierarchical Autoregressive (AR) Transformer generator. Independent Pyramid-VQ tokenizers are trained for the body and the hands to accommodate scale discrepancies between upper-body postures and dexterous finger articulations. During tokenization, Pyramid-VQ quantizes the motion sequence across an \(L\)-level residual codebook hierarchy from coarse to fine. During generation, the AR Transformer predicts discrete motion tokens conditioned on text embeddings and speech acoustic features (MFCC) while incorporating a scalar gesture intensity factor via AdaLN-Zero layers. Finally, the decoded tokens reconstruct full 3D SMPL-X motion sequences.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Motion Sequence X"] --> B["Multi-scale Pyramid-VQ<br/>Coarse-to-fine temporal downsampling"]
    B --> C["Layer-wise Decaying Semantic Regularization<br/>Shallow semantic guidance / Deep unconstrained"]
    C --> D["Disentangled Discrete Tokens<br/>Hierarchical residual codebooks"]
    E["Text Prompt + Speech MFCC"] --> F["Cross-Modal Hierarchical AR Generation<br/>Coarse-to-fine token autoregression"]
    D -.->|Codebook supervision| F
    G["Gesture Intensity Control Scalar"] --> H["AdaLN-Zero Adaptive Modulation<br/>Scale and shift latent activations"]
    H --> F
    F --> I["Decoded 3D SMPL-X Motion"]

Key Designs

1. Multi-scale Residual Quantization (Pyramid-VQ): Enforcing frequency-based gesture-action separation via temporal scales Standard RVQ models run every quantization layer at the same temporal resolution, which allows high-frequency rhythmic motion and low-frequency action semantics to leak into arbitrary layers. Pyramid-VQ overcomes this by binding each quantization level \(d \in \{1, 2, \dots, L\}\) to an increasing temporal resolution \(T_1 \le T_2 \le \dots \le T_L = T\). Given the residual from the prior level \(R^{(d)}\), the latent is temporally downsampled via 1D linear interpolation to scale \(T_d\), quantized against layer codebook \(\mathcal{C}_d\), upsampled back to length \(T\), and subtracted from the residual:

\[R^{(d+1)} = R^{(d)} - \text{Upsample}_d(\hat{Z}^{(d)})\]

Temporal downsampling serves as an intrinsic low-pass filter, forcing the shallowest VQ levels to capture slowly evolving global posture trajectories and coarse action dynamics. As the residual is passed down the hierarchy, deeper layers with finer temporal granularities dedicate their capacity to fine-grained, localized gestures. Furthermore, because global actions require a diverse vocabulary to span open-world verbs while local gestures exhibit higher structural repetitiveness, codebook sizes decay monotonically across depth (\(K_1 > K_2 > \dots > K_L\)), perfectly matching representation capacity with temporal scale.

2. Layer-wise Decaying Semantic Regularization: Directing shallow codebooks toward textual semantics Temporal downsampling alone cannot fully prevent linguistic semantics from leaking into deeper layers or acoustic rhythms from invading shallow layers. To guarantee clear semantic boundaries, CtrlCoMo applies layer-wise decaying semantic regularization using a pre-trained CLIP text encoder. At depth \(d\), the quantized representation \(\bar{Z}^{(d)}\) is averaged along the temporal dimension to obtain summary vector \(\tilde{z}^{(d)}\), and its cosine similarity with the CLIP text embedding \(Z_{\text{clip}}\) is optimized:

\[\mathcal{L}_{\text{semantic}}^{(d)} = -\lambda_d \cdot \text{sim}(Z_{\text{clip}},\, \tilde{z}^{(d)})\]

The regularization strength \(\lambda_d\) decays progressively with increasing depth: \(\lambda_d = \left(1 - \frac{d-1}{L}\right)^\gamma\). Consequently, the shallowest layers receive maximum linguistic supervision to anchor the global semantic action, while the deepest layers have zero semantic penalty, allowing them to capture unconstrained, spontaneous, speech-correlated gestures.

3. Cross-Modal Hierarchical Autoregressive Generation: Synchronized body-hand synthesis across scales The autoregressive generation phase uses a 12-layer decoder-only Transformer. For both body and hands, multi-level discrete tokens are unrolled into a 1D sequence in a strict coarse-to-fine and temporal left-to-right raster order. A causal attention mask permits each position to attend to all tokens from coarser pyramid levels alongside preceding tokens at the current level. Two specialized convolutional heads project the shared latent state into categorical distributions over body and hand codebooks, maintaining anatomical coordination between torso articulation and finger gestures.

4. Explicit Gesture Intensity Control via Reconstruction Residuals and AdaLN: Continuous modulation of expressive range Because Pyramid-VQ effectively decouples actions into shallow levels and gestures into deep levels, partial reconstruction using only the first \(l\) layers (\(\hat{X}^{(l)}\)) reconstructs the core action while filtering out gestures. Hence, the difference in reconstruction error between shallow layers and the full stack accurately quantifies gesture amplitude:

\[\text{Intensity} = \mathcal{E}_{L//2} - \mathcal{E}_{L}\]

where \(\mathcal{E}_d = \|X - \hat{X}^{(d)}\|_F^2\). During training, this scalar metric is normalized into \([0, 1]\) and injected into the Transformer layers via AdaLN-Zero (Adaptive Layer Normalization) blocks to modulate layer activations. By randomly dropping the intensity input with probability \(p_{\text{intensity}} = 0.35\), the network supports both default prior generation and user-specified continuous scaling (e.g., \(0.0\) to \(2.0\)) at test time without distorting the underlying action.

Loss & Training

The overall training objective of the Pyramid-VQ tokenizer combines reconstruction loss \(\mathcal{L}_{\text{rec}}\) (joint position \(L_1\), velocity, and kinematic constraints), codebook commitment loss, and layer-wise semantic regularization:

\[\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{rec}} + \sum_{d=1}^{L} \mathcal{L}_{\text{commit}}^{(d)} + \sum_{d=1}^{L} \mathcal{L}_{\text{semantic}}^{(d)}\]

Codebooks are updated using Exponential Moving Average (EMA). The configuration uses \(L=10\) levels with temporal scales \(\{2, 2, 3, 4, 5, 6, 7, 8, 11, 22\}\) for an input sequence of 176 frames (\(176 / 8 = 22\)). Codebook capacity scales down every two layers from 512 to 32. Both Pyramid-VQ and the AR generator are trained for 250k steps using AdamW on 4 NVIDIA A800 GPUs with a learning rate of \(1\times 10^{-4}\).

Key Experimental Results

The models are evaluated on the newly constructed CoHuMo dataset, which provides 370 hours of action-rich co-speech motions across 20+ real-world scene categories and 5+ languages with high-fidelity SMPL-X annotations and motion text captions.

Main Results

CtrlCoMo is benchmarked against established co-speech gesture generation baselines. Standard metrics include Fréchet Gesture Distance (FGD ↓), Mean Average Joint Error (MAJE ↓), Multimodal Distance (MM-Dist ↓), and R-Precision retrieval accuracy within batches of 32 (Top-1/2/3 ↑).

Method	FGD ↓	MAJE ↓	MM-Dist ↓	R-Precision Top-1 ↑	R-Precision Top-2 ↑	R-Precision Top-3 ↑
TalkSHOW	22.98	10.727	1.287	0.245	0.366	0.448
Syntalker	80.57	9.136	1.306	0.174	0.273	0.353
SemGes	183.74	9.467	1.306	0.180	0.284	0.366
Probtalk	29.85	9.288	1.270	0.310	0.444	0.532
CtrlCoMo (Ours)	22.27	8.932	1.257	0.373	0.520	0.608

Ablation Study

The ablation study confirms the individual contributions of multi-scale residual quantization (Pyramid-VQ) and layer-wise semantic regularization (SemReg). "Res-VQ" denotes a baseline utilizing a flat residual VQ with uniform temporal resolutions.

Config	FGD ↓	MAJE ↓	MM-Dist ↓	R-Precision Top-1 ↑	R-Precision Top-2 ↑	R-Precision Top-3 ↑	Note
Res-VQ	109.63	9.342	1.274	0.308	0.441	0.528	Uniform resolution causes severe gesture-action entanglement
Pyramid-VQ	23.61	9.252	1.235	0.344	0.484	0.574	Temporal pyramid drops FGD dramatically from 109.63 to 23.61
Pyramid-VQ + SemReg (Full model)	22.27	8.932	1.257	0.373	0.520	0.608	Decaying CLIP regularization improves Top-1 precision by +2.9%

Ablations on conditioning modalities show that removing text conditions at inference collapses R-Precision Top-1 to 0.102, proving the necessity of text for governing action semantics. Omitting audio input spikes FGD to 109.42, demonstrating the indispensable role of acoustic signals in driving natural rhythmic kinematics. In addition, scaling training data from 10k to 200k samples yields monotonic gains across all metrics (FGD improves from 38.15 to 22.27, Top-1 precision increases from 0.163 to 0.373).

Key Findings

• Multi-scale temporal downsampling is crucial for physical disentanglement: Replacing Pyramid-VQ with standard Res-VQ leads to catastrophic degradation in FGD (jumping from 22.27 to 109.63). Without temporal bottlenecks, high-frequency speech gestures corrupt all quantization levels.
• Decaying semantic guidance guarantees clean action fidelity: The addition of SemReg pushes Top-1 R-Precision to 0.373 and Top-3 to 0.608, ensuring that the primary action requested in the text prompt is accurately executed without being obscured by rhythmic movements.
• Independent, monotonic gesture amplitude modulation: Qualitative inspection verifies that adjusting the input intensity scalar across \([0.0, 2.0]\) smoothly expands or shrinks gesture amplitude while holding the underlying action (such as grasping a teapot or raising a cup) completely intact.

Highlights & Insights

• Adapting visual autoregressive scale pyramids to kinematic disentanglement: By treating temporal downsampling as an inherent low-pass filter, the framework decomposes complex human motion into frequency-isolated representations without relying on cumbersome manual joint-hierarchy heuristics.
• Unsupervised intensity quantification from residual reconstruction gaps: Bypassing external emotion classifiers or manual intensity labels, the model formulates gesture amplitude directly as the energy discrepancy between shallow action approximations and full-stack reconstructions.
• Bridging the chasm between text-to-motion and speech-to-gesture: Introducing the 370-hour CoHuMo dataset and the dual-branch controllable generator establishes an unified benchmark for generating rich actions embedded in conversational contexts.

Limitations & Future Work

• Lack of lower-body locomotion and physical contact modeling: The model is restricted to upper-body and hand articulations using SMPL-X, leaving whole-body walking trajectories, foot-ground contacts, and rigid object affordances unaddressed.
• Global intensity scalar lacks spatial and temporal granularity: The gesture intensity control operates as a sequence-level uniform scalar, preventing localized modulation (e.g., exciting one hand while resting the other) or dynamic beat-level emphasis.
• Future Directions: Integrating whole-body root trajectory navigation and 3D scene interaction constraints into the multi-scale residual hierarchy to empower digital humans to traverse complex environments while conversing and manipulating tools.

Related Work & Insights

• vs TalkSHOW / EMAGE: Existing co-speech models map audio directly into conversational postures without strong text-driven action conditioning; CtrlCoMo enables simultaneous execution of semantic actions and speech-aligned gestures.
• vs HumanML3D / T2M-GPT: Pure text-to-motion methods lack audio conditioning and cannot generate synchronized speech gestures; CtrlCoMo bridges both modalities through dual cross-attention and hierarchical scheduling.
• vs Res-VQ / RVQ-VAE: Conventional RVQ lacks structural inductive biases; CtrlCoMo demonstrates that binding residual layers to distinct temporal resolutions and semantic losses enforces structured feature disentanglement.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Pioneering solution to gesture-action interference in co-speech scenarios via principled multi-scale quantization and residual intensity modeling.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Introduces a landmark 370-hour benchmark dataset (CoHuMo) alongside rigorous metric evaluations, multi-modal ablations, and qualitative visualizations.
• Writing Quality: ⭐⭐⭐⭐⭐ Crisp narrative arc, mathematically elegant problem formulation, and highly intuitive conceptual illustrations.
• Value: ⭐⭐⭐⭐⭐ Substantial practical value for realistic 3D virtual avatars, video game NPCs, and human-robot interaction systems.