Multi-Modal Controlled Coherent Motion Generation¶

title: >- [Paper Note] Multi-Modal Controlled Coherent Motion Generation description: >- [ECCV2026][3D Vision][Motion Generation] Decouples multi-modal diffusion denoising across spatial body parts with per-step refinement and LLM motion planning to synthesize coherent 3D avatar motion without aligned multimodal data. tags: - ECCV2026 - 3D Vision - Motion Generation - Diffusion Models - Multimodal Control date: 2026-09-19 content_hash: bc4ae86da0fa4741

Conference: ECCV 2026
Paper: ECCV Official
Code: https://feifeifeiliu.github.io/MOCO
Area: 3D Vision
Keywords: human motion generation, diffusion models, multi-modal control, co-speech gesture, motion planning

TL;DR¶

Addressing the severe scarcity of aligned multimodal motion data for concurrent walking and talking behaviors, MOCO decouples diffusion denoising across spatial body divisions with per-step iterative refinement and LLM timeline planning, achieving coherent and synchronized full-body avatar motion.

Background & Motivation¶

Driving 3D digital avatars to perform lifelike human behaviors requires coordinating multiple concurrent control signals—such as executing locomotion along a scripted path, gesturing expressively to emphasize spoken words, and modulating body posture in response to textual prompts. While prior research has made great strides in single-modality generation tasks like text-to-motion or speech-to-gesture synthesis, real-world conversational agents and virtual presenters inherently demand simultaneous multi-modal control. Recent unified frameworks attempt to ingest multiple modalities by combining cross-task datasets, but when concurrent control inputs arrive, they typically process them in a sequential or poorly coordinated fashion.

The fundamental bottleneck obstructing simultaneous multi-modal motion synthesis lies in the critical scarcity of aligned multi-modal training data. Capturing synchronized motion capture data that simultaneously exhibits diverse trajectory navigation, rich action semantics, and expressive co-speech gestures is prohibitively expensive. Moreover, existing speech-to-gesture benchmarks (such as BEAT2) are almost entirely dominated by standing-in-place conversational poses, making it impractical to train end-to-end models that produce speech gestures while walking across large spatial areas. Prior work often attempts to combine predictions from separately trained text-driven and audio-driven diffusion models via weighted sum averaging, but the overwhelming standing-pose prior in speech models exerts excessive restorative forces that erase lower-body locomotion; alternative strategies employing speech transcripts as pseudo-text labels suffer from severe domain gaps between spoken dialogue and physical motion descriptions.

To break this deadlock, this work builds upon the natural physiological division of labor in human movement: speech audio predominantly drives upper-body dynamics (head, arm gestures, and facial expressions), whereas text prompts and spatial trajectories dictate lower-body displacement (spine, hips, and leg locomotion). Core idea: instead of demanding aligned multimodal training pairs, MOCO independently pre-trains single-modality denoisers, explicitly decouples full-body generation into spatial sub-regions during each diffusion denoising step, and diffuses the recombined state back into the next iteration to progressively harmonize full-body kinematics.

Method¶

Overall Architecture¶

Given concurrent text descriptions $c_{\text{text}}$, speech audio $c_{\text{audio}}$, and spatial trajectories $c_{\text{traj}}$, MOCO generates coherent 3D avatar animations represented frame-wise as $m = \{b, d\}$, where $b \in \mathbb{R}^{205}$ models full-body skeletal pose and root velocity in SMPL-X format, and $d \in \mathbb{R}^{286}$ captures detailed facial expressions and hand articulation.

The framework is built on four transformer-based denoisers: text-to-motion $G_{\text{T2M}}$ (trained on HumanML3D), speech-to-gesture $G_{\text{S2G}}$ (trained on BEAT2), trajectory-to-velocity $G_{\text{T2V}}$, and speech-to-details $G_{\text{S2D}}$. Under synchronous conditions, MOCO executes multi-modal decoupled denoising: at each denoising step $t$, $G_{\text{T2M}}$ and $G_{\text{S2G}}$ independently predict clean motion candidates from noisy inputs, which are assembled via predefined spatial binary masks and then forward-diffused into step $t-1$. For complex and asynchronous multi-action inputs, an LLM motion planning module decomposes text into elementary motion primitives and schedules a timeline with smooth transitions.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    In["Multimodal Inputs<br/>Text / Audio / Trajectory"] --> Plan["LLM Motion Planning<br/>Decompose primitives & transitions"]
    Plan --> Traj["Kinematic Trajectory Guidance<br/>L-BFGS optimization on velocity"]
    Traj --> Step["Multimodal Decoupled Denoising<br/>Lower body by text + Upper body by audio"]
    Step --> Refine["Per-Step Iterative Refinement<br/>Re-diffuse combined state t to t-1"]
    Refine --> Combine["Facial & Hand Detail Integration<br/>S2D denoiser on expressions & fingers"]
    Combine --> Out["Coherent 3D Avatar Motion"]

Key Designs¶

1. Multimodal Decoupled Denoising: Eliminating Dominance Imbalance without Paired Data
Prior weighted-sum fusion fails because audio-driven gesture models trained predominantly on static speech data exert disproportionately strong restorative forces, flattening text-guided locomotion into standing poses. MOCO bypasses this competition by factorizing the joint conditional probability distribution into two decoupled spatial components: $$ p(b_{t-1} \mid c_{\text{text}}, c_{\text{audio}}, b_t) \approx p(b_{t-1,\text{lower}} \mid c_{\text{text}}, b_t) \cdot p(b_{t-1,\text{upper}} \mid c_{\text{audio}}, b_t) $$ At each reverse diffusion step $t$, $G_{\text{T2M}}$ predicts candidate $\hat{b}_0^{\text{text}}$ under classifier-free guidance from overall noisy pose $b_t$, while $G_{\text{S2G}}$ predicts $\hat{b}_0^{\text{audio}}$. Using a binary spatial indicator vector $I \in \mathbb{R}^{205}$ (set to 1 for spine and legs, 0 for head and arms), the combined state is assembled: $$ b_{t-1} = I \odot b^{\text{text}}{t-1} + (1 - I) \odot b^{\text{audio}} $$ Because both denoisers are trained on clean single-modality datasets without weight sharing, their specialized representations remain uncompromised, allowing the lower body to faithfully execute text commands while the upper body expresses speech rhythms.

2. Per-Step Iterative Refinement: Enforcing Global Biomechanical Coordination
A naive spatial combination performed only once at the final step $t=0$ (the Combine Once baseline) leads to pronounced biomechanical discontinuities, disjointed torso transitions, and temporal jitter. MOCO solves this through per-step closed-loop re-diffusion: after assembling $b_{t-1}$ at step $t$, this intermediate state inherently carries the composite biomechanical context (e.g., lower-body stride phase paired with upper-body arm swing). In step $t-1$, both denoisers take this unified noisy state as input, allowing each modality to refine its subsequent prediction in direct awareness of the other half of the body. Across hundreds of diffusion steps, pelvis and spinal articulation smoothly self-adjust, transforming initial Gaussian noise into fluid, naturally coordinated full-body motion.

3. Kinematic Trajectory Guidance: Constraining Global Root Velocity
To direct avatars along arbitrary 2D floor paths $c_{\text{traj}} \in \mathbb{R}^{N \times 2}$, MOCO parameterizes root translation as velocity vectors $v = [\dot{r}_x, \dot{r}_y, \dot{\theta}]$. The model first predicts initial velocity $\hat{v}_0$ via denoiser $G_{\text{T2V}}$, applying test-time loss guidance formulated with differentiable Forward Kinematics (FK): $$ \mathcal{L}{\text{guidance}} = | \text{FK}(\hat{v}_0) - c |_2^2 $$ Optimizing }$\mathcal{L}_{\text{guidance}}$ with the second-order L-BFGS optimizer ensures that global trajectory displacements match the target paths with sub-centimeter fidelity. The optimized velocity $\hat{v}_0$ is subsequently substituted into the root channels of pose candidate $\hat{b}_0$ during full-body diffusion, steering lower-limb stepping and stance transitions without foot sliding.

4. LLM Motion Planning: Resolving Asynchronous & Complex Temporal Semantics
In practical interactive scenarios, user instructions frequently involve multi-step compound actions (e.g., "kneel down then run forward") interleaved with intermittent speech bursts. MOCO leverages an LLM as a high-level semantic planner. Driven by structured prompt engineering, the LLM parses the input instruction into sequential elementary motion primitives, automatically deduces and injects intermediate biomechanical transitions (e.g., automatically inserting "stand up from ground" between kneeling and running), and assigns start/end frame intervals alongside active body-part masks $I_j$. During diffusion sampling, multi-track temporal mask blending and DiffCollage boundary smoothing prevent jerky frame transitions across heterogeneous intervals.

A Worked Example¶

Consider an instruction: "kneel down then run forward" paired with two speech audio clips (clip 1 at 0–4s, clip 2 at 4–8s) and a straight trajectory: 1. Timeline Scheduling: The LLM planning pipeline translates the prompt into a multi-track schedule: primitive $c_1$ ("kneel down") for legs/spine on frames 0–60; auto-generated transition $c_2$ ("stand up from ground") on frames 60–105; $c_3$ ("run forward") on frames 105–240; simultaneously, audio conditions $c_4$ and $c_5$ govern head and arms over 0–120 and 120–240 frames, respectively. 2. Trajectory Optimization: The trajectory denoiser $G_{\text{T2V}}$ predicts pelvis linear and angular velocities $\hat{v}_0$ aligned with the path via L-BFGS forward kinematics guidance. 3. Iterative Reverse Denoising: Starting from standard Gaussian noise, at each step $t$, $G_{\text{T2M}}$ denoises lower-body joints under active text primitives, while $G_{\text{S2G}}$ denoises upper-body joints via audio cross-attention. The assembled pose is re-diffused into step $t-1$. 4. Detail Blending: At $t=0$, speech-driven facial expressions and finger movements from $G_{\text{S2D}}$ are composited into the motion tensor, yielding a synchronized avatar delivering dialogue while seamlessly kneeling, rising, and running.

Key Experimental Results¶

Main Results¶

Quantitative evaluations are conducted on a purpose-built benchmark consisting of 1,000 multi-modal test clips (each pairing two text prompts with two speech audio segments). Metrics evaluate text-to-motion fidelity (FID+, Top-1 retrieval accuracy R1, Motion-to-Text cosine similarity M2T, Motion-to-Motion similarity M2M) and speech-to-gesture quality (audio FID-A, Beat Consistency BC, and L1 diversity L1div).

Method / Baseline	FID+ ↓	R1 (%) ↑	M2T ↑	M2M ↑	FID-A ↓	BC ↑	L1div ↑
Ground Truth (GT)	0.000	40.0	0.781	1.000	-	-	-
Weighted Sum [Yang et al., ICASSP 2024]	1.335	6.8	0.546	0.537	2.17	2.20	4.08
Pseudo-Text [Ling et al., 2023]	1.593	2.2	0.511	0.503	2.22	2.55	6.43
SynTalker [Chen et al., ACM MM 2024]	0.985	9.8	0.601	0.603	6.60	2.95	9.12
MOCO (Ours)	0.862	24.6	0.649	0.639	3.83	2.72	8.62

Ablation Study¶

The ablation investigates spatial mask assignment (Body Mask), weight sharing across modalities (Share Weight), and per-step iterative combination (Per-Step Comb.) against Transition Smoothness Ratio (TSR ↓, ratio of mean speed in transition windows to non-transition regions).

Variant Config	Share Weight	Body Mask (1-I)	Per-Step Comb.	FID+ ↓	R1 (%) ↑	M2T ↑	FID-A ↓	BC ↑	L1div ↑	TSR ↓
MOCO (Full Model)	✗	head, arms	✓	0.862	24.6	0.649	3.83	2.72	8.62	0.90
Variant 1 (Spine to audio)	✗	head, arms, spine	✓	0.921	22.4	0.634	3.86	2.81	8.35	0.90
Variant 2 (Inverted split)	✗	spine, legs	✓	1.234	7.9	0.554	2.75	2.19	5.11	0.95
Variant 3 (Shared weights)	✓	head, arms	✓	0.866	22.1	0.656	4.24	2.68	8.95	0.85
Variant 4 (Combine at t=0)	✗	head, arms	✗	0.832	24.4	0.645	3.99	2.27	8.60	1.17

A rigorous user study further validates that: - MOCO dominates Pseudo-Text and Weighted Sum in Text Following (99.0% vs 0.0% and 100.0% vs 0.0%) and Beat Synchronization (74.5% vs 12.5% and 83.0% vs 3.5%). - Compared to the single-step combination baseline (Variant 4), MOCO secures superior user preference in Body Coherence (71.0% vs 17.0%) while maintaining a noticeably smoother transition ratio (TSR 0.90 vs 1.17).

Key Findings¶

Per-step re-diffusion is essential for holistic harmony: While combining parts only at the final step (Variant 4) achieves comparable FID+ on isolated frames, its TSR degrades sharply to 1.17, causing jarring boundary artifacts and an overwhelming user rejection in physical coherence (17% vs 71%).
Physiological spatial partitioning must match semantic priors: Assigning the spine to speech audio (Variant 1) degrades text R1 accuracy from 24.6% to 22.4%; inverting the assignment entirely (Variant 2) collapses text-to-motion R1 to 7.9% and BC to 2.19, as static speech gesture priors choke dynamic lower-body locomotion.
Modality-specific denoisers prevent negative transfer: Sharing transformer weights across text and audio tasks (Variant 3) worsens speech FID-A from 3.83 to 4.24 due to conflicting motion dynamics and feature scales.

Highlights & Insights¶

Training Disentanglement with Inference Synergy: MOCO elegantly harnesses the Markovian diffusion reverse chain to harmonize heterogeneous control modalities without requiring a single aligned multi-modal training sample.
LLM as Kinematic Transition Planner: Employing LLMs not just for prompt parsing, but to infer implicit physical state transitions (e.g., auto-inserting "stand up" between "sit" and "walk"), offers an effective paradigm for compositional animation.
Pluggable Kinematic Trajectory Guidance: Combining second-order L-BFGS optimization on differentiable forward kinematics directly in the diffusion sampling loop provides precise spatial trajectory adherence without retraining the base generative models.

Limitations & Future Work¶

Static Upper/Lower Body Partitioning: The rigid assignment of arms to speech audio prevents the model from directly honoring textual arm commands (e.g., "wave hands") unless the user explicitly overrides the timeline tracks manually.
Foot Sliding during Complex Ground Contacts: During transition phases involving physical support shifts (e.g., sitting down or rising from a chair), slight foot sliding persists due to the absence of explicit contact force optimization.
Benchmark Diversity Constraints: The current 1,000-clip benchmark concentrates on conversational and navigation actions; expanding evaluation to high-dynamic athletic activities and acrobatic movements remains an open challenge.

vs Weighted Sum (FreeTalker): FreeTalker blends audio and text diffusion scores via linear weighting, but because speech-gesture datasets are dominated by stationary standing poses, the audio score wipes out text-driven translation. MOCO replaces numerical feature averaging with clean spatial decomposition.
vs Pseudo-Text (MCM): MCM substitutes spoken transcripts for action prompts, creating severe distributional domain gaps. MOCO trains specialized denoisers on unimodal benchmark datasets and coordinates them purely during inference.
vs STMC: While STMC introduced per-step body-part composition for multi-track text prompts, MOCO advances this to heterogeneous, multi-rate multi-modal signals including continuous acoustic waveforms and 2D trajectory geometry.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Elegant utilization of spatial physiological priors and per-step iterative diffusion feedback to synthesize multi-modal motions without aligned data.
Experimental Thoroughness: ⭐⭐⭐⭐☆ Rigorous validation across a newly curated 1,000-clip multimodal benchmark, complete with ablations and user preference studies.
Writing Quality: ⭐⭐⭐⭐⭐ Crystal-clear mathematical formulations, well-motivated problem formulation, and structured architectural exposition.
Value: ⭐⭐⭐⭐⭐ Provides an immediately deployable, modular framework for digital avatars, virtual presenters, and embodied conversational agents.