VersatileMotion: A Unified Framework for Motion Synthesis and Comprehension

Conference: ECCV2026
Paper: ECCV paper page
PDF: Full paper
Area: Human Understanding
Keywords: motion generation, motion understanding, causal tokenizer, multi-person interaction, multimodal autoregression

TL;DR

VersatileMotion uses the world-space causal body tokenizer VerMoVQ to formulate text, audio, and multi-person motion as unified next-token prediction, covering nine motion tasks in one model with competitive generation and understanding under the authors' evaluation protocol.

Background & Motivation

Text-to-motion, music-to-dance, speech-driven gestures, and motion captioning are all cross-modal transformations, but they commonly rely on different models. Specialist generators can tailor their networks to one condition, yet do not directly support the reverse mapping to language; multi-person interaction additionally requires spatial relationships between people. TM2T and MotionGPT already demonstrate the potential of language modeling over discrete motion, but adding modalities does not automatically recover information lost by the motion representation itself.

The paper locates the bottlenecks in tokenization and data coverage: the authors argue that common root-relative representations hinder world-space multi-person reasoning, one vector per frame entangles body structure, and non-causal temporal encoding mismatches forward-only generation. Text paired with single-person motion is also much more abundant than music, speech, or multi-person data, so mixing all examples can expose the model to audio conditions only rarely. The system therefore needs a better discrete motion representation, broader cross-modal data, and control over each task's effective training exposure.

Instead of adding a dedicated conditioning branch for every task, the paper builds compositional motion tokens and changes where conditions and targets appear in the sequence. Core Idea: connect motion to language and audio through world-space body tokens with strict temporal causality, then use a unified message format and staged task sampling to support generation, understanding, and multi-person coordination in one autoregressive model.

Method

Overall Architecture

Inputs can include text, audio, known motion segments, and metadata such as participant count or duration; the target can be motion, text, or music. MotionHub and world-space representation standardize the training material, while VerMoVQ converts continuous body states into discrete motion tokens. Unified motion messages arrange modalities into instructions, conditions, and replies, which an autoregressive Transformer generates token by token from the prefix. Dual-pathway staged training gives the same model both broad motion priors and sufficient exposure to scarce audio conditions.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    sources["Motion, text, and audio data"] --> corpus["MotionHub and<br/>world-space representation"]
    corpus --> tokenizer["Causal body tokenizer<br/>VerMoVQ"]
    tokenizer --> message["Unified motion messages"]
    conditions["Text and audio tokenization<br/>Instructions and metadata"] --> message
    message --> training["Dual-pathway staged training"]
    training --> model["Shared autoregressive model"]
    model --> output["Motion, captions, or music"]

The diagram connects training and use; it does not imply retraining for every inference request. At inference time, the trained tokenizers and model are reused: the system constructs the current task prefix and decodes the generated target tokens.

Key Designs

1. MotionHub and world-space representation: retain information needed for interaction

MotionHub collects approximately 380K single- and multi-person motion clips standardized to SMPL at 30 fps. For music-dance and speech-gesture data without descriptions, the authors add paired text through human and vision-language-model-assisted annotation. Audio-motion examples can consequently support both audio-conditioned generation and motion-language training tasks. The paper also introduces MotionBench as a standardized suite of nine tasks; this organizes evaluation rather than adding another branch to the generator.

A frame contains world-space root position, root velocity, and continuous 6D rotations for the joints. Following Eq. (1) in Section 3.1, the essential representation is:

\[ x_t=(p_t,v_t,R_t). \]

Both \(p_t\) and \(v_t\) are three-dimensional, while \(R_t\) collects the joint rotations. Root position and velocity form one 6D body token, and the remaining body tokens hold the joints' 6D rotations. Multi-person motion therefore need not first move every participant independently to the origin and then recover their relative placement elsewhere. Participants share the tokenizer and discrete space, while the retained world coordinates provide information for spatial reasoning across their token streams. This does not automatically impose contact or collision constraints; it avoids discarding the signals needed to learn such relationships at the representation stage.

2. Causal body tokenizer VerMoVQ: replace whole-frame compression with spatiotemporal factorization

VerMoVQ organizes motion along temporal, body-token, and channel dimensions instead of putting the entire pose into one high-dimensional frame vector. The encoder downsamples time to produce a shorter sequence while retaining the body dimension for quantization. Body tokens at each temporal position are discretized separately, then flattened over time and body indices into a one-dimensional autoregressive stream. Smaller body units reduce the number of full-body combinations that each discrete code must represent, easing quantization pressure.

Every temporal convolution is strictly causal: encoding and decoding the current frame do not access future frames. Clips can consequently be encoded and generated as continuations of a prefix, supporting prediction, variable-length chaining, and streaming use. Temporal convolutions alone would leave different body units independent, so the tokenizer also applies self-attention over the body dimension at each time step. Body attention couples root movement and articulated rotations, so torso rotation and arm posture need not be determined only by their separate temporal histories.

The implemented quantization uses finite scalar quantization, or FSQ: latent dimensions receive fixed discrete levels without learned-codebook or commitment losses. The text first describes generic nearest-codeword VQ and then specifies FSQ; the latter is the concrete quantization choice to retain when interpreting the implementation. The main 2D, 16k configuration is not the candidate with the lowest reconstruction error, but the authors report that it is more suitable for downstream generation. The important distinction is that a tokenizer must both reconstruct motion and produce discrete sequences that the autoregressive model can learn effectively.

3. Unified motion messages: switch task direction by arranging tokens

Text uses a subword tokenizer, audio a VQ-VAE, and motion VerMoVQ, with their discrete symbols entering one vocabulary. A shared vocabulary does not mean that all modalities share an encoder; it means the resulting tokens are processed by one Transformer. A message contains a natural-language instruction, conditions enclosed by modality boundary markers, and a target reply with its own boundaries. Participant count and duration can appear as condition fields without redesigning the generation backbone for each field.

Text-to-motion places text in the condition and motion in the reply; motion-to-text reverses those roles. Music-to-dance and dance-to-music likewise change the arrangement of conditions and targets. Multi-person motion uses the same motion-token space, separates participants' streams with an agent delimiter, and leaves the model to learn cross-person relationships. This permits architectural extension to larger groups, but the quantitative multi-person evidence is mainly two-person, not validation at arbitrary group sizes.

Optional condition fields are stochastically included during training, exposing the model to different combinations such as music plus text. Composite-condition results should therefore not be described as zero-shot behavior with no training exposure to combined conditions. Temporal completion places known segments in the condition; tokenizer causality restricts future access in local encoding, not whether a known endpoint may be supplied to an in-betweening task. Separating these roles lets the same representation serve both forward prediction and completion of intermediate segments.

4. Dual-pathway staged training: prevent abundant tasks from overwhelming scarce conditions

MotionHub is task-imbalanced, and the main text states that music-conditioned samples comprise less than 1% of eligible data. With exclusively data-proportional sampling, training loss can improve while the model rarely gets an opportunity to learn how to use music. Pretraining nevertheless samples in proportion to data volume to establish broad motion and cross-modal priors, supervising next-token prediction over the complete message. Generalist supervised fine-tuning then selects a task uniformly before drawing examples for it and supervises only the reply.

Scarce tasks receive greater exposure during fine-tuning, while abundant skills continue to be rehearsed rather than completely replaced by the new domains. The two pathways refer to different sampling routes used at different training stages, not two generators running at inference time. The authors also compose single-person clips into pseudo two-person pretraining examples to provide more exposure to multi-person formats. Table 8 tests this data strategy separately, but the cache lacks its detailed composition procedure, so physically realistic contact should not be assumed.

A Worked Example

Consider generating complementary dances for two people from music, with one person spinning and the other jumping; this paraphrases the message example in the main text. The description becomes text tokens, the music becomes audio tokens, and participant-count and duration fields join the condition segment. After reading the instruction and conditions, the model generates the participants' motion-token streams in the reply. The agent delimiter distinguishes the performers, while shared world coordinates retain information about their relative placement. The VerMoVQ decoder reconstructs root trajectories and joint rotations from the generated tokens to obtain SMPL motion. To caption the motion instead, the system moves motion tokens into the condition and makes text the reply target, without adding a separate motion-understanding backbone.

Loss & Training

The tokenizer combines parameter reconstruction with a forward-kinematics loss, as given by Eq. (4) on page 8:

\[ \mathcal{L}_{\mathrm{rec}}=\mathcal{L}_{\mathrm{param}}+\lambda_{\mathrm{fk}}\mathcal{L}_{\mathrm{fk}}. \]

The parameter term applies L1 loss to translation, velocity, and rotations; the forward-kinematics term compares root-centered 3D joint positions. The former directly constrains world-space root trajectories and pose parameters, while the latter constrains the body geometry produced by those parameters. Weights, learning rates, and training-step counts absent from the cache are not supplied here. The autoregressive model defaults to LLaMA 3.2-1B initialization and next-token prediction rather than separate generation objectives for individual tasks. Section 3.3 describes two stages, generalist pretraining and generalist supervised fine-tuning, whereas Figure 2 additionally depicts Specialist SFT. This note follows the main text for the training pipeline and retains that figure-text discrepancy as an issue to resolve during reproduction.

Key Experimental Results

Main Results

The following selection comes from main-text Tables 3, 4, and 5; 1P and 2P denote single- and two-person settings. FID measures the difference between generated and real motion feature distributions and is lower-is-better; R-Precision measures retrieval hits for matching text or motion and is higher-is-better. Table 3 uses the authors' trained TMR features, so its FID values should not be compared directly with papers using different evaluators. Table 5 uses Top-3 R-Precision with 64 candidates, which is not the same setting as Top-1 in Table 3.

Source and condition	Metric	VersatileMotion	Comparator	Comparator value
Table 3, HumanML3D, 1P text-to-motion	FID, lower is better	0.062	MotionStreamer	0.065
Table 3, HumanML3D, 1P text-to-motion	R-Precision Top-1, higher is better	0.457	HY-Motion	0.561
Table 3, MotionHub, 1P text-to-motion	FID, lower is better	0.098	Go-To-Zero	0.101
Table 4, MotionHub, 2P text-to-motion	FID, lower is better	0.51	InterMask	0.43
Table 5, HumanML3D, 1P motion-to-text	R-Precision Top-3, higher is better	0.686	TM2T	0.536
Table 5, MotionHub, 2P motion-to-text	R-Precision Top-3, higher is better	0.778	MotionGPT, retrained	0.691

Lower generation FID on HumanML3D does not simultaneously deliver the best text retrieval consistency, so quality and semantic alignment should not be collapsed into an across-the-board leadership claim. Most baselines were not trained on MotionHub, and the cross-domain comparison in Table 3 includes differences in training-data coverage. VersatileMotion does not surpass InterMask in two-person generation; its contribution is strong performance without an interaction-specific architecture. Section 4.4 also re-injects ground-truth root information for most baselines while evaluating this model directly in world space, a processing difference that must remain visible in comparisons.

Ablation Study

The following selection is from Table 8 on page 14: the first three rows compare generation and understanding training for the 1B model, while the last two test pseudo data on two-person tasks. "Not evaluated" means that the configuration has no result in that direction, not a score of zero.

Source and configuration	Text-to-motion FID, lower is better	Motion-to-text R-Precision Top-3, higher is better	Condition
Table 8(i), generation only	0.074	Not evaluated	Single-direction training
Table 8(i), understanding only	Not evaluated	0.658	Single-direction training
Table 8(i), joint generation and understanding	0.062	0.686	Joint bidirectional training
Table 8(iii), without pseudo multi-person pretraining	0.60	0.544	2P tasks
Table 8(iii), with pseudo multi-person pretraining	0.51	0.778	2P tasks

Joint generation and understanding improve both directions, showing that unification offers more than a reduction in model count. Pseudo two-person data improves multi-person performance, but its rows should not be ranked against single-person configurations. Table 7 also reports benefits from all-task mixing over single-task specialists, supporting the interpretation of transferable cross-task motion priors.

The tokenizer analysis below is selected from Table 2 on page 10, evaluated consistently on the MotionHub test set. MPJPE measures joint-position error, PA-MPJPE applies alignment before measuring it, and both use mm; MPJRE measures joint-rotation error in degrees.

Participants and tokenizer	MPJPE, lower is better	PA-MPJPE, lower is better	MPJRE, lower is better	Codebook utilization
1P, Go-To-Zero, FSQ 64k	27.19	21.19	6.35	83.7%
1P, VerMoVQ 1D, RVQ 1k×6	12.34	9.49	3.43	100.0%
1P, VerMoVQ 2D, FSQ 16k	18.13	10.51	1.93	97.3%
1P, VerMoVQ 2D, FSQ 64k	13.54	8.38	1.50	59.2%
2P, VerMoVQ 2D, FSQ 16k	18.52	10.73	2.07	97.8%

Key Findings

• The 1D RVQ variant has lower position reconstruction error than the default 2D configuration, yet the authors choose the latter for generation: optimal reconstruction does not imply optimal sequence modeling.
• Enlarging the 2D discrete space reduces reconstruction errors but also lowers utilization; Table 2 alone cannot establish the best generation configuration.
• The 2P error of 18.52 mm is close to the 1P error of 18.13 mm, supporting reconstruction transfer with a shared tokenizer, not proving correct contact.
• In Table 6 on page 13, in-betweening with "1 frame plus text" reaches FID 0.166 versus MotionGPT3's 0.276; the conditioning regime must accompany these numbers.
• The main text reports music, gesture, and long-sequence capabilities, but some detailed tables are in unavailable supplementary material; this note does not expand them into an unverified full leaderboard.

Highlights & Insights

• World-space information is a representation choice, not merely an extra condition for multi-person models. Retaining root trajectories first makes it possible for an ordinary sequence model to learn relative placement.
• Body factorization affects both quantization and subsequent prediction difficulty. Tokenizers should be evaluated through geometry reconstruction and generation, not only minimum MPJPE.
• Generation and understanding can supply complementary supervision. The bidirectional improvements in Table 8 move the argument beyond the engineering convenience of calling several tasks through one model.
• Whether a condition is used depends on training exposure. Learning broad motion priors before increasing scarce-task sampling is a more direct data-side intervention than merely enlarging the vocabulary.

Limitations & Future Work

• The following points are primarily review judgments based on the main text, not all limitations explicitly acknowledged by the authors.
• Architectural support for arbitrary group sizes is different from validation at arbitrary group sizes; quantitative multi-person results mainly concern two people, warranting tests with larger groups and complex occlusion or contact.
• Direct SMPL output does not guarantee physical executability, and position, rotation, and retrieval metrics cannot replace evaluation of foot sliding, interpenetration, forces, or contact quality.
• Data aggregation and human/VLM-assisted descriptions affect semantic quality; without the appendix, source proportions, annotation consistency, and split details cannot be fully checked.
• TMR evaluation features, training-data coverage, and two-person root processing all affect comparisons, so advantages under this protocol should not be equated with leadership on every standard leaderboard.
• Causality supports prefix consistency, but body-token and participant concatenation increase sequence length; the main text does not establish detailed latency and resource scaling for large groups or long durations.
• Useful next steps include comparing sampling strategies under matched data budgets, adding contact and latency metrics, and clarifying Specialist SFT in Figure 2 relative to the two-stage main-text account.

Related Work & Insights

• vs TM2T and MotionGPT: all connect discrete motion with language; this paper targets world-space and temporal-causality constraints in tokenization while extending audio and multi-person coverage.
• vs Go-To-Zero: both involve SMPL and FSQ, but this work additionally uses body-axis factorization, causal temporal processing, and unified task organization; the gains cannot be assigned solely to FSQ.
• vs MotionStreamer: both emphasize causal motion representations and continuous generation; this work additionally includes understanding tasks such as captioning in the same discrete vocabulary and autoregressive interface.
• vs InterMask: InterMask still obtains better two-person generation FID in this paper; VersatileMotion emphasizes generality without interaction-specific architecture rather than dominance on every multi-person metric.
• Research direction: independently control tokenization, training data, and sampling to identify whether transfer comes from better geometry, language supervision, or greater exposure to scarce tasks.

Rating

• Novelty: 4/5. The main contribution connects world-space causal body tokenization with unified task training rather than inventing a quantizer in isolation.
• Experimental Thoroughness: 4/5. The main text covers reconstruction, multiple task directions, and training ablations, but some details require supplementary material and comparisons depend on data and evaluation protocols.
• Writing Quality: 3/5. The main mechanisms are clear, but Figure 2 and the text disagree on training stages, and generic VQ exposition must be distinguished from the concrete FSQ choice.
• Value: 4/5. The representation and training ideas are reusable for multitask human-motion models, with larger groups, physical constraints, and practical inference costs still requiring validation.