Skip to content

UniMotion: A Unified Framework for Motion-Text-Vision Understanding and Generation

Conference: ECCV 2026
arXiv: 2603.22282
Code: https://wangzy01.github.io/UniMotion
Area: Human Understanding / Multimodal VLM
Keywords: Human Motion Generation, Tri-modal Unification, Continuous Motion Representation, Cross-modal Alignment, Motion VAE

TL;DR

UniMotion treats human motion as a continuous modality with equal status to RGB within a shared LLM. By replacing discrete tokens with a Continuous Motion Aligned VAE (CMA-VAE) and a dual-path embedder, and incorporating Dual Posterior KL Alignment (DPA) to inject visual semantic priors along with Latent Reconstruction Alignment (LRA) to resolve the motion path cold-start issue, UniMotion unifies and achieves comprehensive state-of-the-art (SOTA) performance across seven understanding, generation, and editing tasks on Motion-Text-RGB tri-modalities.

Background & Motivation

Unified multimodal large models have recently made substantial progress in joint "text + image" understanding and generation (e.g., Show-o, Janus-Pro, Show-o2), utilizing a single Transformer for both autoregressive understanding and diffusion generation to conduct cross-modal reasoning in a shared semantic space. However, human motion—a high-value dynamic modality—has consistently been excluded from such unified frameworks. Motion sequences encode temporal dynamics and spatial structures essential for game animation, embodied AI, VR, healthcare rehabilitation, and privacy-preserving behavior analysis. Yet, no framework has unified the Motion-Text-RGB tri-modalities, let alone supported both understanding and generation. Existing works only address halves of the puzzle: MotionGPT uses VQ-VAE to tokenize motion, achieving Motion↔Text unification, but cannot interpret or generate images; UniPose integrates human pose into vision-language models, but only handles single-frame static pose estimation and image understanding, lacking generation capabilities.

Furthermore, both directions rely on discrete tokenization. Vector quantization (VQ) irreversibly loses information, introducing temporal jitter to motion, compromising structural fidelity, and limiting diversity due to codebook collapse. Moreover, a natural asymmetry exists between discrete tokens and continuous RGB feature spaces, making cross-modal alignment highly awkward. Quantifications in the supplementary material of the paper highlight this clearly: the reconstruction absolute position error (APE) of VQ-VAE is as high as 17.15 cm, and the error for end joints like wrists climbs to over 210 mm, injecting highly visible jitter artifacts in high-frequency bands. Therefore, the core challenge is that to unify tri-modal understanding and generation in a single model, motion—a modality characterized by skeletal topology, kinematic constraints, and dense dynamics—must share the same representational form as RGB. Discrete tokenization, however, pushes it to the opposite side of RGB.

The key insight of this paper is that since RGB utilizes a continuous latent path in unified frameworks, rather than tokenizing motion, a symmetrical continuous path should be established for it to let cross-modal alignment emerge naturally at the structural level. The core idea is to treat motion as a continuous modality with equal status to RGB—utilizing continuous CMA-VAE to encode motion and symmetrical dual-path embedders to construct parallel continuous pathways. This structurally eliminates quantization errors, uses DPA to distill visual semantics into the motion encoder, and leverages LRA for self-supervised warm-up of the motion path, eventually unifying tri-modal understanding and generation within a shared LLM.

Method

Overall Architecture

UniMotion aims to simultaneously support seven tasks: Text-to-Motion (T2M), Motion-to-Text (M2T), motion prediction, motion editing, Vision-to-Motion (reconstructing motion from images), Vision-to-Text, and Motion-guided Image Editing (MGIE). Based on a Show-o2 1.5B backbone, its mechanism can be summarized in one sentence: both motion and images are encoded as continuous latent representations, through symmetrical dual pathways, input into the same LLM backbone, and decoded/generated by modality-specific flow heads.

The pipeline consists of three main components. First is CMA-VAE: compressing variable-length motion sequences into continuous low-dimensional latent codes \(z\), where visual semantics from paired images are implicitly injected into the motion encoder during training via DPA (the image branch is completely discarded during inference, incurring zero overhead). Second is the unified multimodal architecture: a dual-path embedder splits \(z\) into "semantic abstraction" and "detail preservation" branches for the LLM. Mixed attention and modality-routing LoRAs adapt motion and text/RGB on shared parameters, while modality-specific flow heads predict velocity fields in the corresponding latent spaces. Third is LRA: prior to downstream tasks, a self-supervised "motion-to-motion" reconstruction warm-up calibrates the motion path (embedder + flow head + motion LoRA) before multi-stage progressive fine-tuning.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Motion Sequence / RGB Image / Text"] --> B["Continuous Motion Paradigm: CMA-VAE<br/>Encode motion into continuous latent code z"]
    B -->|Inject visual semantics during training| C["Dual Posterior KL Alignment DPA<br/>Distill vision-fusion encoder into pure motion encoder"]
    C --> D["Latent Reconstruction Alignment LRA<br/>Motion-to-motion self-supervised warm-up of the motion path"]
    D --> E["Unified Multimodal Architecture<br/>Dual-path embedder + Mixed attention + Routing LoRA"]
    E -->|Multi-stage progressive fine-tuning| F["Seven Tasks: Understanding / Generation / Editing"]

Key Designs

1. Continuous Motion Paradigm and CMA-VAE: Aligning motion and RGB on the same representational form

This step directly targets the two fatal flaws of discrete tokenization: irreversible information loss and mismatch between discrete tokens and continuous RGB spaces. UniMotion utilizes a continuous VAE to encode motion sequences into low-dimensional latent codes. Frame-by-frame, it passes through linear projection + learnable position embeddings + a SkipTransformer encoder, outputting Gaussian parameters which are then reparameterized to sample \(z\in\mathbb{R}^{T_z\times d}\). This preserves the temporal continuity and structural fidelity of motion. Crucially, the continuous motion latent codes and continuous RGB latent codes share a symmetrical representational form, allowing cross-modal alignment to occur naturally at the architectural level without forcing discrete compromises. Supplementary experiments show that CMA-VAE's reconstruction error (APE = 3.53 cm) is an order of magnitude lower than VQ-VAE (17.15 cm) and standard MLD-VAE (9.28 cm); end-joint errors are nearly on par with the torso, and frequency-domain residuals are the lowest across the entire spectrum. This confirms that "continuous + cross-modal anchoring" preserves both global pose and high-frequency details of distal joints.

However, continuity alone is insufficient. Standard motion VAEs (such as MLD-VAE) encode motion in isolation, meaning their latent space lacks visual semantic structure. CMA-VAE's "Cross-Modal Aligned" nature stems from being inherently cross-modal: a pure motion encoder \(q_\phi(z\mid\mathbf{m})\) used for inference is coupled through posterior alignment to a training-only vision-fusion encoder \(q_\psi(z\mid\mathbf{m},\mathbf{v})\)—leading to DPA.

2. Dual Posterior KL Alignment (DPA): Leveraging visual semantics during training, relying solely on motion during inference

The limitation is specific: to imbue the motion latent space with visual semantics (e.g., plausible body configurations, global balance), the most direct way is to feed paired images during training. However, users typically only provide motion, not images, during inference. The mechanism of DPA is to train the two encoders separately. The vision-fusion encoder extracts RGB image features using a frozen HRNet, and then performs bilinear grid sampling (GridSample(HRNet(v), j_2d(m))) on the 2D joint positions projected from the motion skeleton. Consequently, visual features are precisely extracted from body-related regions rather than the global image, inherently carrying cross-modal sensitivity. DPA then constrains the posterior of the pure motion encoder to approximate the vision-fusion posterior, instructing the former to "absorb" visual semantics into its parameters during training. The overall objective consists of reconstruction loss + dual-path KL regularization + alignment loss, where the alignment term is only computed for samples with paired images (e.g., Human3.6M) and omitted for datasets without paired images (e.g., HumanML3D), making it highly flexible.

The alignment loss is formulated as the KL divergence between two diagonal Gaussians, detaching the vision-fusion posterior \(q_\psi\) as a fixed teacher:

\[\mathcal{L}_{\mathrm{align}}=D_{\mathrm{KL}}\bigl(q_{\phi}(z\mid\mathbf{m})\,\|\,q_{\psi}(z\mid\mathbf{m},\mathbf{v})\bigr)\]

The direction is chosen deliberately: under the "student \(\rightarrow\) teacher" distillation convention, this is a reverse KL, which possesses a mode-seeking property. It forces \(q_\phi\) to contract to the most prominent semantic modes of \(q_\psi\), producing compact, highly confident motion representations while naturally filtering out viewpoint-dependent visual noise unrelated to motion mechanics. Conversely, a forward KL would be mode-covering, forcing \(q_\phi\) to cover all modes of \(q_\psi\) (including noisy ones), resulting in over-diffused posteriors, overestimated variance, and diluted representation precision. Because both encoders share front-end parameters, the gradients computed by DPA on H36M indirectly improve performance on HumanML3D. This explains why removing DPA impairs even pure text tasks like T2M (R@3 dropped from 0.841 to 0.818), despite HumanML3D having no paired images.

3. Unified Multimodal Architecture: Dual-path embedder + Mixed attention + Modality-routing LoRA integrating motion into Show-o2

The continuous latent code \(z\) produced by CMA-VAE passes through a dual-path embedder before entering the LLM, which is key to the architectural symmetry. It operates in two parallel branches: a semantic branch (MLP + Transformer encoder) extracts high-level semantics, mirroring SigLIP on the vision side; and a generation branch (MLP + learnable position embeddings) projects \(z\) directly into the LLM's hidden dimension to retain fine-grained motion details, mirroring PatchEmbed on the vision side. The two branches are concatenated and fused via RMSNorm+MLP into a unified representation used for all tasks. This decoupling of "semantic abstraction / detail preservation" is crucial for motion-conditioned generation (editing, prediction)—the source motion must provide both high-level structural constraints and joint-level details, which a single branch cannot achieve. Supplementary experiments show that retaining only the generation or semantic branch performs significantly worse than the dual-path setup: the semantic branch yields a higher M2T BertScore, while the generation branch excels in T2M/editing, demonstrating complementarity over redundancy, especially in editing and prediction tasks where the dual-path gap is widest.

Once inside the LLM, two additional designs are employed. Mixed attention maintains a global causal order at the sequence level (for text autoregression) but opens bidirectional full attention inside each motion span. This accommodates the flow matching objective (which requires predicting velocity fields simultaneously across the entire motion latent code) while strictly ensuring that motion/image tokens can only see tokens within their own span and prior spans, preventing future leakage. Modality-routing LoRA attaches two low-rank branches to the Q/K/V/O matrices of each attention layer: one for Text/RGB and one for Motion. It utilizes a deterministic modality mask for routing (since modality identity is known from sequence construction), adapting to modality-specific features with only a ~2% parameter increase, avoiding the routing noise and load-balancing losses associated with MoE-LoRA. Under the RGB path, a frozen pose-aware visual backbone (initialized from TokenHMR) is added to provide structural body representations, supplementing SigLIP's global semantics and maintaining symmetry with the motion side's dual-path setup.

4. Latent Reconstruction Alignment (LRA): Warming up the motion path with dense motion latent codes

Even after DPA pre-training, the motion path is not fully calibrated; the embedder, motion flow head, and motion LoRA have not been aligned jointly. Discarding LRA and directly proceeding to multi-task training causes a notable performance drop (T2M R@3 to 0.801 vs. 0.841). The root cause is supervisional mismatch: the target motion to be generated is dense and kinematically rich, yet the text serving as the primary learning signal is sparse (a single sentence only describes coarse motion semantics, omitting stride sizes, limb coordination, and fine temporal transitions). Learning a generative path from such under-determined signals yields blurry, unstable, and low-fidelity results. In contrast, CMA-VAE's latent code \(z\) is the model's own, lossless, dense motion encoding—its self-reconstruction is an explicit one-to-one mapping, serving as an ideal zero-cost signal to warm up the motion path.

Consequently, LRA designs a "Motion-to-Motion" (M2M) self-reconstruction task: CMA-VAE encodes \(z\), the dual-path embedder projects \(z\) into the LLM, and the LLM's hidden states reconstruct \(z\) from noise via the motion flow head:

\[\mathcal{L}_{\mathrm{M2M}}=\mathbb{E}_{z_0\sim\mathcal{N}(0,I),\,t}\bigl[\|v_\theta(z_t,t\mid\mathrm{Embed}_{\mathrm{fused}}(z))-(z-z_0)\|^2\bigr]\]

where \(z_t=t\cdot z+(1-t)\cdot z_0\). Crucially, the LLM receives the clean condition \(\mathrm{Embed}_{\mathrm{fused}}(z)\), while the noisy \(z_t\) is injected only into the flow head via AdaLN. This ensures the motion path learns "structured encoding of motion semantics" rather than collapsing into a basic denoiser. M2M calibrates three components simultaneously: the embedder (compressing \(z\) into LLM-readable tokens), the motion LoRA in the shared backbone (extracting structural cues), and the flow head (mastering latent space geometry). This provides critical geometric feedback that sparse text supervision cannot offer. Once calibrated, this path serves as a shared foundation for all downstream tasks: T2M directly benefits from the pre-calibrated flow head, M2T reuses the embedder's semantic compression, and Vision-to-Motion simplifies cross-modal mapping for the LLM. The authors demonstrate that LRA does not collapse into a trivial identity mapping through architectural necessity, information bottlenecks (applying 20–50% temporal downsampling, 15% feature dropout, and \(\sigma=0.02\) perturbation to the condition to ensure input \(\neq\) target), and shuffled-condition control experiments (matching condition FID = 0.008 vs. shuffled FID = 2.34, a \(\sim\)300x degradation).

Loss & Training

CMA-VAE training uses \(\mathcal{L}_{\mathrm{VAE}}=\mathcal{L}_{\mathrm{recon}}+\lambda_{\mathrm{KL}}(\mathcal{L}_{\mathrm{KL}}^\phi+\mathcal{L}_{\mathrm{KL}}^\psi)+\lambda_{\mathrm{align}}\cdot\mathcal{L}_{\mathrm{align}}\), where the alignment term uses linear warm-up (first 10k steps) to avoid disrupting early reconstruction. Generation employs flow matching: \(\mathcal{L}_{\mathrm{flow}}=\mathbb{E}[\|v_\theta(x_t,t)-u_t\|^2]\) (velocity field \(u_t=x_1-x_0\)), with \(t\) sampled from a Logit-Normal distribution to densify middle time points. The total generation loss is \(\mathcal{L}_{\mathrm{gen}}=\lambda_{\mathrm{ntp}}\mathcal{L}_{\mathrm{NTP}}+\lambda_{\mathrm{flow}}\mathcal{L}_{\mathrm{flow}}\) (\(\lambda_{\mathrm{ntp}}{=}1.0,\lambda_{\mathrm{flow}}{=}0.8\)). Training occurs in progressive stages: CMA-VAE pre-training (210k steps) \(\rightarrow\) Stage 0 LRA warm-up (M2M, 80k) \(\rightarrow\) Stage 1a/1b motion-text alignment (T2M \(\rightarrow\) +M2T+prediction+editing) \(\rightarrow\) Stage 2 cross-modal expansion (+V2M/V2T/MGIE, activating the image path) \(\rightarrow\) Stage 3 full-task fine-tuning (partially unfreezing the LLM). The entire pipeline is trained with AdamW in bf16 on 4 \(\times\) A6000 GPUs, using Euler ODE (50 steps) with CFG (scale 3.0) for inference.

Key Experimental Results

Main Results

Multi-task unified comparison (selecting one representative metric per task): UniMotion is the only method covering all seven tasks, utilizing a backbone of only 1.5B (compared to competitors mostly at 7B).

Task Metric UniMotion Prev. SOTA Description
T2M R@3↑ 0.841 0.802 (MG-MotionLLM) Text-to-motion semantic alignment
M2T BertScore↑ 41.2 36.7 (MG-MotionLLM) Motion description quality
Motion Prediction ADE↓ 3.172 4.745 (MotionGPT) Future pose prediction
Motion Editing R@3↑ 84.94 73.23 (MG-MotionLLM) Text-guided editing
V2M MPJPE↓ 75.0 81.8 (UniPose) MLLM-based image-to-motion
V2T BLEU-4↑ 21.9 17.3 (UniPose, 7B) Pose description
MGIE Mot.Acc↑ 0.67 0.59 (OpenPose+ControlNet) Motion-guided image editing

On T2M, UniMotion significantly leads in R-Precision and MMDist, and its Diversity (9.583) closely matches real data (9.503). Although the single-task discrete method MoMask has a lower FID (0.045 vs 0.194 due to single-distribution fitting), it often misses crucial spatial constraints (e.g., placing hands only at shoulder height instead of raising them above the head). On M2T, it leadingly outperforms others across BertScore/CIDEr/Bleu@4 (41.2/39.3/20.7 vs 36.7/29.2/13.04). The unified multi-task model consistently outperforms its single-task variant UniMotion† of the same architecture, validating the positive cross-modal transfer brought by RGB supervision.

Ablation Study

DPA and LRA ablation (Table 9):

Configuration T2M R@3↑ M2T BertScore↑ Editing R@3↑ Description
Full UniMotion 0.841 41.2 84.94 Full model
w/o DPA 0.818 38.4 80.35 Without visual semantic alignment, all tasks degrade
w/o LRA 0.801 38.1 78.72 Uncalibrated motion path, significant drop due to cold start

Motion representation ablation (Table 6): VQ-VAE reconstruction APE = 17.15, downstream T2M R@3 = 0.771; MLD-VAE APE = 9.28, T2M R@3 = 0.810; CMA-VAE (with DPA) APE = 3.53, T2M R@3 = 0.841, achieving the best in both reconstruction and downstream performance. Merely adding DPA to the same architecture improves T2M R@3 from 0.818 to 0.841 and editing R@3 from 80.35 to 84.94.

Architecture ablation (Table 11): Keeping only the generation or semantic branch performs worse than the dual-path setup (the generation branch yields superior T2M/editing, while the semantic branch favors M2T, proving their complementarity); mixed attention outperforms pure global causal attention (editing R@3 84.94 vs 79.6); routing LoRA outperforms shared LoRA and frozen LLM (V2M MPJPE 75.0 vs 90.4 vs 99.6).

Key Findings

  • LRA degradation is more severe than DPA: Removing LRA drops T2M R@3 to 0.801 (lower than w/o DPA's 0.818), indicating that geometric cold-start calibration of the motion path is the primary bottleneck—sparse text supervision cannot provide such dense geometric feedback.
  • Continuous representation vs. discrete is a global distributional improvement: CMA-VAE's reconstruction MPJPE CDF shifts entirely to the left (~70% of sequences meet the 40mm threshold vs. ~30% for VQ). VQ-VAE exhibits a pronounced heavy tail (>2000mm) indicating catastrophic quantization failures, and end joints (wrist 212mm) suffer significantly higher errors than the torso—showing that discrete codebooks severely damage high-magnitude distal dynamics.
  • Implicit transfer of DPA across datasets: Even though HumanML3D lacks paired images, sharing the front-end parameters of the encoder allows the DPA gradient from H36M to indirectly improve pure text T2M—an implicit benefit from the "shared architectural layers + posterior alignment" combination.
  • Zero-shot generalization: Trained only on indoor H36M, V2M generalizes zero-shot to the outdoor 3DPW dataset with 93.6 MPJPE (outperforming UniPose's 99.4), showing that the continuous representation captures transferable physical body structure priors rather than overfitting.

Highlights & Insights

  • "Motion as an equal continuous modality" is the core thesis: Once motion shares the continuous representation format with RGB, cross-modal alignment naturally transitions from "forced coupling" to "structural emergence," fundamentally eradicating quantization errors. This perspective is highly transferable; any work aiming to inject new modalities into a unified MLLM can adopt this design pattern of finding a symmetrical continuous representation form first.
  • "Borrow vision during training, discard during inference" in DPA is highly practical: By using the vision-fusion encoder as a training-time teacher and the pure motion encoder as the inference-time student via reverse KL distillation and parameter sharing, the model acquires visual priors without adding any image dependency during inference. The mode-seeking explanation for the reverse KL direction also clarifies "why this direction was chosen" over empirical trial-and-error.
  • LRA uses the model’s own latent codes as "dense prompts" to break cold starts: When external supervision (text) is too sparse to calibrate a new pathway, utilizing the modality's own lossless encoding for self-reconstruction provides explicit, dense, one-to-one geometric supervision. This progressive training concept ("learning first from the most information-dense self-encoding before tackling sparse cross-modal signals") is readily applicable to warming up any new modality path. The shuffled-condition control experiment (FID degrading by 300x) is an elegant way to prove non-triviality.
  • Deterministic modality-routing LoRA: Guided by the fact that modality identity is known during sequence construction, deterministic mask routing replaces learnable MoE routing. This bypasses load-balancing losses and routing noise, adding only 2% parameters—an elegant engineering intuition of "not learning what is already known."

Limitations & Future Work

  • The authors acknowledge that the 1.5B backbone still incurs computational overhead, limiting deployment in resource-constrained scenarios. The pose-aware visual backbone depends on a frozen pre-trained human encoder, and its robustness under severe occlusion, camera motion, and diverse outdoor scenes has not been fully verified (as vision-motion alignment was primarily established on the indoor Human3.6M).
  • The current tri-modal benchmark centers on single persons, and evaluations focus on single-person motion. Per-span mixed attention is compatible with multi-person interactions, but multi-person scenarios are left for future work.
  • V2M evaluations currently use frame-level visual inputs. Extending the same interface to richer video-level temporal reasoning is a promising direction (preliminarily verified on MoVid in the appendix for multi-frame V2T).
  • Self-discovered limitation: MGIE success is defined by a somewhat loose PA-MPJPE \(\le\) 100mm hit rate threshold. Furthermore, a noticeable gap remains between V2M and expert models (e.g., SMPLer at 50.8 MPJPE); while acceptable for a generalist framework, its absolute precision still falls short of dedicated methods.
  • vs. MotionGPT: Both achieve Motion\(\leftrightarrow\)Text unification, but MotionGPT uses VQ-VAE for discrete tokenization (introducing quantization errors and remaining blind to images). This work integrates RGB with continuous CMA-VAE, achieving "continuous + tri-modal + full coverage of generation & understanding." The trade-off is a heavier backbone, but the supported tasks scale from 3 to 7.
  • vs. UniPose: UniPose tokenizes poses into a VLM, but only handles single-frame static pose estimation and image understanding without generation capabilities (at 7B). This work models continuous motion sequences, supports bidirectional understanding + generation + editing, and surpasses UniPose in V2M/V2T using only a 1.5B backbone.
  • vs. MLD-VAE (Continuous VAE path): MLD performs diffusion in the VAE latent space to balance quality and efficiency, but encodes motion in isolation and lacks cross-modal anchoring, suffering from a position-velocity trade-off (AVE 0.981). CMA-VAE anchors visual semantics into the latent space via DPA, resolving this trade-off to achieve the best overall reconstruction metrics.
  • vs. Show-o2: This paper adopts Show-o2 as a backbone, extending it from Text-RGB to Text-RGB-Motion. The key challenge lies not in reusing the pre-trained MLLM, but in "how to insert a missing, kinematically-constrained continuous motion modality into the shared space," which is precisely what the CMA-VAE+DPA+LRA trio addresses.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ First continuous Motion-Text-RGB tri-modal unified framework. The "motion as an equal continuous modality" perspective and the DPA/LRA alignment strategies are highly original.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive comparison across seven tasks + multi-level ablations (representation/architecture/attention/LoRA/DPA/LRA) + frequency/temporal reconstruction analysis + zero-shot generalization + non-triviality control experiments. Exceptionally solid.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear motivational progression. Both the reverse KL and the non-triviality of LRA are explained with theoretical intuition, with excellent coherence between the main text and appendix.
  • Value: ⭐⭐⭐⭐⭐ Provides a reusable paradigm for integrating new modalities into unified MLLMs, offering direct application potential in animation, AR/VR, robotics, and healthcare rehabilitation.