UMO: Unified In-Context Learning Unlocks Motion Foundation Model Priors

Conference: ECCV2026
Paper: ECCV paper page · Paper PDF
Project: UMO
Authors: Xiaoyan Cong, Zekun Li, Zhiyang Dou, Hongyu Li, Omid Taheri, Chuan Guo, Abhay Mittal, Sizhe An, Taku Komura, Wojciech Matusik, Michael J. Black, Srinath Sridhar
Area: Human Understanding
Keywords: human motion generation, in-context learning, motion editing, temporal inpainting, geometric control

The paper title exactly matches the ECCV 3418 record. Xiaoyan Cong and Zekun Li contributed equally, and Srinath Sridhar is the corresponding author; affiliations include Brown University, MIT, Meta Reality Lab, Max-Planck Institute for Intelligent Systems, and University of Hong Kong. The task concerns 3D skeletal human motion, not image generation. The cache contains the main paper and references but not the supplementary material cited throughout; details available only there are not reconstructed below.

TL;DR

UMO fine-tunes HY-Motion with three frame-level operation embeddings and input-level temporal fusion, extending a text-to-motion model to unified completion, editing, geometric control, and reaction generation while reaching a HumanML3D text-to-motion FID of 9.460 with only 0.207M additional conditioning parameters.

Background & Motivation

Human motion generation extends beyond turning a sentence into a motion clip. Animation workflows also require preserving keyframes, completing missing segments, changing the semantics or style of an existing motion, following a trajectory, and responding to another person's actions. Conventional approaches often introduce separate conditioning branches and training pipelines for these tasks, encoding source motion differently and limiting the sharing of data and motion priors.

Foundation models such as HY-Motion learn pose structure, joint coordination, and temporal coherence from large motion-text datasets, but their pretraining interface primarily accepts complete motion descriptions. It does not specify which input frames must remain unchanged or distinguish modification of an existing motion from generation without a reference. Simply placing coordinates in a prompt or applying training-free inversion to observed frames does not ensure that the motion backbone uses these signals correctly.

UMO starts by aligning task intent with the motion timeline, then teaches a pretrained backbone to read that intent instead of training another multitask model from scratch. Core Idea: represent the relationship between source context and target motion through frame-level preserve, generate, and edit operations, express the remaining requirements in text, and activate existing motion priors through temporally aligned fusion.

Method

Overall Architecture

The inputs are per-frame source motion, per-frame operation labels, and text describing the intended motion or constraints; frames without a source reference use zero vectors. The output is a sequence of 3D human motion rather than text or video. The interface consists of Frame-Level Meta-Operations, Unified Language Conditioning, and Temporal Fusion: the first two specify what may change and how, while fusion injects motion context into HY-Motion's flow-matching generation process.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Source motion and task"] --> B["Frame-Level<br/>Meta-Operations"]
    B --> D["Temporal Fusion"]
    A --> C["Unified Language<br/>Conditioning"]
    C --> E["HY-Motion backbone<br/>Flow-matching sampling"]
    N["Current noisy motion"] --> D
    D --> E
    E --> O["Target human motion"]

The representation follows HY-Motion: each frame has 201 dimensions comprising 3D root translation, 6D root orientation, 21 local joint rotations with 6 dimensions each, and 22 local joint positions with 3 dimensions each. All motions are resampled to 30 fps and normalized to zero mean and unit variance, so a sequence of \(T\) frames contains \(T\times201\) values. This representation matters because operation embeddings are added to raw motion features frame by frame rather than appended as a natural-language task label outside the motion sequence.

Key Designs

1. Frame-Level Meta-Operations: express tasks as source-to-target relationships

Each frame receives exactly one operation: [preserve] retains supplied content, [generate] synthesizes without a reference, and [edit] generates a modification conditioned on source motion. The labels correspond to three learnable 201-dimensional embeddings. Preserve and edit frames use real source motion, whereas generate frames use zero vectors, separating the source content from the instruction for treating it. The same pose can therefore mean either "retain this" or "modify this" depending on its operation embedding, without requiring a different backbone for each task.

For example, text-to-motion assigns [generate] to every frame, instruction-based editing assigns [edit] throughout, and prediction assigns [preserve] only to the observed prefix. Backcasting preserves the suffix, in-betweening preserves both ends, and keyframe infilling preserves sparse temporal positions; the remaining positions use generation or editing depending on whether reference content is available. Instead of attaching a single "inpainting" category to the entire sequence, this exposes the temporal locations of preserved content, allowing different temporal mask patterns within the same interface.

Dual-identity reaction generation also uses the edit configuration: another person's motion serves as source context for generating the target person's response under an interaction description. Here, editing means reference-conditioned generation rather than necessarily modifying the same identity, bringing cross-person coordination into an interface originally used for single-person motion. An interface that can describe a new task does not imply that the model can solve any new task without training. The paper demonstrates adaptation after fine-tuning on downstream data.

Likewise, [preserve] is not a hard-copy instruction executed by a program. UMO learns preservation intent and does not replace conditioning frames after sampling, so preserved frames can still have nonzero joint errors. Evaluating whether the model reads its context therefore requires a preservation-specific metric, not just a judgment of whether generated motion appears natural.

2. Unified Language Conditioning: share a text interface for semantics and geometry

Frame-level labels specify whether to preserve or generate, while text describes the intended content. Table 1 defines four prompt templates: motion descriptions, editing instructions, parameterized trajectories, and spatial constraints. The first three can specify a complete motion, its desired modification, or the type and parameters of a line, arc, sinusoid, or Bézier curve. Trajectories are serialized with type and params fields; obstacle avoidance specifies start and end positions together with obstacle centers and safety radii in meters.

This does not discretize every coordinate into a new motion token or introduce a separately trained spatial encoder. Instead, structured numerical specifications pass through the existing language-conditioning path. HY-Motion already uses Qwen3-8B and CLIP text encoders; UMO introduces no editing-specific or geometry-specific conditioning module. Fine-tuning must teach the motion backbone to map these previously unfamiliar language features to executable trajectory constraints, so downstream supervision remains necessary.

The design reduces interface expansion costs: a new curve type can be described with a new prompt template rather than a backbone redesign. However, template extensibility establishes input compatibility, not numerical precision on unseen geometric relations; the experiments also show that optimization-based control retains lower trajectory error.

3. Temporal Fusion: retain frame alignment without extending the sequence

The model adds operation embeddings to source motion and processes the result with a context MLP encoder. This encoder is initialized by copying HY-Motion's pretrained input encoder, then adapted to clean, task-annotated motion context. Noisy motion passes through the original input encoder; the two feature streams are added elementwise at matching temporal positions before entering the DiT backbone. In notation equivalent to the paper's textual description:

\[ x'_t = E_{\mathrm{in}}(x_t) + E_{\mathrm{ctx}}\!\left(s + \operatorname{Emb}(\tau)\right). \]

Here, \(s\) is the source sequence, \(\tau\) is the per-frame operation sequence, and \(x_t\) is noisy motion at flow-matching time \(t\). Fusion neither pools the timeline into a global vector nor doubles sequence length, preserving keyframe locations without the additional attention cost of a longer sequence. Generation still uses HY-Motion's multimodal DiT; an unchanged backbone architecture should not be confused with permanently frozen backbone weights.

The paper tests three alternatives. Sequential concatenation joins context and noisy motion into \(2T\) tokens, with an asymmetric attention mask making context read-only, but substantially increases computation. AdaLN pools the entire context into a global vector for network modulation, reducing cost while losing frame-specific constraint locations. ControlNet freezes the backbone and copies its double-stream blocks into a trainable branch, injecting conditions through zero-initialized residual connections at a much larger parameter cost. Temporal fusion succeeds not by making the network deeper but by retaining the required temporal resolution at an inexpensive input location.

A Worked Example

Consider a walking clip with fixed beginning and ending segments and missing motion in between; this is an interface illustration, not an additional experimental setting from the paper. Place the observed endpoint motions into the source sequence with [preserve], fill the missing region with zeros and [generate], and provide a description of the intended walking motion. The context encoder reads "known pose plus preserve embedding" and "zero vector plus generate embedding," then fuses these features framewise with current noisy motion to generate a continuous connection. As flow-matching sampling progresses, the generated motion state changes while supplied source motion and operation labels remain conditioning inputs. To modify the style of an existing middle segment instead, use its real source motion with [edit] and replace the text with a style prompt; no separate editing network is required.

Loss & Training

UMO follows rectified flow. With Gaussian noise \(x_0\) and clean motion \(x_1\), the intermediate state and velocity-regression objective are:

\[ x_t=(1-t)x_0+t x_1,\qquad \mathcal{L}_{\mathrm{FM}}=\mathbb{E}_{t,x_0,x_1}\left\|v_\theta(x_t,c,t)-(x_1-x_0)\right\|_2^2. \]

Here, \(c\) denotes conditions and \(t\in[0,1]\). Inference solves the velocity-field ODE from \(t=0\) to \(t=1\), with the conditioning interface adapting the same generative process to different tasks. The backbone is the 460M-parameter HY-Motion-Lite. Training uses 4 NVIDIA B200 GPUs, a batch size of 256, and a learning rate of \(5\times10^{-5}\). UMO-Unified trains jointly across all tasks and datasets for 100k steps, while each UMO-Expert trains on its respective task for 6k steps. Inference uses a 50-step Euler ODE solver and classifier-free guidance scale 2.0; all evaluations run on NVIDIA A6000 GPUs. The 0.207M figure counts additional conditioning parameters, not all trainable parameters. The main paper does not provide a complete freezing specification, task sampling proportions, or additional loss weights, so this should not be recast as training only a small adapter.

Key Experimental Results

Main Results

HumanML3D supports text-to-motion and temporal inpainting, MotionFix supports instruction-based editing, and Inter-X and InterHuman support reaction generation. Trajectory control and obstacle avoidance each use 2,000 constructed sequences; the cache does not include the supplementary construction and split details.

The following selection from Table 4 uses the official MotionStreamer evaluator. FID is a distributional distance, with lower values preferred; R@3 is a top-3 text-motion retrieval proportion, not percentage-form reporting; MM-D measures text-motion feature distance and has no physical unit.

Method	FID, lower is better	R@3, higher is better	MM-D, lower is better
MotionStreamer	11.790	0.859	16.08
HY-Motion	61.035	0.876	17.53
UMO-Expert	17.04	0.931	15.49
UMO-Unified	9.460	0.933	15.22

Temporal inpainting in Table 5 instead uses the MotionLab evaluator, downsampling outputs from 30 fps to 20 fps and converting them to a 263-dimensional representation, so its FID is not directly comparable with Table 4. Unified-model FID is 0.056, 0.057, 0.050, and 0.040 for prediction, backcasting, in-betweening, and keyframe infilling, respectively. MPJPE is mean per-joint position error across the full sequence, while [P]-MPJPE restricts evaluation to frames labeled [preserve]; both use cm. For keyframe infilling, the unified model reports 2.67 cm and 0.95 cm, respectively, rather than zero-error copying.

The following selection from Table 6 reports MotionFix generated-to-target retrieval. Batch retrieves within a batch, whereas Full retrieves from the full candidate set; R@3 is explicitly reported as a percentage, and AvgR is average rank, with lower values preferred.

Method	Batch R@3 (%)	Full R@3 (%)	Full AvgR
PartMotionEdit	90.21	53.36	16.24
UMO-Expert	100.0	91.31	1.76
UMO-Unified	100.0	91.51	1.75

Ablation Study

The following results come from Table 3's keyframe-infilling architecture ablation. Overheads are relative to HY-Motion when generating 360 frames with a 50-step Euler solver; parameter, FLOPs, and latency units are M, G, and s. FID uses the MotionStreamer evaluator after conversion to a 272-dimensional representation; [P]-MPJPE downsamples 30 fps outputs to 20 fps to match Table 5. Consequently, the FID values in Tables 3 and 5 are not directly comparable even though both concern keyframe infilling.

Conditioning architecture	[P]-MPJPE (cm)	FID	Extra parameters (M)	Extra FLOPs (G)	Extra latency (s)
ControlNet	5.19	6.520	234.2	85.12	0.49
AdaLN	11.1	8.860	4.400	1.660	0.02
Sequential Concat	2.04	11.77	0.207	198.6	0.89
Temporal Fusion	0.95	0.476	0.207	0.140	0.01

Key Findings

• Temporal fusion obtains both the lowest preserved-frame error and the lowest FID in the architecture ablation. AdaLN has the highest R@3 at 0.922 but a preserved-frame error of 11.1 cm, showing that semantic retrieval cannot replace conditioning fidelity.
• In Table 7, the unified model has trajectory error 18.78 cm and average inference latency 0.759 s/sample, versus OmniControl's 17.89 cm and 68.10 s. UMO approaches its precision at lower latency but is not more precise.
• Optimization-based TLControl achieves 2.930 cm trajectory error at 3.243 s, illustrating the precision-latency trade-off. Table 7 labels success rate (%) but gives UMO-Unified a cell value of 0.95; this note retains the raw value and flags the unit ambiguity rather than silently interpreting it as 0.95% or 95%.
• For InterHuman reaction generation in Table 8, the unified model achieves FID 2.055 versus InterMask's 2.990, but its R@1 is 0.431 versus InterMask's 0.462. It does not lead on every metric.

Highlights & Insights

• The most reusable idea separates source content from operation intent. The same input pose can support preservation, modification, or cross-person reaction without encoding every task difference in network architecture.
• The location of conditioning matters more than parameter count alone. Global pooling loses information required by frame-level constraints, while sequence concatenation increases cost; aligned input addition offers a direct alternative.
• Structured text connects geometric control to an existing language encoder. The experiments support learning to use this interface through fine-tuning, but do not independently prove that geometric reasoning was fully present in the pretrained language encoder.

Limitations & Future Work

• The authors note that the three operations act at whole-body, per-frame granularity and do not support body-part-level control; the method also does not handle music or speech conditioning.
• Reviewer observation: the unified model trains for 100k steps versus 6k for experts, with different data coverage. Its advantage cannot be attributed exclusively to task synergy without training-budget-matched controls.
• Reviewer observation: learned preservation does not guarantee exact keyframes, and geometric control does not replace high-precision optimization. Part-level operations, hard-constraint correction, and compositional-task evaluation are potential extensions, not implemented components of this paper.
• The cache lacks supplementary material, preventing verification of task mask ratios, geometric data splits, and further ablations. Claims about arbitrary new tasks should be read as interface expressibility rather than verified unlimited generalization.

Related Work & Insights

• vs HY-Motion: UMO inherits text-to-motion priors and learns contextual conditioning through operation embeddings and temporal fusion; the base model does not solve every downstream task zero-shot.
• vs MotionLab / CondMDI: These provide important references for unified motion tasks or temporal inpainting. UMO emphasizes larger-scale pretrained priors and learned preservation rather than hard replacement of conditioning frames.
• vs ControlNet / AdaLN: Conditioning should be assessed not only by parameter count or retrieval scores but also by whether information remains at the temporal resolution the task requires.
• vs InterMask: Single-person priors transfer to reaction generation, but better distributional fidelity does not automatically imply better text alignment; both metric families should be reported.

Rating

• Novelty: 4/5. Frame-level operations and a unified conditioning interface offer a concise, systematic adaptation of strong motion priors.
• Experimental Thoroughness: 4/5. Multiple tasks and architecture ablations are covered, but training budgets differ and some replication details reside in unavailable supplementary material.
• Writing Quality: 4/5. Task instantiation and architecture comparisons are clear, while success-rate units and some generalization claims require care.
• Value: 4/5. The design provides a reusable controllable interface for existing motion foundation models.