ActionPlan: Future-Aware Streaming Motion Synthesis via Frame-Level Action Planning

Conference: ECCV 2026
Paper: Official paper page · Paper PDF
Project: https://coral79.github.io/ActionPlan/
Area: Human Understanding
Keywords: Text-to-motion generation, frame-level action plans, streaming generation, latent-specific noise scheduling, rectified flow

TL;DR

ActionPlan generates temporally aligned semantic action plans before progressively synthesizing 3D human motion, combining offline generation and streaming output in one model and reducing streaming FID from MotionStreamer's 11.790 to 5.735 on the HumanML3D-272 test set.

Background & Motivation

Text-to-motion generation must produce plausible poses while also executing the actions and ordering requested by a description. For “walk forward, turn around, then sit down,” a global text embedding expresses the overall intention but does not explicitly assign actions to temporal positions. Offline methods can use bidirectional sequence context to improve coherence, but their dependence on complete sequences conflicts with immediate output. Streaming methods such as MotionStreamer generate incrementally in a causal latent space, yet can omit later actions or put them in the wrong order.

The choice is not simply between observing actual future motion and ignoring the future altogether: future poses remain unknown, but their intended semantics can be predicted first. ActionPlan separates the semantic timeline from detailed joint motion, allowing early output to anticipate a later turn or sitting action. It also addresses an efficiency bottleneck: finishing every denoising step for one motion token before starting the next creates substantial cumulative waiting. Core idea: use one generative model to establish a frame-level action plan, then execute it through independently scheduled, overlapping motion denoising, separating access to future semantics from the timing of motion output.

Method

Overall Architecture

The input is a sequence-level natural-language description, and the output is a sequence of 3D human poses that can drive SMPL—not an RGB video or a restored image. Each pose uses a 272-dimensional representation containing root motion and positions, velocities, and rotations for 22 joints. ActionPlan adopts MotionStreamer's Causal Temporal AutoEncoder, freezes it after training, and works in its continuous latent space. The causal decoder can turn completed latents into poses without waiting for the entire sequence.

The added mechanisms are frame-level semantic alignment, heterogeneous-noise joint training, action-plan-first generation, and overlapping progressive denoising. Training teaches the model to generate both motion and semantic latents; inference selects when each type becomes clean. Offline inference completes the plan before activating motion positions in random order. Streaming inference generates the plan alongside the first motion latent, then activates subsequent motion positions chronologically. Thus, planning is neither an external text-model call nor a requirement to generate all future poses before beginning output.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Sequence text and training motion"] --> B["Frame-level semantic alignment"]
    B --> C["Heterogeneous-noise joint training"]
    C --> D["Action-plan-first generation"]
    D --> E["Overlapping progressive denoising"]
    E -->|"Offline random / streaming temporal order"| F["Causal decoding to human motion"]

Key Designs

1. Frame-level semantic alignment: attach the global intention to temporal positions

With only a sequence-level condition, the generator must infer action boundaries implicitly, making short actions within longer descriptions easy to miss. ActionPlan encodes frame-level descriptions with CLIP and compresses them through a pretrained MLP autoencoder to a 16-dimensional space matching the motion latent dimension. Unlike the temporal motion encoder, the text autoencoder operates independently on each latent, preserving distinctions between neighboring action labels. Its training combines embedding reconstruction, neighborhood preservation for label retrieval, and variance regularization against dimensional collapse.

Text latents are downsampled temporally by a factor of four and concatenated with aligned motion latents before entering the shared Transformer denoiser. “Frame-level” therefore refers to temporal semantic alignment; it does not mean that inference generates a separate natural-language sentence for every original frame. The plan consists of continuous text latents, which can be decoded to CLIP embeddings and displayed as labels through nearest-neighbor retrieval. The original sequence-level CLIP condition remains available: it expresses the complete intention, while the plan supplies its temporal organization.

2. Heterogeneous-noise joint training: learn from context at different stages of completion

Conventional sequence diffusion commonly places every position at the same denoising stage. That is a poor match for streaming inputs containing completed history, partially generated current motion, and ungenerated future positions. ActionPlan assigns motion latents separate noise timesteps, while all action-plan latents share another global timestep. The denoiser receives these timestep conditions and predicts rectified-flow velocities for both modalities, learning to use a clean plan together with partially denoised motion. Motion timestep sampling follows the cited mean-timestep algorithm rather than simply sampling every position independently and uniformly; its precise implementation is deferred to supplementary material.

Training combines HumanML3D-272 sequence descriptions with BABEL frame-level annotations where the datasets overlap. Only approximately 30% of the motion samples carry these fine-grained labels, so requiring complete labels would discard most available motion data. The model retains the full motion training set and enables text supervision only when frame-level labels are present; the remaining examples still supervise motion generation. This preserves the same benchmark motion pool used by the baselines, but adds finer semantic supervision, an important qualification for the comparison.

3. Action-plan-first generation: settle the semantic timeline before motion details

Joint training does not require synchronous generation at inference. In offline stage one, all motion latents remain pure noise while the model denoises the action-plan latents. Stage two holds the completed plan as a semantic condition, giving motion synthesis access to both available motion context and the intended actions at later positions. This differs from guessing the future solely from generated poses, and from generating complete future poses before filling earlier motion.

Streaming inference combines plan generation with denoising the first motion latent to reduce startup waiting. Subsequent motion latents are activated in temporal order under the resulting plan. “Future-aware” means access to a predicted semantic future under the current description—not advance knowledge of user instructions that have not arrived. The comparison between joint text–motion generation and plan-first inference shows that frame-level supervision alone does not exhaust the benefit: fixing the semantic plan before most motion synthesis provides an additional gain.

4. Overlapping progressive denoising: start new tokens before earlier tokens finish

Fully denoising one motion latent before beginning the next serializes expensive computation across the sequence. ActionPlan maintains waiting and active latents, periodically activating another position while existing positions continue denoising. Several active positions can therefore be updated together at different noise levels. The selected configuration activates another latent after two denoising steps, while each latent still follows a 25-step rectified-flow trajectory; this is not two-step total generation. Offline inference uses random activation order, whereas streaming uses chronological activation and immediately decodes completed tokens.

The same ability to handle clean and noisy positions supports editing and in-betweening without another trained model. For editing, preserved motion latents are fixed at zero noise while selected positions are regenerated under the new prompt. For in-betweening, clean start and end anchors condition generation of the intermediate motion. Long motion is generated in successive chunks, each conditioned on its own prompt and the previous chunk's final tokens. These applications demonstrate the flexibility of the sampling interface, but their largely qualitative evaluation should not be mistaken for a comprehensive editing benchmark.

A Worked Example

Consider “walk forward, turn around, then sit down” as an explanatory input rather than an additional measured example. CLIP first encodes the global description, while the semantic latents establish a walk–turn–sit timeline. In streaming mode, the plan is denoised alongside the first motion token, so later walking poses can already condition on the upcoming turn and sitting action. New motion latents join the active set chronologically while earlier ones continue denoising; completed tokens are decoded and emitted. A subsequent editing operation could preserve selected motion latents and regenerate another region, but that does not mean already displayed motion can be retroactively changed in a live stream.

Loss & Training

The denoiser uses 16 Transformer layers, 16 attention heads, and a hidden dimension of 1024. Motion training minimizes the squared error between predicted flow velocity and the clean motion latent minus Gaussian noise; annotated examples receive corresponding text-velocity supervision. The text autoencoder's neighborhood-preservation and variance objectives should be distinguished from motion reconstruction and the main model's mentioned latent-variance term. Equations (2)–(4) in the local text extraction contain missing symbols, particularly in the masked text loss and total objective, so this note does not reconstruct an unverified combined formula. The main text establishes masked supervision and velocity prediction, but exact loss weights, mean-timestep sampling details, and the complete text-autoencoder setup require the unavailable supplement.

Key Experimental Results

Main Results

The selected entries below come from Table 1 on the HumanML3D-272 test set using MotionStreamer's TMR-based evaluator. Baselines were retrained from their official implementations with the same 272-dimensional motion representation; ActionPlan additionally uses BABEL frame labels on overlapping training examples. FID measures the distance between generated and real motion feature distributions, with lower values preferred. R@3 is top-three text–motion retrieval accuracy, with higher values preferred; MatchS measures text–motion feature distance, with lower values preferred.

Method	Mode	FID ↓	R@3 ↑	MatchS ↓
MotionStreamer	Streaming	11.790	0.859	16.081
MARDM	Offline baseline	7.044	0.860	15.892
ActionPlan	Streaming	5.735	0.877	15.315
ActionPlan	Offline	5.522	0.892	15.09

Calculated from Table 1, streaming ActionPlan reduces FID by approximately 51.4% relative to MotionStreamer and 18.6% relative to MARDM; offline ActionPlan reduces it by approximately 21.6% relative to MARDM. These are relative changes in FID, not directly measured percentage improvements in physical motion quality. Section 4.2 measures latency on one NVIDIA A100 for a motion token representing four frames: ActionPlan takes 146 ms for the first token and 40 ms for subsequent tokens. The corresponding first/subsequent latencies are 210/210 ms for MARDM and 360/360 ms for MotionStreamer. Consequently, subsequent-token speedups are 5.25× and 9× respectively; those ratios do not share the same baseline.

Ablation Study

The following selected Table 2 entries all use offline generation on the same HumanML3D-272 test split. The no-frame-text configuration retains latent-specific scheduling; co-generation adds frame-level text but does not establish the plan before motion generation.

Configuration	FID ↓	R@3 ↑	MatchS ↓
Vanilla parallel diffusion	6.520	0.866	15.410
Vanilla diffusion with action plan	6.200	0.880	15.340
Latent-specific schedule, no frame-level text	6.952	0.873	15.390
Frame-level text–motion co-generation	6.093	0.878	15.234
Full ActionPlan	5.522	0.892	15.086

The full streaming configuration has FID 5.878 in Table 2, versus 5.735 in Table 1; the main text does not explicitly explain this difference. This note retains each table's reporting context rather than treating the values as one identical run. Table 3 additionally reports offline FID 7.863 when training only on the approximately 30% overlapping subset, versus 5.522 with full mixed-data training; that comparison also changes the amount of motion training data.

Key Findings

• Both semantic supervision and generation order matter: offline FID moves from 6.952 without frame text to 6.093 with co-generation and 5.522 with plan-first generation in Table 2.
• Not every added component independently improves FID: latent-specific scheduling without frame text scores 6.952, worse than vanilla parallel diffusion at 6.520. Scheduling and semantic planning must be assessed together.
• In Table 4, fully parallel sampling achieves better FID, 5.420 versus 5.522, but lower R@3, 0.886 versus 0.892. The chosen two-step activation interval is a compromise, not a winner on every metric.
• The short-motion user study in Section 4.4 covers 20 prompts and 30 participants, with ActionPlan receiving 67.5% preference. This supports the quantitative trend but remains a relatively small study.

Highlights & Insights

• Separate semantic foresight from future poses. An early semantic timeline does not require complete future motion, explaining why streaming need not exclude future context.
• Train jointly, generate in stages. The central change is not an extra planner, but a model trained to handle different completion levels across semantic and motion latents.
• Use noise levels as a control interface. Clean regions become anchors and noisy regions remain editable, letting streaming, editing, and in-betweening share mechanisms while retaining distinct evaluation requirements.

Limitations & Future Work

• Author-stated limitations: the model lacks finger articulation and facial expressions, as well as scene and object awareness. Plausible body motion therefore does not establish successful interaction with an environment.
• Reader assessment: evidence mainly concerns HumanML3D-272 with partial BABEL labels, not broad cross-distribution or open-environment generalization. Obtaining fine-grained supervision is also a practical prerequisite.
• Evidence boundaries: editing, in-betweening, and long-motion applications are largely qualitative; streaming values differ between Tables 1 and 2, some extracted equations are damaged, and the supplement is absent from the local cache.
• Useful next evaluations: measure latency under prompt changes, error accumulation across chunks, and object-contact constraints before interpreting longer generated sequences as general interactive competence.

Related Work & Insights

• Versus MotionStreamer: ActionPlan inherits its causal motion latent space and evaluation protocol, adding semantic plans and overlapping denoising rather than redesigning the pose representation.
• Versus MARDM: both use continuous motion latents, and MARDM provides an important offline quality reference. ActionPlan aims to retain strong quality under chronological streaming activation through explicit planning.
• Versus UniMotion and Diffusion Forcing: the former informs joint motion–text modeling, while the latter motivates flexible frame-specific noise levels. ActionPlan combines early generation of temporally aligned semantics with task-dependent motion sampling orders.

Rating

• Novelty: 4/5. Plan-first generation and flexible noise scheduling form a task-specific contribution, although several underlying components are inherited.
• Experimental Thoroughness: 3/5. Retrained baselines, ablations, latency measurements, and a user study are useful, but multi-dataset tests and quantitative downstream evaluations remain limited.
• Writing Quality: 3/5. The main mechanism is understandable, while streaming results and speedup references require careful reconciliation.
• Value: 4/5. Directly relevant to low-latency digital-human animation, but not a substitute for environment-constrained motion control.