ARMS: Anchor–Relational Motion Streaming for Seamless Solo-Social Motion Transitions

Conference: ECCV2026
Paper: Official page · PDF
Project: https://hkliu.com/arms/
Area: Human Motion Generation / Human-Human Interaction / Causal Diffusion
Keywords: anchor-relational representation, causal latent space, segment-wise denoising, mode gating, solo-social transitions

TL;DR

ARMS continuously extends one person's trajectory while locating a partner relative to that person, using causal segment-wise denoising and relational gating to enter or leave interactions without restarting generation; solo-interaction transition PJ falls from 3.131 for adapted InterMask to 0.077, although ARMS does not lead every generation-quality metric.

Background & Motivation

Text-to-motion systems can synthesize a short walking sequence or an interaction such as two people bowing. Real behavior is not organized into independent clips: someone walks alone, meets another person, interacts, and eventually leaves. Methods such as InterGen and InterMask predominantly target a fixed two-person configuration. Supplying an earlier clip as context for the next one does not automatically preserve position, velocity, or interpersonal geometry. A new action can match its caption while still beginning with an abrupt change of stance or location.

Motion representation makes this particularly difficult. A single person's pose is often expressed around the root, with incremental updates accumulated into a global trajectory. If two people independently accumulate those updates, their integration errors can disrupt their relative alignment. Global coordinates preserve their spatial relationship more directly but tie generation to the coordinate distributions observed during training. Recanonicalizing each segment can instead create discontinuities at segment boundaries. The challenge is therefore not merely to enlarge the generation window, but to assign different responsibilities to individual progression and interpersonal alignment.

Core idea: maintain one continuously integrated anchor trajectory, position the interacting partner relative to that anchor, and use relational gating with overlapping denoising in a causal latent space to connect solo and social behavior within one extendable generation process.

Method

Overall Architecture

ARMS takes incrementally updated text instructions, previously generated motion, and the current solo or two-person configuration. It returns motion segments that continue the existing stream. An asymmetric dynamics representation organizes the agents' states, shared causal encoding produces latent streams, and a gated relational denoiser models temporal and interpersonal dependencies. At inference, overlapping refinement advances the stream while dynamic history conditioning adjusts the retained context when instructions change.

Causality here is primarily between latent segments: tokens within the current segment can be refined together, but future segments are not visible. The causal encoder also avoids representing an earlier motion state using future frames, which would make an ordinary bidirectional encoder unsuitable for incremental synthesis.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Text, history, and<br/>solo or two-person mode"] --> Representation["Asymmetric dynamics<br/>representation"]
    Representation --> Encoding["Shared causal encoding"]
    Encoding --> Denoising["Gated relational denoising"]
    Denoising --> Streaming["Overlapping denoising<br/>and dynamic history"]
    Streaming --> Output["Continuous motion stream"]
    Output -->|Retained history and new instructions| Streaming

Key Designs

1. Asymmetric dynamics representation: only one agent integrates the global trajectory

Both agents use the same state fields, but their positions are recovered differently. Each frame contains root yaw encoded as sine and cosine, root velocity and relational displacement on the world-frame horizontal plane, root-local joint positions and velocities, continuous six-dimensional rotations for non-root joints, and four foot-contact indicators. The sine-cosine orientation avoids discontinuities at angular wraparound. Local pose coordinates avoid forcing the model to relearn the same articulation at every global location.

For the Anchor agent, relational displacement is fixed to zero and the horizontal root position is recovered by integrating velocity. During interaction, the Relational agent stores its horizontal displacement from the Anchor, so its horizontal position is recovered by adding this displacement to the Anchor position rather than independently integrating another global trajectory. This specifically reduces relative drift caused by two independent integrations; it does not guarantee that the Anchor itself has no absolute-position error. Identical fields with different dynamical roles also permit parameter sharing across the branches.

Solo generation uses the Anchor state with zero relational displacement, so it does not require a different generator. Entering interaction activates a partner located relative to an already evolving Anchor. This is easier to connect continuously than asking the model to suddenly synthesize an entirely new pair of global trajectories. However, the paper assumes the agents are already within a plausible interaction-distance range covered by training data; it does not solve autonomous long-distance approach planning.

2. Shared causal encoding: place solo and interaction data in one latent space

The temporal variational autoencoder uses one-dimensional causal convolutions and residual blocks. Its encoder produces Gaussian parameters and samples continuous latents through reparameterization; its decoder reconstructs motion from the available latent prefix. Temporal downsampling is fourfold and the latent dimension is 64, so 300 motion frames correspond to 75 latent timesteps. This reduces the cost of modeling long windows directly in high-dimensional skeletal space.

In an interaction, each person is encoded separately by the same shared-weight encoder. Interpersonal structure is carried by the relational displacement and subsequent relational attention, rather than by an encoder that concatenates both bodies into a single specialized high-dimensional state. HumanML3D and InterHuman motions are converted to the shared representation before per-agent encoding, allowing solo data to contribute to the unified streaming model. Generated motions are converted back to each dataset's native representation for evaluation; the authors describe the conversion as deterministic and invertible.

3. Gated relational denoising: refine both people together while excluding future segments

The Anchor and Relational streams, each containing \(L\) latent timesteps, are concatenated. Each agent's timeline is partitioned into segments of \(S\) latents. Tokens in a segment share a noise level, and corresponding segments of both agents receive identical noise levels. This prevents one person from being nearly fixed while the other's corresponding motion remains highly uncertain. Earlier segments have lower noise, whereas later segments retain more freedom to change. With the default \(S=5\), one segment corresponds to 20 motion frames rather than a single latent decision.

The Transformer denoiser processes both streams jointly. Attention visibility requires two conditions: a key must belong to the same or an earlier temporal segment, and the current mode must allow the connection between the two agent identities. Interaction mode permits both within-agent and cross-agent access to current and past segments. Solo mode retains only Anchor-to-Anchor connections and ignores the Relational branch. The rule in the paper's Equation 10 can be expressed as:

\[ M_{qr}=\begin{cases} 0, & s(r)\leq s(q)\ \text{and}\ \Gamma(a(q),a(r))=1,\\ -\infty, & \text{otherwise}. \end{cases} \]

Here \(a\) denotes agent identity, \(s\) is the segment index on that agent's own timeline, and \(\Gamma\) is the mode gate described above. A standard triangular mask over concatenated token positions would implement the wrong visibility because the Relational stream follows the entire Anchor stream in storage, not in physical time. Noise embeddings enter each layer through AdaLN, RoPE supplies temporal position, and a frozen DistilBERT provides text tokens through cross-attention in every layer. The denoiser predicts velocity-like residuals for flow matching rather than classifying the next pose in a single step.

4. Overlapping denoising and dynamic history: preserve continuity without trapping the stream in an old instruction

Inference uses \(K\) refinement iterations, but adjacent segments do not wait for their predecessors to finish completely. Instead, they begin denoising with an offset of \(\delta\) iterations. The defaults are \(K=50\) and \(\delta=5\), so later segments are already being refined at higher noise levels while earlier ones become committed. Together with segment-wise noise sharing, this schedule avoids generating isolated chunks and joining them afterward. Sampling remains iterative: the term streaming alone does not establish a particular real-time frame rate.

When the prompt is unchanged, ARMS retains the full context window of \(L=75\). When the prompt changes, it keeps a shorter history \(H\) and replaces the text condition for future segments. Dropping all history would lose posture and velocity continuity, whereas retaining too much old context could delay adaptation to the new instruction. The main paper does not specify a numerical value for \(H\) and refers the exact inference procedure to supplementary material, so no value is inferred here.

Entering interaction opens the relational gate. The new branch can start from noise or from encoded relative dynamics derived from existing motion; noise initialization is the default reported in the main paper. Leaving interaction masks the Relational branch while the Anchor continues. This is an interface for changing agent configurations, not a planner that automatically decides when or with whom to interact.

A Worked Example

Consider the instruction sequence "walk alone, bow together, then turn and leave." This is a walkthrough of the types of scenes shown in the paper, not an additional quantitative experiment. During walking, only the Anchor stream is active and its pose and root velocity extend the existing history. The bowing instruction retains a short history and activates the Relational agent, whose position is expressed relative to the Anchor; corresponding segments of both agents are denoised together.

On the departure instruction, generation does not restart from a new first frame. Relational visibility is disabled and solo motion continues from the existing Anchor state. If the partner initially lies outside the supported interaction range, solo locomotion must first bring the agents close enough. The example therefore does not imply seamless switching from arbitrary initial positions.

Loss & Training

The main paper describes a causal VAE and flow-matching-based causal diffusion, but does not fully expand the VAE loss weights or the training objective. Some equations in the cached text also have missing characters, so no missing loss formula is reconstructed here. Reported settings are an eight-layer denoiser with hidden width 512 and four attention heads, batch size 64, maximum training length 300 frames, and 500 epochs. AdamW uses a learning rate of \(2\times10^{-4}\), betas \((0.9,0.99)\), and weight decay \(10^{-5}\).

The cross-scenario streaming model is jointly trained on HumanML3D and InterHuman, whereas standard InterHuman interaction comparisons use dataset-specific models; these should not be conflated. The HumanML3D variant contains 26,846 motion sequences. InterHuman contains 7,779 two-person sequences and 23,337 descriptions. InterX contains 11,388 motion sequences and uses an SMPL-X skeleton. InterX results support applicability across skeletal representations, but the main paper does not provide enough detail to describe this as training-free, zero-shot cross-skeleton transfer.

Key Experimental Results

Main Results

The following selection from Tables 1 and 2 retains the reported 95% confidence intervals. R@Top3 measures top-three text-motion retrieval accuracy and is higher-is-better. FID compares generated and real motion feature distributions; MM Dist measures mean embedding distance between paired captions and motions. Both distances are lower-is-better. Values from different datasets should not be compared directly.

Dataset	Method	R@Top3 ↑	FID ↓	MM Dist ↓
InterHuman	HINT	0.672 ± 0.004	3.100 ± 0.035	3.796 ± 0.001
InterHuman	TIMotion	0.734 ± 0.006	4.702 ± 0.069	3.769 ± 0.001
InterHuman	ARMS full-window	0.764 ± 0.004	4.436 ± 0.069	3.763 ± 0.002
InterHuman	ARMS streaming	0.723 ± 0.005	4.444 ± 0.068	3.778 ± 0.001
InterX	Interact2Ar	0.737 ± 0.00	0.148 ± 0.01	3.581 ± 0.01
InterX	HINT	0.682 ± 0.003	0.278 ± 0.012	4.007 ± 0.016
InterX	ARMS streaming	0.780 ± 0.003	0.279 ± 0.010	3.405 ± 0.016

ARMS full-window generation has stronger InterHuman retrieval results, but its FID remains higher than HINT's. On InterX, ARMS streaming improves R@Top3 and MM Dist, yet does not beat Interact2Ar's FID. The paper's blanket statement about outperforming on all metrics is inconsistent with its tables; conclusions should be metric-specific.

Streaming evaluation separately constructs 64 groups of eight consecutive interactions and 64 solo-interaction-solo sequences. Transition metrics use a two-second window centered on each boundary. Jerk is the time derivative of acceleration; PJ is the peak instantaneous jerk magnitude across joints, and AUJ accumulates deviations from average jerk. PJ should approach the real-motion reference, not necessarily zero, while lower AUJ is preferred. The following table reports means from Table 3, omitting confidence intervals.

Method	Solo↔interaction PJ	Solo↔interaction AUJ ↓	Interaction↔interaction PJ	Interaction↔interaction AUJ ↓	Interaction subsequence R@Top3 ↑	Interaction subsequence FID ↓
Ground truth	0.046	0.672	0.074	0.584	0.643	8.344
InterMask Inpainting	Not reported	Not reported	0.269	22.562	0.631	28.554
InterMask Adapted	3.131	7.444	0.265	22.440	0.557	32.307
ARMS streaming	0.077	1.070	0.071	2.925	0.509	30.620

ARMS substantially reduces transition discontinuities, but its interaction subsequence retrieval score is below both InterMask variants. The authors attribute this to continuous motion taking time to adapt to a new instruction, whereas the baselines behave more like restarting for each prompt. The evidence supports a trade-off between continuity and immediate semantic adherence, not a free improvement across all criteria.

Ablation Study

These are means from Table 4. Its full-model FID is 4.446 and is deliberately not replaced by the 4.444 reported in Table 1. Ground-truth transition references are PJ 0.074 and AUJ 0.584.

Configuration	R@Top3 ↑	FID ↓	PJ, closer to real is better	AUJ ↓
ARMS default	0.723	4.446	0.071	2.925
InterGen representation	0.723	4.553	0.130	3.367
Without autoregressive generation	0.764	4.436	0.114	3.209
Segment size S=1	0.674	6.832	0.131	3.353
Segment size S=3	0.718	4.445	0.070	2.708
Segment size S=10	0.729	4.600	0.082	2.980
Sampling steps K=10	0.722	4.731	0.105	3.064
Refinement offset δ=50	0.662	8.645	0.065	3.068

Key Findings

• The representation ablation chiefly supports smoothness: InterGen's representation changes PJ from 0.071 to 0.130 and AUJ from 2.925 to 3.367, while mean R@Top3 is unchanged. It does not demonstrate a retrieval improvement.
• Joint segment refinement matters: S=1 has FID 6.832, substantially worse than the default. However, S=3 has better AUJ, 2.708 versus 2.925, so S=5 is not optimal on every metric.
• At δ=50, PJ is numerically lower but not closer to the real-motion reference, and FID rises to 8.645. Selecting a configuration solely for minimum jerk would misjudge its quality.

Highlights & Insights

• Separating error responsibilities matters more than simply adding relational attention. Integrating only the Anchor and recovering the partner through relative position directly targets misalignment from independently drifting trajectories.
• Mode changes become changes to a shared state and attention visibility. This lets solo and interaction data participate in one generation process instead of switching between specialized generators.
• Dynamic history suggests a reusable mechanism for updating conditions in continuous control. Retaining motion state while reducing old semantic context can enable adaptation, although the appropriate history length still needs validation.

Limitations & Future Work

• The authors explicitly note that solo generation does not condition on surrounding people or the environment. Crowd avoidance, obstacle avoidance, and correct environmental contact are therefore not guaranteed.
• The formulation currently covers at most two people and assumes plausible distances when interaction activates. Scaling requires higher-order relationship reasoning and Anchor selection, not merely duplicating pairwise relations.
• Hard relational gating can still cause abrupt changes in difficult transitions. Gradually blending relational attention is proposed as future work, without a corresponding quantitative validation.
• From an evaluation perspective, the streaming benchmark is constructed by the authors and primarily compares adapted InterMask. It does not establish universal superiority over streaming methods, and lower AUJ does not prove accurate contact or absence of interpenetration.
• The main paper lacks end-to-end latency, throughput, memory measurements, and longer-horizon drift curves. It also does not independently ablate mode gating or history length H; these remain deployment-relevant gaps.

Related Work & Insights

• Versus InterGen: Global representation helps preserve interpersonal geometry, while ARMS retains an incremental Anchor trajectory and explicit partner offsets. The ablation supports smoother transitions, not universal improvement in short-clip metrics.
• Versus InterMask: InterMask uses masked generation in a discrete VQ space, whereas ARMS incrementally denoises continuous causal latents. Adapted baselines can retain stronger isolated-subsequence semantics, while ARMS mainly improves boundary continuity.
• Versus MotionStreamer and causal motion diffusion: ARMS inherits causal compression and segment-based generation ideas. Its additional contribution centers on unified solo-interaction representation and mode-aware relational modeling, rather than inventing causal motion generation from scratch.
• Versus HINT and Interact2Ar: These methods already investigate autoregressive interaction, and achieve better FID in some comparisons. ARMS is especially interesting for its configuration-switching interface and transition evaluation, not because prior interaction work universally lacks long-horizon generation.

Rating

These are the note author's subjective assessments on a five-point scale, not metrics reported by the paper.

• Novelty: 4/5. Asymmetric states and mode gating form a clear contribution for solo-social switching, while the underlying causal diffusion components have precedents.
• Experimental Thoroughness: 3/5. Two interaction benchmarks, streaming evaluation, and multiple ablations are useful, but isolated gating effects, latency, and very-long-horizon stability remain untested.
• Writing Quality: 3/5. The methodological thread is clear, but claims of superiority on all metrics conflict with the tables, and some essential inference details are deferred to supplementary material.
• Value: 4/5. A useful modeling direction for virtual characters that repeatedly enter and leave social behavior, but not yet a complete environment-aware control system.