InterEdit: Navigating Text-Guided 3D Dyadic Human Motion Editing¶

Conference: ECCV 2026
arXiv: 2603.13082
Code: https://github.com/YNG916/InterEdit
Area: Human Understanding
Keywords: Dyadic Motion Editing, Text-guided, Conditional Diffusion Model, Interaction Modeling, DCT Frequency Domain Alignment

TL;DR¶

InterEdit proposes a new task, Text-guided 3D Dyadic Motion Editing (TMME), builds the first large-scale dyadic motion editing dataset InterEdit3D (comprising 5,161 source-target-text triplets), and designs the InterEdit method based on conditional diffusion models. By utilizing semantic-aware plan token alignment to capture high-level editing intentions and interaction-aware frequency token alignment to constrain interaction rhythm using DCT frequency band energy, InterEdit significantly outperforms four baseline methods in editing faithfulness and motion realism.

Background & Motivation¶

Text-guided 3D human motion editing has made notable progress in single-person scenarios. Methods like MotionFix and MotionLab can selectively modify a single person's actions based on textual instructions while preserving the unmodified parts. However, a large number of human behaviors in the real world involve dyadic or even multi-person interactions—such as handshaking, dancing, martial arts sparring, and passing a ball. The significance of these interactive movements comes not only from individual motions but also from the precise spatial-temporal coupling between two individuals: who moves first and who moves second, how much distance is maintained between the two, whether waving is synchronous or alternating, and whether the timing of contact is accurate. Editing such interactive motions is far more difficult than editing single-person motions. A tiny change to one person could disrupt the synchronization rhythm, making the actions look like two independent individuals acting on their own rather than truly interacting. This constraint of "maintaining overall coordination while changing a small part" is particularly strict in interactive scenarios.

The core of the difficulty lies in the lack of support from both data and methodologies. On the data level, existing motion datasets either provide only single-person motion (HumanML3D, KIT-ML) or only dyadic generation data (InterHuman, Inter-X), lacking the triplet structure of "source motion before editing $\rightarrow$ target motion after editing + editing instructions". On the methodological level, existing single-person motion editing methods are unaware of the interactive structure, resulting in poor performance when directly concatenating the features of two people. Meanwhile, although dyadic motion generation methods can model interactive structures, they lack the editing capability to "modify designated parts while keeping the rest". Without the source motion acting as an anchor, the generation process tends to drift away from the original action framework, even leading to global drifting.

This paper advances from both the data and methodological aspects simultaneously. In terms of data, the InterEdit3D dataset is proposed—built based on InterHuman using a semi-automatic motion retrieval and human annotation pipeline, with the goal of finding motion pairs where "the base action of one participant is similar, but the interaction structure is different", followed by annotators writing editing instructions. In terms of methodology, under the conditional diffusion model framework, two complementary token alignment mechanisms are introduced: semantic-aware plan tokens learn the high-level semantic embedding of the target motion from a frozen teacher model, telling the model "what to modify it into"; interaction-aware frequency tokens decompose dyadic interaction signals into the frequency domain via DCT to constrain frequency band energy, telling the model "how to maintain the interaction rhythm". These two sets of tokens guide the editing process from two orthogonal dimensions: semantic content and temporal rhythm. Core Idea: Decouple dyadic interactive motion-editing into content decisions at the semantic level and rhythm preservation at the frequency-domain level, modeling these two dimensions within the conditional diffusion model using learnable plan tokens and frequency tokens, supplemented by synchronous classifier-free guidance to achieve precise editing.

Method¶

InterEdit models dyadic motion editing as a conditional diffusion generation process. Given a source dyadic motion sequence $x^s$ and an editing text instruction $y$, the model generates the edited target motion sequence $\hat{x}_0$, requiring that the modified parts match the instruction while preserving the unmodified contents of the source motion and maintaining the spatial-temporal coupling of the dyadic interaction.

Overall Architecture¶

The overall architecture of InterEdit is shown in the diagram below, which is divided into several key stages:

(1) Conditional Encoding: The source motion sequence is encoded by a learnable Transformer encoder (with CLS query tokens) to extract the source conditional embedding $c_{src}$. The text instructions are encoded by a frozen CLIP ViT-L/14 encoder to extract the text conditional embedding $c_{text}$. Both embeddings are projected linearly and added together with the timestep embedding to synthesize the conditional embedding $e_t$, which is injected into each layer of the denoiser via AdaLN modulation.

(2) Token Sequence Construction: The denoiser is a 5-layer Transformer parameterized with Start_X—directly predicting the clean motion $x_0$ instead of noise. The input token sequence consists of three parts: symmetrically interleaved dyadic motion tokens (arranged alternately by both characters' frame-level motion features, concatenating both forward and backward character orders), 16 learnable plan tokens (for capturing high-level editing semantics), and 6 frequency control tokens (derived from the DCT frequency band energy of the current noisy motion to constrain the interaction rhythm).

(3) Diffusion Denoising and Target Reconstruction: The three groups of tokens interact through the joint self-attention of the Transformer. During the denoising process, the motion tokens simultaneously perceive the high-level editing intentions provided by the plan tokens and the interaction rhythm constraints provided by the frequency tokens, ultimately outputting the edited target motion. Inference adopts DDIM sampling (50 steps) and Synchronous Classifier-Free Guidance (SCFG) to enhance condition response.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Source Dyadic Motion Sequence"] --> B["Source Motion Encoder<br/>(Learnable Transformer)"]
    C["Editing Text Instruction"] --> D["CLIP Text Encoder<br/>(Frozen)"]
    B --> E["Conditional Embedding Synthesis<br/>AdaLN Modulation"]
    D --> E
    E --> F["Conditional Diffusion Denoiser<br/>5-Layer Transformer"]
    F --> G["Symmetrically Interleaved Motion Tokens<br/>A-B / B-A Dyadic Joint Modeling"]
    F --> H["Plan Tokens (16)<br/>Target Semantic Contrastive Alignment"]
    F --> I["Frequency Tokens (6)<br/>DCT Band Energy Regression"]
    G --> J["Joint Self-Attention<br/>Mutual Interaction of Three Token Types"]
    H --> J
    I --> J
    J --> K["DDIM Sampling (50 steps)<br/>SCFG Guidance"]
    K --> L["Edited Target Dyadic Motion"]

Key Designs¶

1. Semantic-Aware Plan Token Alignment: Teaching the Model What to Modify through Contrastive Learning

For dyadic motion editing, the core issue is that the model must accurately determine "which part of the source motion to change and what to change it into", while preserving the contents that should not be changed. Simply concatenating the source motion and textual instructions as conditions makes it difficult for the model to map "which motion segment corresponds to which change in the instruction".

InterEdit appends 16 learnable plan tokens to the token sequence of the diffusion denoiser. These tokens are not static; instead, they learn to represent the high-level semantics of the target motion through a contrastive alignment signal during training. Specifically, the paper uses a frozen TMR motion teacher encoder (trained on InterHuman via contrastive learning) to extract a compact semantic embedding $z_{tgt} = f_T(x_0) \in \mathbb{R}^C$ from the ground-truth target motion $x_0$. At the 3rd Transformer block of the denoiser, the 16 plan tokens are mapped to the same semantic space via a linear projection, and InfoNCE loss is employed to constrain each token's projection to be semantically similar to $z_{tgt}$ and dissimilar to other target embeddings in the batch:

\[ \mathcal{L}_{\text{plan}} = \frac{1}{N_M}\sum_{k=1}^{N_M}\left[-\log\frac{\exp(\tilde{z}^{(k)\top}\tilde{z}_{\text{tgt}} / \tau)}{\sum_n \exp(\tilde{z}^{(k)\top}\tilde{z}_{\text{tgt}}^{(n)} / \tau)}\right] \]

This discriminative constraint forces the plan tokens to capture "what makes this target motion unique compared to other motions" rather than just "memorizing" the average features of the target motion—which is exactly what is needed for editing intent representation. Experiments show that using plan tokens alone improves g2t R@1 from 12.54 to 14.52, and the InfoNCE loss outperforms Cosine similarity and MSE.

2. Interaction-Aware Frequency Token Alignment: Locking the Interaction Rhythm from a Frequency-Domain Perspective

The "realism" of dyadic interactions largely depends on subtle temporal coordination—whether two people wave synchronously or alternately, whether the timing of contact is accurate, and whether one person retreats when the other approaches. These rhythmic features are difficult to model explicitly in the original motion coordinate space: a minute temporal shift does not affect the single-frame spatial coordinates, yet completely breaks the sense of interactive synchronization.

InterEdit departs from the frequency-domain perspective and proposes interaction-aware frequency token alignment. For a dyadic motion sequence $x = (x^A, x^B)$, the paper constructs two interaction signals: the average signal $z_S = (x^A + x^B)/2$ captures synchronous components, and the difference signal $z_D = x^A - x^B$ captures relative/adversarial components. Discrete Cosine Transform (DCT) is applied to these two signals along the temporal axis, pooling the DCT coefficients into three frequency bands (low, medium, high) to obtain 6 frequency band energy descriptors (3 bands × 2 signals). These descriptors are projected into 6 frequency control tokens, which are appended to the token sequence for joint self-attention.

During training, the frequency tokens predict the DCT band energy distribution of the target motion via a weighted regression loss, where the weight of high-frequency components is intentionally reduced (0.25) to decrease sensitivity to noise. The elegance of this design lies in: low-frequency components correspond to large-scale interaction coordination (such as the overall rhythm of one punching while the other retreating), and high-frequency components correspond to fine contact details (such as hand synchronization during a high-five). Permitting the model to explicitly align these attributes in the frequency domain is far more straightforward than indirectly maintaining the interaction rhythm solely via spatial-domain losses. Moreover, randomly dropping frequency tokens (drop rate = 0.04) during training serves as a regularization to prevent the model from over-relying on frequency guidance at the expense of generation diversity.

3. Synchronous Classifier-Free Guidance: Custom Condition Injection Strategy for Editing

Standard Classifier-Free Guidance (CFG) is widely used in text-conditioned generation models, but for editing tasks dependent on two conditions, "text instruction + source motion", independently dropping either condition may lead to guidance confusion—the text condition provides editing intent, whereas the source condition demands keeping the original state, leading to mutual cancellation during independent guidance.

InterEdit proposes Synchronous Classifier-Free Guidance (SCFG): during training, both conditions are dropped simultaneously with a probability of $p_{scfg}=0.1$ to obtain an entirely unconditional denoising branch. During inference, conditional and unconditional predictions are interpolated as follows: $$\hat{x}_0 = \gamma \cdot D_\theta(x_t, t; c_{\text{text}}, c_{\text{src}}) + (1-\gamma) \cdot D_\theta(x_t, t; 0, 0)$$ Synchronous dropping avoids information entanglement between the text instruction and the source motion, yielding a cleaner guidance direction. Experiments compared the "two-branch SCFG" (the proposed scheme) with the "three-branch SCFG" (introducing an additional source-only branch). While both show comparable performance, the two-branch scheme is more efficient by omitting one forward propagation during inference.

Loss & Training¶

The training loss of InterEdit consists of three hierarchical levels. Motion Loss $\mathcal{L}_{\text{motion}}$ includes diffusion reconstruction loss (Start_X parameterization, MSE between predicted $x_0$ and ground truth), individual geometric loss (velocity smoothness $\mathcal{L}_{\text{vel}}$, foot contact $\mathcal{L}_{\text{foot}}$, bone length $\mathcal{L}_{\text{BL}}$), and dyadic interaction loss (masked distance map $\mathcal{L}_{\text{DM}}$, relative orientation $\mathcal{L}_{\text{RO}}$). Plan Token Alignment Loss $\mathcal{L}_{\text{plan}}$ constraints the plan tokens to align with the teacher target embedding via InfoNCE ($\lambda_p = 0.03$). Frequency Token Alignment Loss $\mathcal{L}_{\text{freq}}$ performs weighted MSE regression on the DCT band energy of the target motion ($\lambda_f = 0.01$). The total loss is $\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{motion}} + \lambda_p \mathcal{L}_{\text{plan}} + \lambda_f \mathcal{L}_{\text{freq}}$.

Training uses the AdamW optimizer, with a peak learning rate of $10^{-4}$ paired with cosine decay + a 10-epoch warmup, for a total of 1500 epochs with a batch size of 32. Both the CLIP text encoder and the TMR teacher encoder are frozen. The model was trained on 8 RTX Pro 6000 Blackwell GPUs for approximately 4 hours and 35 minutes, with 85 million trainable parameters. Inference uses DDIM sampling with 50 steps, with an SCFG guidance scale of $\gamma = 3.5$.

Key Experimental Results¶

Main Results¶

InterEdit is compared against four adapted baseline methods on the self-built TMME benchmark: single-person motion editing methods (MotionFix, MotionLab) and dyadic motion generation methods (InterGen, TIMotion). All baselines were retrained on InterEdit3D. The core metrics include generation-to-target retrieval (g2t, higher is better, reflecting instruction following), generation-to-source retrieval (g2s, higher is better, reflecting source preservation), and FID (lower is better, reflecting motion realism).

Method	FID ↓	g2s R@1 ↑	g2s R@2 ↑	g2s R@3 ↑	g2t R@1 ↑	g2t R@2 ↑	g2t R@3 ↑
MotionFix	2.109	3.10	7.20	9.10	11.60	17.80	21.10
MotionLab	0.428	12.40	18.90	23.00	18.50	25.60	30.50
InterGen	0.624	9.52	15.03	18.91	18.93	26.74	31.64
TIMotion	0.445	12.54	18.38	22.33	24.97	33.77	40.68
InterEdit	0.371	17.08	24.04	29.32	30.82	40.84	47.65

InterEdit comprehensively outperforms the strongest baseline TIMotion across all metrics: g2t R@1 improves by 5.85 percentage points (+23%), g2s R@1 improves by 4.54 percentage points (+36%), and FID is reduced by 16.7%. It is noteworthy that dyadic generation methods (InterGen, TIMotion) outperform single-person editing methods on g2t, indicating that interaction-aware modeling is vital for dyadic editing. However, they lack explicit constraints for "preserving source content," which results in significantly lower g2s compared to InterEdit.

In an additional human preference evaluation (20 prompts, 10 participants), InterEdit achieved overwhelming advantages over TIMotion in overall preference (75.5% win rate), instruction following (78.5% win rate), source preservation (71.0% win rate), and interaction realism (81.0% win rate).

Ablation Study¶

The core ablation study evaluates the independent and joint effects of the two main modules:

Configuration	FID ↓	g2s R@1 ↑	g2t R@1 ↑
Without plan/freq tokens (Baseline)	0.445	12.54	24.97
Plan tokens only	0.367	14.52	28.72
Freq tokens only	0.380	14.24	28.75
plan + freq tokens (Ours/Full)	0.371	17.08	30.82

Using only plan tokens significantly improves semantic editing (g2t +3.75), while using only frequency tokens also improves retrieval while maintaining a competitive FID. The combination of both yields the largest gain in g2s (+4.54), validating the complementary design of "semantics guiding what to change, and frequency constraining how to change". The optimal drop rate for frequency tokens during random dropping is 0.04; too low leads to over-reliance, while too high weakens the alignment signal.

Furthermore, InterEdit also outperforms the state-of-the-art TIMotion on the InterHuman dyadic generation benchmark (R-Precision R@1 of 0.523 vs. 0.491), showing that the interaction modeling capabilities brought by semantic plan tokens and frequency tokens are generalizable and not limited to editing scenarios.

Key Findings¶

Strong complementarity of the two modules: Plan tokens primarily improve g2t (semantic editing faithfulness), while frequency tokens primarily improve g2s (source preservation). Combining them yields the largest leap in g2s (+4.54), indicating that frequency regularization effectively prevents interaction drift during editing.
InfoNCE > Cosine > MSE: Contractive alignment for plan tokens significantly outperforms regression-based alignment, indicating that discriminative semantic representations are better suited for editing intent encoding.
Two-branch SCFG is sufficient: Three-branch SCFG (adding an extra source-only conditional branch) does not bring significant improvement, whereas two-branch is more concise and efficient.
Insensitive to hyperparameters: Performance remains stable under reasonable variations of $\lambda_p$ and $\lambda_f$, though over-tuning slightly degrades FID.

Highlights & Insights¶

Dyadic to Frequency-Domain Perspective Shift: Transforming the dyadic interaction temporal coordination problem into DCT band energy alignment is an elegant dimensionality reduction design. Average/difference signals capture synchronous and adversarial components respectively, and three frequency bands correspond to coarse-to-fine timescales, which is physically intuitive and mathematically differentiable.
Contrastive Alignment vs. Regression Alignment: Plan tokens use InfoNCE, while frequency tokens use MSE—the two alignment approaches have different emphases: the former encourages discriminative semantic representation, while the latter constraints quantitative energy distribution, fitting perfectly with the complementary goals of "what to change" and "how to change".
Transferability from Editing to Generation: InterEdit's leading performance on a standard dyadic motion generation benchmark proves that the interaction representations learned via semantic and frequency token alignment are universal and transferable, rather than restricted to editing tasks.
Dataset Construction Strategy: The "motion retrieval + human annotation" pipeline of InterEdit3D is highly scalable—mining editing triplets from existing generative datasets avoids the massive costs of recording from scratch.

Limitations & Future Work¶

Gesture Ambiguity: The model still suffers from confusion in fine-grained hand gesture semantics. For example, "clapping hands" (self) can be confused with "high-fiving each other", making it difficult for the model to precisely differentiate when only hand interaction modes change between source and target. Future work can introduce explicit modeling at the hand-joint level.
Long-term Spatial Drifting: The model is prone to global drift for one of the characters in long sequences with high-dynamic interaction (e.g., continuous dancing), indicating that maintaining long-range spatial relations remains a challenge. Trajectory-level constraints or explicit spatial anchors could be explored.
Limited Scale: The dataset contains only 5,161 triplets with a vocabulary of 1,754 words. While close to the single-person editing dataset MotionFix (6,730 triplets), it is far from sufficient for multi-person (3+) editing. Collaborative labeling for multi-person interaction is extremely costly; using self-play or physical simulation to generate training data is a key direction.
Dyadic Only: The current method only supports dyadic interaction editing. Scaling up to three or more people requires redesigning token arrangement and interaction modeling strategies.
Limitations of Evaluation Metrics: The g2t/g2s retrieval metrics are proxy measures that cannot fully capture perceptual interaction quality (e.g., whether contact is natural and realistic), which calls for more sophisticated evaluations of interaction realism.

vs. MotionFix/MotionLab: These pioneered text-guided single-person motion editing, but cannot maintain interaction structures when directly concatenating dyadic features. InterEdit proves that interaction-aware modeling is essential for dyadic editing, providing a baseline for future extension to multi-person scenarios.
vs. TIMotion: Share the symmetric interleaved token aggregation architecture, but TIMotion is a pure generative model. InterEdit adapts it into an editing model via plan and frequency tokens. The key is "not just adding source conditions, but designing two sets of alignment mechanisms to guide the editing direction."
vs. InterGen: An evolution from dyadic generation to dyadic editing, with the core increments being paired source-target supervision signals and a conditional diffusion framework. InterGen’s dyadic interaction data (InterHuman) provided the raw materials for the InterEdit3D dataset.
Application of DCT Frequency Analysis in Motion: The frequency band energy pooling proposed in this paper is a highly generalizable trick—any task involving temporal rhythm (dance generation, gesture synchronization, human-robot interaction) can try using DCT decomposition + band constraints to stabilize interaction rhythms.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Proposes the text-guided dyadic motion editing task for the first time, contributing the first large-scale dataset and a complete methodological framework, filling the gap between single-person editing and dyadic interaction editing.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ 4 baseline methods + human evaluation + generalization test + loss function ablation + detailed ablation on module positions/hyperparameters/drop rates. Confidence intervals and statistical rigor are well-established.
Writing Quality: ⭐⭐⭐⭐⭐ Clear motivation, well-organized methodology (narrating the complementary design of semantic vs. frequency dimensions), and structured, comprehensive interpretation of experiments.
Value: ⭐⭐⭐⭐⭐ Provides substantial interaction editing capabilities for animation, gaming, and social robotics; both the dataset and method are open-sourced with strong reproducibility.