InterCMDM: Block-Causal Diffusion for Autoregressive Human Interaction Generation

Conference: ECCV 2026
Paper: ECCV 2026
Area: Human Understanding
Keywords: Two-Person Interaction Generation / Block-Causal Diffusion / Autoregressive Motion Synthesis / Dual-Stream Attention / Multi-Task Attention Masking

TL;DR

InterCMDM introduces a block-causal latent diffusion framework for autoregressive two-person interaction generation via a Dual-Stream Causal DiT, multi-task block attention masking, and latent rollout, achieving state-of-the-art realism and semantic alignment without requiring ground-truth prefix initialization.

Background & Motivation

Text-conditioned synthesis of two-person interactions is foundational for interactive digital avatars, embodied multi-agent coordination, and immersive virtual environments. Unlike single-person motion synthesis, multi-human interactions require concurrently maintaining long-range temporal biomechanical plausibility within each individual while orchestrating microsecond-level spatial orientation, mutual contact timing, and responsive motion reciprocity between partners. Minor asynchronous offsets or joint trajectory deviations quickly compound across the interaction timeline, escalating into severe coordination drift such as interpenetration, phantom physical contacts, unnatural foot sliding, and role desynchronization.

Existing paradigms for human interaction synthesis are limited by fundamental trade-offs. The majority of prevailing diffusion-based interaction models, such as InterGen and TIMotion, rely on bidirectional full-sequence attention across the entire temporal window. By inherently conditioning on future frames, bidirectional denoising destroys temporal causality, preventing online streaming generation and suffering from prohibitive computational scaling and boundary seam artifacts during long-horizon generation. Conversely, existing autoregressive interaction approaches enforce temporal causality but typically iterate frame-by-frame in raw motion space, leading to rapid accumulation of compounding drift across steps. Moreover, several existing autoregressive baselines depend heavily on ground-truth motion prefixes for warm-starting, precluding genuine unconstrained text-driven generation.

This paper approaches the challenge from a structural factorization perspective: human interaction fundamentally emerges as two coupled but distinct temporal causal processes dynamically coordinated through attention mechanisms. Collapsing both actors into a single joint sequence obscures individual kinematic identities, while unconstrained bidirectional denoising forfeits causal streaming capability. Core idea: extend causal diffusion to two-person interaction by introducing a Dual-Stream Causal Diffusion Transformer (DS-Causal-DiT) coupled with multi-task block attention masks, achieving block-parallel denoising and causal autoregressive rollout in latent space for stable, controllable long-horizon interaction generation.

Method

Overall Architecture

The end-to-end pipeline of InterCMDM is built upon three closely coordinated components: a 1D causal temporal motion VAE, the Dual-Stream Causal Diffusion Transformer (DS-Causal-DiT), and a block-wise diffusion denoising objective.

First, continuous motion sequences of two interacting subjects \(x^{(1)}, x^{(2)} \in \mathbb{R}^{T \times D}\) are independently downsampled \(4\times\) along the temporal axis via a shared 1D causal convolutional motion VAE, yielding compact latent token sequences \(z^{(1)}, z^{(2)} \in \mathbb{R}^{T' \times d_z}\) (\(T'=T/4\), where each latent token corresponds to a 4-frame temporal receptive field). The latent sequence is then partitioned into non-overlapping temporal blocks of size \(B\) (representing \(4 \times B\) motion frames). During diffusion denoising, DS-Causal-DiT takes noisy latent blocks with independent per-block diffusion timesteps, maintains two parallel causal streams for Person 1 and Person 2, and integrates inter-person interaction dynamics and text conditioning through unified dual-stream attention. During autoregressive rollout, the model initializes generation from pure Gaussian noise on the first block and autoregressively generates subsequent clean blocks directly in latent space, decoding continuous full-length motions in real time without boundary drift.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Text Prompt + Initial Dual-Person Latent Noise"] --> B["DS-Causal-DiT & Unified Dual-Stream Attention<br/>Decouple intra-person causal streams & join via concatenated attention"]
    B --> C["Multi-Task Block Attention Masking<br/>Enforce structured causal constraints for simultaneous/reactive/leader/independent"]
    C --> D["Block-Wise Diffusion Objective & Latent Rollout<br/>Intra-block parallel denoising + cross-block causal autoregression"]
    D --> E["Output: Causal VAE Decoding into Long-Horizon Interaction Motion"]

Key Designs

1. DS-Causal-DiT with Unified Dual-Stream Attention: Decoupled Streams with Joint Coordination

Prior interaction diffusion architectures either concatenate both subjects into a single token stream, blurring person-specific dynamics, or employ disconnected separate self- and cross-attention blocks, ballooning parameter count and inference cost. DS-Causal-DiT addresses this dilemma through a parallel-stream yet unified attention formulation. The backbone stacks \(L\) dual-stream blocks. Within each layer, Person 1 and Person 2 are processed through dedicated linear projection layers for Query, Key, and Value, preserving role-asymmetric behavioral representations. The two token sets are concatenated along the temporal sequence dimension and processed by a unified masked attention operator, followed by independent text cross-attention and feed-forward networks for each stream. This formulation strictly preserves intra-person temporal ordering while enabling dynamic inter-person coordination within a single attention mechanism.

2. Multi-Task Block Attention Masking: Multi-Mode Supervision and Zero-Shot Controllability

Human interaction exhibits multifaceted coordination regimes differing in information availability and temporal causality, ranging from symmetric dancing to reactive dodging and uncoordinated walking. Training a separate model for each pattern impairs sample efficiency and generalizability. InterCMDM introduces multi-task block attention masking, defining four structured attention topologies within the unified dual-stream attention matrix: 1. Simultaneous Mask: Both individuals attend causally to their own history and the partner's current and past blocks in a symmetric fashion; 2. Reactive Mask: Person 2 attends to Person 1 only after Person 1's action block is fully resolved, modeling stimulus-response behaviors such as evading or being struck; 3. Leader–Follower Mask: The follower attends to the leader with an explicit temporal lag, enforcing asymmetric coordination dynamics; 4. Independent Mask: Inter-person cross-attention is disabled, modeling uncoupled concurrent movements.

During training, these masks are randomly sampled according to empirical dataset proportions (60% simultaneous, 20% reactive, 10% leader–follower, and 10% independent), serving as structured interaction-level data augmentation. At inference time, users can steer the interaction regime zero-shot simply by supplying the corresponding attention mask without modifying model weights.

3. Block-Wise Diffusion Objective and Pure Latent Rollout: Intra-Block Parallelism and Long-Horizon Stability

Full-sequence diffusion models cannot perform causal streaming, whereas single-frame autoregressive models accumulate high step-wise compounding drift. InterCMDM generalizes diffusion forcing to temporal blocks in latent space. The latent sequence is partitioned into \(N_B = T'/B\) blocks, where each block \(b\) is assigned an independent diffusion timestep \(k_b \in [0, K]\):

\[z_{k_b}^{(i,b)} = \sqrt{\bar{\alpha}_{k_b}} z_0^{(i,b)} + \sqrt{1 - \bar{\alpha}_{k_b}} \boldsymbol{\epsilon}^{(i,b)}, \quad \boldsymbol{\epsilon}^{(i,b)} \sim \mathcal{N}(0, \mathbf{I})\]

The training loss optimizes joint noise residual prediction across both streams given all temporal blocks up to index \(b\):

\[\mathcal{L}_{\text{train}} = \mathbb{E}_{\mathbf{k}, \boldsymbol{\epsilon}} \left[ \sum_{b=1}^{N_B} \sum_{i=1}^2 \left\| \boldsymbol{\epsilon}^{(i,b)} - \mu_\theta^{(i)} \left( \mathbf{z}_{\mathbf{k},\le b}^{(1)}, \mathbf{z}_{\mathbf{k},\le b}^{(2)}, k_b, c \right) \right\|^2 \right]\]

During autoregressive inference, the model executes pure Latent Rollout: the initial block (\(b=1\)) is denoised from pure Gaussian noise, while each subsequent block (\(b>1\)) is conditioned on previously generated clean latent blocks \(z_0^{(i, <b)}\). Because temporal context is propagated entirely within the compact continuous latent space, this approach completely eliminates the repeated decode-encode cycles (Generate \(\to\) Decode to Pose \(\to\) Prefix Extraction \(\to\) Re-encode \(\to\) Generate) required by traditional frame-space autoregressive frameworks, preventing cumulative quantization distortion and boundary discontinuities.

Loss & Training

The framework follows a decoupled two-stage training scheme: 1. Temporal Motion VAE: Trained independently on 64-frame motion clips with a sliding window of 10 frames. The objective combines geometric pose reconstruction loss and per-person KL divergence: \(\mathcal{L}_{\text{VAE}} = \sum_{i=1}^2 (\mathcal{L}_{\text{rec}}^{(i)} + \beta \mathcal{D}_{\text{KL}}^{(i)})\). Optimized using AdamW with learning rate \(2 \times 10^{-4}\) for 50 epochs (~1 hour on a single A100 GPU). 2. DS-Causal-DiT Diffusion Backbone: Frozen DistilBERT text encoder and pretrained VAE. The diffusion transformer comprises 8 layers with hidden dimension 512, 4 attention heads, and block size \(B=3\) (corresponding to 12 frames). Flow Matching scheduler is adopted, optimized via AdamW (learning rate \(10^{-4}\), batch size 64) for 500 epochs (~8 hours on a single A100 GPU). Inference uses 50 diffusion steps, with classifier-free guidance scales of 3.0 on InterHuman and 1.5 on Inter-X.

Key Experimental Results

Main Results

Evaluations are conducted on InterHuman (22-joint AMASS skeleton, 262-dim features) and Inter-X (56-joint SMPL-X format, 336-dim features with expressive hand/face motions). Among all fully generative methods requiring no ground-truth motion warm-starting, InterCMDM achieves state-of-the-art text-motion alignment, motion realism, and interaction diversity.

Dataset	Method	R-Precision Top-1 ↑	R-Precision Top-3 ↑	FID ↓	MM Dist ↓	Diversity →	MModality ↑
InterHuman	Real Motions	0.452 ± 0.008	0.701 ± 0.008	0.273 ± 0.007	3.755 ± 0.008	7.948 ± 0.064	—
	InterGen (Diffusion)	0.371 ± 0.010	0.624 ± 0.010	5.918 ± 0.079	5.108 ± 0.014	7.387 ± 0.029	2.141 ± 0.063
	InterMask (Masked)	0.449 ± 0.004	0.683 ± 0.004	5.154 ± 0.061	3.790 ± 0.002	7.944 ± 0.033	1.737 ± 0.020
	TIMotion (Diffusion)	0.501 ± 0.005	0.734 ± 0.006	4.702 ± 0.069	3.769 ± 0.021	7.943 ± 0.034	1.005 ± 0.020
	InterCMDM (Ours)	0.529 ± 0.007	0.757 ± 0.005	4.492 ± 0.087	3.765 ± 0.002	8.065 ± 0.034	3.044 ± 0.104
Inter-X	Real Motions	0.429 ± 0.004	0.736 ± 0.003	0.002 ± 0.000	3.536 ± 0.013	9.734 ± 0.078	—
	InterGen (Diffusion)	0.207 ± 0.004	0.429 ± 0.005	5.207 ± 0.216	9.580 ± 0.011	7.788 ± 0.208	3.686 ± 0.052
	InterMask (Masked)	0.403 ± 0.005	0.705 ± 0.005	0.399 ± 0.013	3.705 ± 0.017	9.046 ± 0.073	2.261 ± 0.081
	TIMotion (Diffusion)	0.411 ± 0.005	0.707 ± 0.004	0.261 ± 0.014	3.737 ± 0.015	9.112 ± 0.079	2.475 ± 0.075
	InterCMDM (Ours)	0.523 ± 0.009	0.820 ± 0.007	0.215 ± 0.014	3.154 ± 0.021	9.490 ± 0.120	2.391 ± 0.136

Ablation Study

1. Architecture and Attention Causality Ablations (InterHuman Dataset)

Comparison between bidirectional and causal attention across various block sizes \(B\), evaluating single-stream (Single), hybrid-stream (Hybrid, dual-stream in early layers and single-stream in later layers), and full dual-stream (Dual) backbones:

Attention Type	Stream Design	R-Precision Top-1 ↑	R-Precision Top-3 ↑	FID ↓	MM Dist ↓	Note
Bidirectional	Single	0.535 ± 0.003	0.771 ± 0.002	4.456 ± 0.058	3.758 ± 0.016	Non-causal global attention
Bidirectional	Hybrid	0.538 ± 0.003	0.768 ± 0.002	4.468 ± 0.061	3.760 ± 0.017	Partial dual-stream layers
Bidirectional	Dual	0.549 ± 0.003	0.778 ± 0.002	4.562 ± 0.064	3.754 ± 0.015	Non-causal upper bound
Causal (\(B=3\))	Single	0.484 ± 0.003	0.717 ± 0.003	5.068 ± 0.071	3.782 ± 0.018	Concatenated single-stream causal baseline
Causal (\(B=3\))	Hybrid	0.499 ± 0.003	0.734 ± 0.003	4.577 ± 0.063	3.774 ± 0.017	Half-layer decoupled baseline
Causal (\(B=3\))	Dual (Ours Full)	0.529 ± 0.007	0.757 ± 0.005	4.492 ± 0.087	3.765 ± 0.002	Optimal balance of causality and fidelity
Causal (\(B=1\))	Dual	0.514 ± 0.007	0.735 ± 0.009	4.551 ± 0.098	3.773 ± 0.002	Granular token-level blocks
Causal (\(B=5\))	Dual	0.532 ± 0.007	0.763 ± 0.004	4.545 ± 0.097	3.763 ± 0.002	Higher block latency in streaming

2. Long-Horizon Generation and Transition Boundary Continuity

Evaluated on 64 long-horizon interaction sequences across 32 continuous caption segments on InterHuman:

Model	Rollout Strategy	Segment FID ↓	R-Top1 ↑	Position Discrepancy (Pos Disc.) ↓	Boundary Discontinuity Ratio (B/NB Ratio) ↓	Foot Skating ↓
InterMask	AR Rollout	27.239 ± 0.39	0.320 ± 0.008	0.048 ± 0.002	2.51 ± 0.07	0.172 ± 0.004
InterCMDM	Naive (Independent Stitching)	18.37 ± 0.37	0.423 ± 0.008	1.375 ± 0.053	98.58 ± 1.21	0.260 ± 0.006
InterCMDM	Composition (Decode-then-Encode)	25.758 ± 0.35	0.338 ± 0.008	0.026 ± 0.001	1.91 ± 0.05	0.136 ± 0.003
InterCMDM	Latent Rollout (Ours)	25.104 ± 0.40	0.352 ± 0.007	0.024 ± 0.001	1.66 ± 0.04	0.157 ± 0.004

Key Findings

1. Dual Streams Disentangle Asymmetric Dynamics: Under strict block-causal attention (\(B=3\)), the dual-stream architecture improves R-Precision Top-1 from 0.484 to 0.529 and drops FID from 5.068 to 4.492 compared to the single-stream variant. Furthermore, dedicating separate parameter projections for Person 1 and Person 2 outperforms shared-parameter dual-stream blocks (Top-1 0.506), proving that asymmetric weights are crucial for distinct actor/reactor or leader/follower coordination.
2. Latent Rollout Eliminates Boundary Fractures: While independently generating and stitching segments (Naive) yields low isolated segment FID, its boundary discontinuity ratio explodes to 98.58, producing jarring visual jerks. Latent Rollout compresses the boundary discontinuity ratio to 1.66 (ideal continuity = 1.0) and outperforms repeated decode-encode cycles (Composition) with lower position error (0.024) and superior text alignment (Top-1 0.352).
3. Multi-Task Masks Transfer to Specialized Reactive Tasks: When evaluated on one-way reactive generation (Person 1 given, Person 2 synthesized), the model trained with multi-task masks achieves an FID of 2.121 and Top-1 of 0.484, significantly outperforming models trained exclusively on simultaneous masks (FID 2.585) or reactive masks (FID 2.216), showing that multi-task masking acts as structured dynamic data augmentation.

Highlights & Insights

• Unifying Interaction Regimes into Configurable Masks: Abstracting two-person coordination into attention masks within a unified attention matrix enables multi-task data augmentation during training and seamless, zero-shot behavioral steering during inference without retraining.
• Synergy of Diffusion Forcing and Latent Dynamics: Marrying block-level causal diffusion forcing with a causal 1D temporal VAE grants the distribution fidelity of diffusion models while avoiding single-frame compounding drift and prefix dependence.
• Ultra-Fast Real-Time Streaming: Yielding an initial latency of only 0.662 seconds and streaming iteration latency of 50.3 ms per 12-frame block, the system reaches an inference throughput of ~89 FPS with under 600MB peak GPU memory, highly suitable for real-time interactive game and VR deployment.

Limitations & Future Work

• Restricted to Two-Person Scenarios: The architecture explicitly relies on a dual-stream design; extending it to arbitrary multi-human crowds requires reformulating attention into dynamic graph attention or multi-token routing mechanisms.
• Absence of Explicit Physics and Contact Encoders: The framework does not incorporate explicit rigid-body penetration or contact-force penalties, which can lead to occasional interpenetration in tight grappling or wrestling sequences.
• Coarse Temporal Conditioning: Text conditioning relies on global sentence embeddings from DistilBERT, limiting fine-grained temporal phase control over individual limbs or asynchronous sub-actions. Per-person compositional text guidance presents a promising future direction.

Related Work & Insights

• vs InterGen / TIMotion: These diffusion baselines employ full-sequence bidirectional attention. While full context aids single-clip denoising, it prevents streaming generation and causes boundary seams; InterCMDM enforces block causality, enabling true streaming rollout without quality degradation.
• vs InterMask: InterMask relies on collaborative masked modeling but accumulates sliding and coordination drift over long sequences; InterCMDM's continuous latent space reduces boundary discontinuity by over 34% (1.66 vs 2.51) without ground-truth prompt frames.
• vs HINT / Interact2Ar: These autoregressive models require ground-truth motion prefixes for evaluation and stabilization; InterCMDM denoises the initial block cold from pure Gaussian noise, providing a fully generative framework.

Rating

• Novelty: 4.5/5 (First to introduce block-causal diffusion with dual-stream attention to two-person interaction synthesis)
• Experimental Thoroughness: 4.5/5 (Rigorous evaluation on InterHuman and Inter-X, featuring extensive long-horizon continuity metrics and mask ablations)
• Writing Quality: 4.5/5 (Precise problem formulation, coherent structural narrative, and informative ablations)
• Value: 4.5/5 (Provides a practical, low-latency, and drift-resilient blueprint for streaming digital avatar interactions)