What Moves? Localized Motion Representations for Compositional Scene Control

Conference: ECCV 2026
Paper: ECCV Full Paper
Area: Video Generation
Keywords: Localized Motion Representation / Compositional Scene Control / Motion Transfer / Trajectory Prediction / Diffusion Models

TL;DR

Addressing the fundamental tension where localized motion representations require global context to resolve relative dynamics, this paper introduces a promptable, region-specific motion encoder trained via masked future point-trajectory prediction, enabling precise compositional motion transfer and localized action recognition without motion leakage.

Background & Motivation

Motion is a defining characteristic of the physical world, where natural visual scenes are intrinsically compositional and involve multiple distinct entities whose dynamics unfold simultaneously in a shared environment. In a typical driving scene, moving vehicles, pedestrians, and background camera parallax exhibit distinct temporal patterns; similarly, complex human-object interactions consist of coordinated yet separable movements of interacting body parts and handled tools. However, current video generation and motion control paradigms remain bifurcated between global latent motion embeddings and dense, low-level pixel signals. Global motion encoders (such as DisMo) compress the entire scene dynamics into a unified vector, entangling multiple moving objects with camera motion and inevitably causing severe motion leakage and unintended spatial distortions during motion transfer. On the other hand, low-level representations like optical flow fields, dense point trajectories, or trajectory-conditioned diffusion transformers (such as Tora or WanMove) offer spatial locality but lack high-level semantic abstraction; their guidance signals are rigidly tied to the original camera viewpoint, object morphology, and initial scene geometry.

A seemingly straightforward attempt to isolate entity-level motion is to apply global motion encoders to isolated spatial video crops. Nevertheless, motion is fundamentally relational: its physical interpretation inherently depends on a global frame of reference, background camera ego-motion, and mutual interactions between objects. Once an entity is cropped away from its surrounding environment, its localized optical displacements lose their physical grounding, making it impossible to separate object motion from camera movement and introducing fatal ambiguities into the extracted dynamics. Conversely, performing post-hoc feature masking on global video features fails to guide the encoder's internal attention to focus on the target entity during representation learning.

To reconcile the tension between spatial localization and the necessity of global scene context, this work proposes a promptable localized motion representation framework that operates directly on the full video while conditioning on spatial region masks. Core idea: retain full-scene video context throughout the motion encoder while using time-invariant content embeddings as region-specific queries, and train the localized motion tokens by self-supervised masked future point-trajectory prediction to achieve contextually grounded, appearance-invariant, and compositionally transferable motion representations.

Method

Overall Architecture

Given a video \(V = \{x_t\}_{t=1}^T\) of \(T\) frames and \(K\) initial entity masks \(S = \{s_k\}_{k=1}^K\) defined on the first frame, the framework aims to extract persistent, entity-specific motion tokens \(m_k = \{m_{k,t}\}_{t=1}^T\) describing the dynamic evolution of region \(k\). The architecture is composed of three cooperative components: a Content Encoder that captures time-invariant semantic identity and appearance grounding for the queried region; a Motion Encoder that observes the full video and accepts content embeddings as queries to output entity-level motion tokens; and a Masked Tracks Decoder that supervises the motion tokens by deterministically predicting future sparse point trajectories within the queried region.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Full Video V<br/>and Initial Entity Mask sk"] --> B["Context-Preserving Promptable Motion Encoding<br/>Full scene context with content query guidance"]
    A --> C["Time-Invariant Content Grounding & Disentanglement<br/>Static semantic embedding to isolate appearance"]
    C --> B
    B --> D["Masked Sparse Trajectory Prediction Supervision<br/>Predicting future point tracks conditioned on motion tokens"]
    C --> D
    D --> E["Downstream Applications: CogVideoX Compositional Generation<br/>and Localized Action Recognition"]

During generation and transfer, the learned entity motion tokens are injected as conditioned temporal sequences into a pretrained video diffusion model (e.g., CogVideoX-5B). This allows users to selectively extract dynamics from specific subjects across multiple source videos and attach them onto distinct subjects in a target image, unlocking high-precision compositional scene animation.

Key Designs

1. Context-Preserving Promptable Motion Encoding: Extracting Entity Dynamics from Full Video Context

Conventional approaches that crop regional image patches strip away background optical flow, camera ego-motion, and inter-object interactions. In contrast, the motion encoder \(E_\theta\) processes the entire uncropped video sequence \(V\). To isolate the specific entity's motion from the dense multi-object scene, the encoder uses the entity content embeddings \(c = \{c_k\}_{k=1}^K\) produced by the content encoder as cross-attention queries. This promptable design allows the encoder to perceive the global physical coordinate frame and camera trajectory while steering attention exclusively toward the targeted region, effectively preventing motion signals from distractor entities from contaminating the representation.

2. Time-Invariant Content Grounding & Disentanglement: Isolating Appearance from Dynamics

A persistent vulnerability in motion representation learning is appearance entanglement, where dynamic tokens inadvertently memorize static texture, color, or shape attributes rather than pure motion patterns. The content encoder \(C_\phi\) maps a reference frame \(x_i\) and the entity region mask \(s_k\) into a static content token \(c_k = C_\phi(x_i, s_k)\). This token provides an appearance and semantic reference of "what entity is selected" to both the motion encoder and the trajectory decoder. Because static identity is explicitly delegated to \(c_k\), the temporal motion tokens \(m_{k,t}\) are relieved of encoding spatial appearance, thereby forcing them to capture purely abstract, transferable physical dynamics.

3. Masked Sparse Trajectory Prediction Supervision: Self-Supervised Point Tracking as an Inductive Bias

To train the localized motion representations at scale without expensive human action annotations, the framework leverages sparse point trajectory prediction as a self-supervised objective. For each entity \(k\), sparse query points are sampled inside its region \(S_k\) and tracked across time to form ground-truth tracks \(p_{k,1:T}\). At timestep \(t\), the deterministic masked tracks decoder \(D_\psi\) takes the current video frame \(x_t\), current point locations \(p_{k,t}\), content token \(c_k\), and localized motion token \(m_{k,t}\), and is tasked with predicting future track coordinates \(p_{k,t+1:t+L}\) over the next \(L\) frames. Because the reconstruction objective is strictly confined to valid future points within the entity's boundary, the motion tokens are compelled to encode fine-grained localized dynamics while filtering out extraneous background noise.

Loss & Training

The masked tracks decoder is trained using a deterministic \(\ell_2\) trajectory reconstruction loss evaluated exclusively on valid point positions within the entity region:

\[\mathcal{L}(\theta, \psi) = \mathbb{E}_{V, k, t} \left\| D_\psi(x_t, p_{k,t}, c_k, m_{k,t}) - p_{k,t+1:t+L} \right\|_2^2\]

The encoder and decoder are jointly trained using the AdamW optimizer for 600k iterations with a batch size of 128 on open-world videos from OpenVid-1M alongside an internally collected video corpus. For downstream generative tasks, the localized motion tokens are integrated into a frozen CogVideoX-5B backbone, enabling zero-shot or lightweight compositional motion transfer.

Key Experimental Results

Main Results

In the Composable Motion Transfer benchmark, methods extract dynamics from separate source subjects and apply them simultaneously to distinct designated entities in multi-subject target initial frames. Baselines include state-of-the-art low-level trajectory guidance models (ATI, WanMove, Tora) and semantic motion transfer approaches. Metrics evaluate Temporal Consistency, In-Region Motion Fidelity, Out-of-Region Motion Leakage (lower is better), and overall Motion Selectivity (fidelity minus leakage, higher is better).

Model	Generative Backbone	#Params	Temporal Consistency ↑	In-Region Motion Fidelity ↑	Out-of-Region Motion Leakage ↓	Motion Selectivity ↑
ATI [1]	Wan 2.1	14B	0.9634	0.7148	0.3857	0.3291
WanMove [7]	Wan 2.1	14B	0.9792	0.7059	0.4110	0.2949
Tora [59]	CogVideoX	5B	0.9766	0.6520	0.3904	0.2616
Ours	CogVideoX	5B	0.9831	0.6718	0.2604	0.4114

In the Human Preference Study, participants evaluated synthesized videos across three criteria: overall Realism, Motion Transfer Quality and Locality, and the absence of visual artifacts (Fewer Artifacts):

Model	Generative Backbone	#Params	Realism (%) ↑	Transfer Quality (%) ↑	Fewer Artifacts (%) ↑
ATI [1]	Wan 2.1	14B	1.96	4.90	5.23
WanMove [7]	Wan 2.1	14B	14.05	14.38	14.38
Tora [59]	CogVideoX	5B	13.73	17.32	15.69
DisMo [35]	CogVideoX	5B	21.24	26.80	20.26
Ours	CogVideoX	5B	49.02	36.60	44.44

Ablation Study

On the A2D localized action classification benchmark, ablations evaluate the necessity of full-scene context versus localized masking. Supplying the model with cropped RGB videos (\(\text{Ours}_{\text{rgb}}\)) or eliminating localized querying via a full-frame mask (\(\text{Ours}_{\text{global}}\)) severely degrades performance. When evaluated in the global classification regime (providing a full-frame mask without architectural changes), the model exhibits competitive or superior performance across standard motion-centric datasets (Top-1 Accuracy via kNN with \(k=20\)):

Setting / Model	SSv2 (%) ↑	Jester (%) ↑	Diving48 (%) ↑	ARID (%) ↑	IARD (%) ↑
DINOv2 [31]	10.3	23.2	9.39	14.5	76.1
VideoMAE [44]	7.1	20.1	-	17.3	73.4
V-JEPA2 [2]	22.2	40.8	11.1	28.0	87.6
SemanticMoments\(_{\text{DINO}}\) [18]	11.4	37.3	9.85	22.0	62.7
SemanticMoments\(_{\text{V-JEPA2}}\) [18]	31.5	52.2	12.7	38.3	92.5
DisMo [35]	24.6	69.8	20.9	55.3	92.0
Ours (Global Query Mode)	26.5	72.8	17.1	55.4	93.0

Furthermore, under controlled appearance transformations on the synthetic split of the SemanticMoments benchmark (viewpoint variation, attribute modification, entity replacement, and artistic stylization), our method achieves 89.6% top-1 retrieval accuracy, matching the specialized global motion model DisMo (90.0%) while substantially outperforming appearance-dominated foundation models such as DINOv2 (44.4%), VideoMAE (79.2%), and V-JEPA2 (74.4%).

Key Findings

• Drastic Reduction in Motion Leakage: In compositional motion transfer, the proposed method reduces unintended motion leakage to non-target subjects from 0.3857–0.4110 down to 0.2604 (over 32% reduction), achieving a top Motion Selectivity score of 0.4114 compared to baselines (0.2616–0.3291).
• Scene Context Disambiguates Localized Dynamics: Cropping input patches (\(\text{Ours}_{\text{rgb}}\)) leads to a drastic drop in localized action classification, demonstrating that motion cannot be resolved in isolation without global camera and scene reference frames. Meanwhile, unprompted global encoding (\(\text{Ours}_{\text{global}}\)) suffers from distractor interference, validating that the union of full-scene context and localized query conditioning is strictly superior.
• Robust Appearance Invariance: By supervising localized motion embeddings via sparse point trajectories and separating static appearance into content embeddings, the learned motion tokens remain highly invariant to dramatic shifts in subject appearance, viewpoint, and style.

Highlights & Insights

• Unified Global Context and Local Querying: By reframing localized motion extraction as a query-conditioned cross-attention operation over full-scene video features, the model neatly sidesteps the false dilemma between losing scene context through cropping and losing spatial specificity through global pooling.
• Self-Supervised Point Trajectory Prediction: Using sparse point tracking as an inductive learning objective forces the representation to prioritize physical kinematics and displacement over superficial RGB appearance, providing a scalable pretraining path without manual action labels.
• Dual Capability Across Local and Global Scales: The promptable representation naturally generalizes to full-frame queries, matching or exceeding specialized global motion encoders on standard video understanding benchmarks.

Limitations & Future Work

• Tracking Drift Under Extreme Mutual Occlusion: In crowded scenes where multiple identical entities undergo frequent mutual occlusions, time-invariant content embeddings may suffer from attention dispersion, potentially misallocating localized dynamics.
• Fine-Grained Complex Non-Rigid Dynamics: Sparse point tracking targets primarily capture macro-level skeletal and rigid-body displacements; modeling high-frequency deformations such as fluid splashes, billowing smoke, or intricate fabric wrinkles may require dense continuous field supervision.
• Future Directions: Extending the localized representation to 3D motion tokens using dynamic neural radiance fields or deformable 3D Gaussians would provide true view-consistent compositional scene manipulation.

Related Work & Insights

• vs. DisMo [35]: DisMo extracts motion as an entangled global latent representation, preventing entity-level targeting and causing cross-entity motion leakage; this paper introduces promptable spatial region conditioning to unlock selective multi-entity extraction and compositional recombination.
• vs. Tora [59] & WanMove [7]: Trajectory-guided diffusion models enforce strict, low-level point-tracking constraints in pixel coordinates, which frequently fail or cause visual shearing when transferring motion across substantial viewpoint changes or divergent entity morphologies. This work extracts higher-level semantic motion tokens that preserve kinematic semantics while generalizing across diverse appearances.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Resolves the long-standing conflict between localized motion extraction and the relational requirement for full-scene context via promptable masked encoding.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation encompassing multi-entity composable generation, motion leakage quantification, user preference studies, localized action recognition, and invariant retrieval.
• Writing Quality: ⭐⭐⭐⭐⭐ Rigorous conceptual framing, clear architectural exposition, and tightly coupled experimental validation.
• Value: ⭐⭐⭐⭐⭐ Provides an extensible, plug-and-play localized motion representation paradigm for controllable video diffusion models and fine-grained video understanding.