STANCE: Controllable Video Generation for Structured Dynamics via Sparse-To-dense ANChored Encoding

Conference: ECCV 2026
Paper: ECCV Official
Code: [Pending Release]
Area: Video Generation
Keywords: Video Generation, Physical Interaction, Controllable Diffusion, Dense RoPE, Auxiliary Geometry Stream

TL;DR

STANCE resolves motion cue dilution and physical incoherence in controllable video diffusion by lifting sparse user hints into pixel-aligned 2.5D Instance Cues, anchoring motion tokens via Dense RoPE, and coupling RGB with a shared-backbone auxiliary structural stream as a geometry witness.

Background & Motivation

Recent advancements in Diffusion Transformers (DiTs) have drastically pushed the frontier of generative video realism and long-horizon spatio-temporal modeling. Nevertheless, synthesizing structured physical interactions—such as rigid-body collisions, momentum transfer, and contact timing—remains notoriously unreliable. Video backbones often suffer from glaring physical violations: objects hover before impact, bounce off each other without physical contact, clip through geometry, or suffer severe appearance drift across frames. These failures underscore a systemic modeling bias where architectures heavily prioritize high-frequency texture reconstruction over faithful physical motion dynamics.

Tracing this breakdown to the control pipeline reveals two pragmatic bottlenecks. First, user-provided motion hints (e.g., sparse 2D trajectory arrows or click handles) are spatially sparse; upon patchification and early transformer attention, they collapse to an insignificant fraction of total tokens, rapidly diluting guidance and leaving thin or small objects unguided. Second, standard 2D control signals inherently lack depth and mass awareness. They cannot resolve projective ambiguities caused by simultaneous camera and object motion, nor can they reverse collision outcomes based on mass disparity. While trajectory-conditioned frameworks offer motion stability, they demand laborious frame-by-frame path scripts that severely undermine user friendliness.

To maintain generative versatility while enforcing rigorous physical plausibility, STANCE tackles sparse signal densification and latent spatial preservation. Core idea: lift sparse 2D directional arrows and physical scalars into pixel-aligned 2.5D Instance Cues, anchor salient motion tokens with first-frame Dense RoPE, and train a shared-weight auxiliary structural stream inside the DiT to serve as a geometry witness against physical drift.

Method

Overall Architecture

STANCE builds directly upon state-of-the-art Diffusion Transformers (such as CogVideoX). Inputs include a text prompt, an initial keyframe, instance masks, coarse 2D directional arrows, a depth delta \(\Delta z\), and a scalar mass tag \(m\). The framework executes across three sequential stages: first, sparse user controls are converted into dense, camera-relative 2.5D Instance Cues; second, active non-zero sites are sampled and assigned first-frame Rotary Position Embeddings via Dense RoPE, preserving spatial addressability under fixed token budgets; third, the DiT sequence dimension is expanded to host an auxiliary structural stream (depth or segmentation) alongside RGB tokens, regularizing geometry via unified spatio-temporal self-attention.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: First Frame + Instance Masks<br/>+ Velocity Arrows / Depth Delta / Mass"] --> B["2.5D Instance Cue Construction<br/>Flow Averaging & Scalar Mask Broadcasting"]
    B --> C["Dense RoPE Spatial Anchoring<br/>Active-Site Selection & Fixed-Budget RoPE"]
    C --> D["Sequence-Level Dual-Stream DiT<br/>Joint Self-Attention of RGB & Auxiliary Stream"]
    D --> E["Output: Physically Coherent Video<br/>+ Geometry Witness Verification"]

Key Designs

1. 2.5D Instance Cue Construction: Resolving Depth Ambiguity and Injecting Physics

Traditional 2D trajectory drag handles operate strictly on image planes, resulting in severe motion direction ambiguity whenever the camera pans, zooms, or tilts, while completely ignoring mass disparity during impacts. STANCE constructs dense, pixel-aligned 2.5D Instance Cues to bridge this gap. During training, the per-instance average motion vector is extracted from ground-truth optical flow \(\mathbf{O}\) over mask \(\Omega^{(i)}\): $$ \bar{\mathbf{v}}^{(i)} = \frac{1}{|\Omega^{(i)}|} \sum_{(x,y)\in\Omega^{(i)}} \mathbf{O}(x,y) $$ Simultaneously, a per-instance depth displacement scalar \(\Delta z_i = \text{mean}_{p\in M_i} (D_{t+1}(p) - D_t(p))\) is computed from monocular depth predictions. During inference, coarse 2D velocity arrows drawn by users are rasterized across the instance mask, while user-specified \(\Delta z\) (positive into the screen, negative toward the camera) and mass scalar \(m\) are broadcast uniformly over the same mask. The resulting four-channel tensor \((u, v, \Delta z, m)\) is strictly pixel-aligned with image features, allowing the DiT to explicitly interpret out-of-plane trajectories and mass-driven momentum transfer.

2. Dense RoPE Spatial Anchoring: Preventing Token Dilution and Preserving Spatial Addressability

Standard patchification downsamples control feature maps, rendering local motion masks extremely sparse within the latent token grid. For small or elongated objects, informative control tokens are easily drowned by surrounding zeros in shallow attention layers. Dense RoPE circumvents this dilution by isolating active non-zero grid indices \(\Omega = \{i \in \{1,\dots,L\} : \mathbf{M}_i = 1\}\). Under a pre-allocated token budget \(N\), indices are either uniformly subsampled (if \(|\Omega| > N\)) or tiled and truncated (if \(|\Omega| \le N\)) to construct an index set \(\mathcal{J}\) of exact size \(N\).

Each selected motion token feature \(\mathbf{x}^{\text{Cue}}_j\) (\(j \in \mathcal{J}\)) is modulated with first-frame Rotary Position Embeddings (RoPE) and scaled by a learnable gain \(g_k\): $$ \mathbf{q}^{\text{Cue}}_j = \mathbf{W}_q \left( \mathbf{x}^{\text{Cue}}_j \mathbf{p}^{\text{Cue}}_j \right), \quad \tilde{\mathbf{k}}^{\text{Cue}}_j = g_k \mathbf{W}_k \left( \mathbf{x}^{\text{Cue}}_j \mathbf{p}^{\text{Cue}}_j \right) $$ These compact, high-salience motion tokens are concatenated directly into the sequence. Because each motion token retains a definitive spatial anchor from the initial frame, spatio-temporal attention blocks in subsequent frames reliably attend to exact origin coordinates, preventing trajectory drift, hover delays, and identity swapping.

3. Sequence-Level Dual-Stream DiT: Geometry Witness Regularization

Optimizing solely for RGB denoising often lures the model into prioritizing photorealistic surface textures over rigorous structural boundaries. STANCE duplicates video tokens along the sequence dimension to handle two modality-aligned streams of equal length \(L\), without altering the backbone parameter count or attention heads: $$ \mathbf{X}^{1:2L}{\text{video}} = \left[ \mathbf{X}^{1:L}}} ; \, \mathbf{X}^{1:L{\text{aux}} \right] $$ RGB and auxiliary tokens share identical spatial-temporal rotary positional codes at identical coordinates \(m\). A zero-initialized learnable domain vector \(\mathbf{d}_{\text{aux}}\) is added exclusively to auxiliary queries and keys for modality identification. Full self-attention operates jointly across text, RGB, and auxiliary tokens: $$ \mathbf{Z} \in {\mathbf{Q}, \mathbf{K}, \mathbf{V}} = \left[ \mathbf{W}_z(\mathbf{x}}}) ; \, \mathbf{fz(\mathbf{X}^{1:L}^*}}) ; \, \mathbf{fz(\mathbf{X}^{1:L}) \right] $$ Training is supervised via joint diffusion objectives: $$ \mathcal{L} = \mathbb{E}}{t,\epsilon} \left[ |\hat{\epsilon} - \epsilon|^2 \right] $$ Acting as a strict "geometry witness," the auxiliary structural stream directly penalizes geometric distortion, compelling the attention layers to preserve object boundaries and physical contacts during RGB synthesis.}} - \epsilon|^2 + \lambda_{\text{aux}} |\hat{\epsilon}_{\text{aux}

Key Experimental Results

Main Results

Evaluation is conducted on a held-out test split of 200 clips (100 synthetic primitive collisions and 100 photorealistic GSO composite scenes) generated via Kubric. The primary metric is Physics IQ (aggregating Spatial IoU, Spatio-temporal IoU, Weighted Spatial IoU, and Motion MSE on a 0–100 scale, higher is better), alongside FVD (lower is better).

Method	Model Size	Physics IQ (↑)	FVD (↓)
SG-I2V	-	15.42	113.54
Drag-Anything	-	24.86	92.78
MoFA-Video	-	29.71	98.30
MotionPro	-	31.58	74.27
VLIPP	-	36.40	57.90
Wan-Move	14B	47.50	45.78
STANCE-2B (Ours)	2B	45.90	54.97
STANCE-5B (Ours)	5B	47.62	50.74

Ablation Study

Ablation on Control Signals and Auxiliary Supervision:

Config	Physics IQ (Regular) ↑	Physics IQ (Small) ↑	FVD (Regular) ↓	FVD (Small) ↓	Note
text-conditioned	24.08	—	97.40	—	Text only, no spatial motion cues
+ 2D-Map	43.72	31.92	56.20	58.59	Naive low-res 2D map injection
+ Dense RoPE (RGB only)	46.89	41.83	54.63	56.32	Token re-sampling & first-frame RoPE
+ Segmentation stream	47.96	45.12	53.09	53.35	Sharp boundary regularization
+ Depth stream (Full)	49.03	45.63	50.39	51.32	Continuous 2.5D geometry witness

Disentangling Dense RoPE Components:

Active Re-sampling (Re-samp.)	First-Frame RoPE (FF-RoPE)	Physics IQ (↑)	FVD (↓)	Note
\(\times\)	\(\times\)	31.92	66.20	Standard patchification; cue dilution
\(\checkmark\)	\(\times\)	37.45	58.18	Recovers token count but lacks persistent coordinates
\(\times\)	\(\checkmark\)	35.21	54.74	Positional anchoring without token density recovery
\(\checkmark\)	\(\checkmark\)	37.83	54.63	Additive gain in spatial fidelity and motion accuracy

Key Findings

• Crucial Gains on Small Objects: On the challenging Small-object split, standard 2D map conditioning experiences a severe drop in Physics IQ from 43.72 down to 31.92 (-27.0%), whereas Dense RoPE lifts performance back to 41.83. This confirms that token dilution was the primary driver of control failure on thin/tiny objects.
• Structural Regularization Without Added Parameters: Introducing the auxiliary depth stream enhances Regular Physics IQ from 46.89 to 49.03 while dropping FVD from 54.63 to 50.39. Depth signals supply continuous geometric continuity that prevents premature bouncing and floating.
• Competitive with 14B Trajectory Prior: At 5B parameters, STANCE-5B achieves 47.62 in Physics IQ, surpassing the 14B Wan-Move model (47.50), proving that DiT generative dynamics can self-rollout accurate physical trajectories from initial cues alone.

Highlights & Insights

• Dense RoPE's Token Densification: By coupling non-zero active site re-sampling with first-frame rotary position embeddings, STANCE elegantly solves the token dilution pathology inherent to transformer visual tokenizers.
• Zero-Parameter Auxiliary Supervision: Doubling tokens along the sequence dimension and decoding a secondary structural stream provides powerful geometric constraints without ballooning model parameters or altering DiT architectures.
• Physical Controllability via Broadcasting: Incorporating depth deltas \(\Delta z\) and scalar mass \(m\) as broadcasted channels establishes an intuitive interface where users can seamlessly manipulate out-of-plane motions and collision momentum outcomes.

Limitations & Future Work

• Strictly Rigid-Body Focus: The current formulation assumes rigid objects and struggles to model non-rigid deformation, tearing, or fluid dynamics.
• Sensitivity to Extreme Arrow Noise: While resilient against small mask segment perturbations, heavy inconsistencies or physical contradictions in user-drawn arrows force the model to compromise with its prior, damping the generated motion.
• Future Directions: Extending the formulation to soft-body physics, articulated robotics manipulation, and contact force estimation.

Related Work & Insights

• vs VLIPP: VLIPP relies on multimodal linguistic prompts to guide physical priors, lacking spatial precision and causing pre-impact bounces; STANCE provides explicit pixel-aligned 2.5D cues that enforce exact contact onsets.
• vs MoFA-Video & MotionPro: Prior flow-adaptation approaches suffer identity drift and mask leakage under long horizons; STANCE's auxiliary geometry stream locks object identity and boundary continuity.
• vs Wan-Move (Trajectory Baselines): Trajectory controllers demand dense, full-path supervision; STANCE operates on sparse initial conditions (who moves, where, how fast, with what mass) and relies on DiT rollouts for realistic physics.

Rating

• Novelty: ⭐⭐⭐⭐☆ Elegant combination of Dense RoPE and sequence-level auxiliary geometric streams for physics-steered video generation.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across 200k training clips, dedicated small-object splits, multi-facet ablations, user studies, and real-world tabletop experiments.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear motivation, well-articulated technical formulation, and comprehensive visual analysis.
• Value: ⭐⭐⭐⭐☆ Highly relevant for interactive video authoring, physics-aligned world models, and robotic simulation environments.

Related Papers

• [ECCV 2026] Controllable Egocentric Video Generation via Occlusion-Aware Sparse 3D Hand Joints
• [ECCV 2026] EcoVideo: Entropy-Orchestrated Video Generation Paradigm in Cloud-Edge Dynamics
• [ICLR 2026] NewtonGen: Physics-consistent and Controllable Text-to-Video Generation via Neural Newtonian Dynamics
• [CVPR 2026] Unified Camera Positional Encoding for Controlled Video Generation
• [CVPR 2025] InterDyn: Controllable Interactive Dynamics with Video Diffusion Models