GraphVid: Interactive Graph-Controllable Video Generation

Conference: ECCV 2026
Paper: ECCV Paper
Code: https://plan-lab.github.io/graphvid
Area: Video Generation
Keywords: controllable video generation, interaction scene graph, edge-aware GNN, diffusion transformer, physical dynamics

TL;DR

GraphVid introduces directed interaction scene graphs to interactively guide multi-object dynamics in single-image-to-video generation. By incorporating relation semantics via an edge-aware graph neural network and modulating a frozen video Diffusion Transformer with lightweight LoRA adapters, GraphVid outperforms trajectory-conditioned baselines while requiring only 0.6B trainable parameters and 27K video clips.

Background & Motivation

Controllable Image-to-Video (I2V) generation aims to synthesize temporally coherent video sequences from a single static image while following user-specified dynamic conditions. Prevailing methodologies in this space predominantly focus on trajectory-conditioned motion guidance, such as Wan-Move, Motion-I2V, Tora, and MagicMotion. These paradigms treat dynamics primarily as low-level pixel displacement, requiring users to supply dense point tracks, bounding-box flows, or optical flow fields. In complex scenes featuring multiple entities, drawing precise trajectories for each object scales poorly with scene complexity. Moreover, under occlusion, overlap, or non-rigid deformations, small geometric inconsistencies can easily yield implausible global physical artifacts, such as surface penetrations, unnatural sliding, and desynchronized motion.

While physics-aware alternatives attempt to address these shortcomings by conditioning generation on force vectors, velocity fields, or physics-centric textual categories (e.g., WISA, Force Prompting), they predominantly formulate physical rules as low-level continuous fields or rigid predefined classes. This restricts their capacity to reason about complex, open-domain compositional interactions between multiple co-existing entities. Crucially, existing trajectory-based and physics-based frameworks rely heavily on immense datasets and compute resources: Wan-Move is trained on 2 million videos, Motion-I2V consumes 10 million samples, and both fine-tune multi-billion parameter backbones. Such resource demands constrain their practical utility, interactive flexibility, and generalization.

The core insight of this work is that physical motions in natural scenes are fundamentally relational—arising from reciprocal forces and interactions between entities (such as pushing, pulling, lifting, and supporting) rather than isolated trajectory displacements. By framing user intent as an attributed directed interaction graph over detected entities, the system bypasses tedious trajectory authoring while directly providing causal relational inductive biases to the generative process. The core idea is to represent multi-object video control via directed interaction scene graphs, leverage an edge-aware Graph Isomorphism Network (GINEConv) to infuse relational semantics into message passing, and steer a frozen video DiT backbone via lightweight adapters and attention LoRA layers under a flow-matching objective.

Method

Overall Architecture

GraphVid is built upon a pretrained video Diffusion Transformer (DiT), utilizing LTX-Video as its default backbone while maintaining compatibility with Wan 2.2. The DiT backbone and the 3D VAE encoder-decoder are kept strictly frozen, and only lightweight graph reasoning layers, projection adapters, and attention LoRA modules are updated during training. Given an input conditioning frame \(I_0\), the framework detects visual foreground entities and constructs an attributed directed interaction graph \(\mathcal{G} = (\mathcal{V}, \mathcal{E})\) through user interaction or open-vocabulary relation extraction. Nodes represent scene entities, directed edges capture causal actions (e.g., push, pull, lift, hold, along with coarse direction and magnitude), and self-loops encode unary transformations (e.g., rotate, move). Node and edge features are routed through an Edge-Aware Graph Reasoning module; the resulting interaction-aware node embeddings are projected into transformer conditioning tokens and injected alongside edge text tokens to modulate the frozen DiT via cross-attention and LoRA adaptation under a flow-matching objective.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image I₀ & User Interaction Prompts"] --> B["Scene Graph Construction & Multimodal Embeddings<br/>Entity detection pooling & open-vocabulary interaction edges"]
    B --> C["Edge-Aware Graph Reasoning<br/>GINEConv infusing directed relational edge attributes"]
    C --> D["Graph-to-Video Adapter & Backbone Modulation<br/>Latent projection & attention LoRA injection"]
    D --> E["Generated Interaction-Consistent Video"]

Key Designs

1. Scene Graph Construction & Multimodal Embeddings: Disentangling visual entities and open-vocabulary interaction intents

To overcome the inability of trajectory-based inputs to convey both fine-grained visual identity and causal semantic intent, this design extracts structured multimodal representations directly from the static input frame. A vision-language detector (Qwen3-VL) identifies foreground objects. For each detected entity \(v_i\), three feature modalities are fused: a visual embedding \(f_i^{vis} \in \mathbb{R}^{8192}\) extracted via mean and max pooling over the feature map within bounding box \(b_i\), a text embedding \(f_i^{txt} \in \mathbb{R}^{2560}\) encoding the entity label, and normalized spatial coordinates \(b_i \in \mathbb{R}^4\). These are concatenated to form the unified node vector \(x_i = [f_i^{vis}; f_i^{txt}; b_i] \in \mathbb{R}^{10756}\), jointly capturing appearance, semantic category, and spatial location. Directed interaction edges \(e_{ij}\) accept open-vocabulary user directives (e.g., "woman moves closer to viewer", "hand grips cup"), which are encoded by Qwen3-Embedding into relational attributes \(e_{ij} \in \mathbb{R}^{2560}\). Unary motions are formulated as self-loops \(e_{ii}\), cleanly decomposing the visual scene into an intuitive topological graph.

2. Edge-Aware Graph Reasoning: Injecting directed relational semantics directly into message passing

Standard Graph Convolutional Networks (GCNs) treat edges as binary connectivity indicators, solely aggregating node states across adjacent vertices. In physical dynamic modeling, this assumption is fundamentally flawed: opposing interactions (such as pushing versus pulling, or colliding versus supporting) induce completely different force dynamics and trajectories, which standard GCNs flatten into spatial proximity smoothing. To resolve this, GraphVid adopts an edge-aware Graph Isomorphism Network (GINEConv). Node and edge features are first projected into a shared hidden dimension \(d = 512\): \(h_i^{(0)} = \omega_n(x_i)\), \(a_{ij} = \omega_e(e_{ij})\), where \(\omega_n\) and \(\omega_e\) are learnable MLPs. Information is subsequently propagated across \(L\) GINEConv layers:

\[h_i^{(l+1)} = \text{MLP}^{(l)}\left( h_i^{(l)} + \sum_{j \in \mathcal{N}(i)} \text{ReLU}\left( h_j^{(l)} + a_{ji} \right) \right)\]

By directly incorporating the directed relation attribute \(a_{ji}\) into neighbor aggregation, node updates become explicitly conditioned on the physical interaction type governing the relationship. After \(L\) layers, each node embedding \(h_i \in \mathbb{R}^{512}\) captures both object identity and its anticipated motion propagation within the relational causal network.

3. Graph-to-Video Adapter & Backbone Modulation: Preserving generative priors while steering dynamics with minimal parameters

Training large-scale DiT architectures from scratch requires prohibitive compute and risks compromising pretrained spatio-temporal priors. GraphVid instead adopts a frozen-backbone adaptation strategy. Interaction-aware node embeddings \(h_i\) are projected through an adapter MLP into the DiT latent dimension \(d_l = 4096\), yielding semantic conditioning tokens \(z_i\). To accommodate variable entity counts across scenes, node sequences are padded or truncated to a fixed maximum budget \(N_{max} = 30\). These tokens, combined with edge text tokens, are provided as encoder hidden states to the DiT. Simultaneously, Low-Rank Adaptation (LoRA) modules of rank 128 are integrated into the Query, Key, Value, and Output projection matrices of every transformer block. By freezing the backbone and updating only the LoRA weights, the model learns to modulate self-attention over the graph tokens, requiring only 0.6B trainable parameters across the entire system.

4. Interaction-Centric Dataset Curation: Benchmarking multi-entity compositional dynamics

The scarcity of video datasets paired with structured interaction graphs has historically impeded relational video generation. The authors curate GraphVid-Bench, comprising approximately 27K interaction-centric video clips standardized to 81 frames at 16 fps. Each clip is paired with a directed interaction graph, averaging 7.76 nodes and 4.85 edges. The dataset spans four foundational physical interaction categories: force & manipulation, kinematics & motion, physical contact, and support & position. Structurally, 10,982 samples feature single interactions, 12,641 contain multiple composite interactions (\(N > 1\)), and 3,881 samples are non-interactive (\(N = 0\)) baseline clips capturing natural unprompted motion. This composition provides strong supervisory grounding for learning how scene dynamics shift when novel interaction edges are introduced at inference time.

Loss & Training

The framework is optimized end-to-end using the conditional flow-matching (CFM) objective. Given a target video \(x_0\) encoded into latent space by the pretrained 3D VAE, timesteps \(t \sim p(t)\) are sampled from a logit-normal distribution to form noisy latents \(x_t = (1 - t)x_0 + t x_1\) with Gaussian noise \(x_1 \sim \mathcal{N}(0, \mathbf{I})\). The network optimizes the velocity vector field \(v_\omega(x_t, t, c)\) by minimizing the mean squared error against the true target flow:

\[\mathcal{L}_{CFM}(\omega) = \mathbb{E}_{c \sim \{I_0, \mathcal{G}\}, t \sim p(t), x_0 \sim p_{data}(\cdot|c), x_1 \sim \mathcal{N}(0, \mathbf{I})} \left\| v_\omega(x_t, t, c) - (x_1 - x_0) \right\|_2^2\]

During optimization, the 3D VAE, Qwen3 feature encoders, and DiT backbone parameters remain strictly frozen. Only the GNN layers, adapter MLPs, and attention LoRA matrices are optimized, ensuring efficient and stable convergence from symbolic graphs to continuous generative dynamics.

Key Experimental Results

Main Results

GraphVid is quantitatively evaluated against state-of-the-art trajectory-based and physics-conditioned video generation methods on the interaction-centric subset of MoveBench. Perceptual fidelity is measured by FID and FVD; reconstruction quality is assessed via PSNR and SSIM; and motion accuracy is quantified via End-Point Error (EPE) against ground-truth optical flow.

Method	Train Data	Trainable Params (B)	Inference Time (s) ↓	FID ↓	FVD ↓	PSNR ↑	SSIM ↑	EPE ↓
Wan-Move	2M	14.5	1800	15.56	82.17	17.21	0.61	2.6
Motion-I2V	10M	1.2	790	28.32	159.32	9.87	0.38	3.9
Tora	630K	5.0	1200	24.45	110.47	11.27	0.54	3.3
MagicMotion	23K	1.5	750	26.57	105.12	12.04	0.51	3.2
WISA	80K	1.0	1000	25.98	107.89	15.08	0.53	4.1
FlashMotion	23K	13.0	677	19.02	104.12	14.04	0.56	3.8
GraphVid (Ours)	27K	0.6	200	17.02	99.42	15.98	0.61	2.9

In the multi-object interaction subset of MoveBench, where relational reasoning over multiple entities is essential:

| Method | Train Data | Trainable Params (B) | FID ↓ | FVD ↓ | PSNR ↑ | SSIM ↑ | EPE ↓ | |---|---|---|---|---|---|---| | Wan-Move | 2M | 14.5 | 31.29 | 252.0 | 16.69 | 0.61 | 2.2 | | Tora | 630K | 5.0 | 56.04 | 369.0 | 14.98 | 0.52 | 3.5 | | WISA | 80K | 1.0 | 60.12 | 341.0 | 13.13 | 0.55 | 3.9 | | FlashMotion | 23K | 13.0 | 55.03 | 311.0 | 12.12 | 0.52 | 3.9 | | GraphVid (Ours) | 27K | 0.6 | 49.45 | 291.0 | 14.44 | 0.55 | 3.0 |

Ablation Study

Ablation experiments analyze conditioning modalities, node capacity budgets, backbone generalization, and LoRA rank capacity.

Config / Variant	FID ↓	FVD ↓	PSNR ↑	SSIM ↑	EPE ↓	Note
Text + Image (LTX Baseline)	20.04	109.34	11.23	0.53	3.6	Global prompt struggles to bind motion to specific entities
Graph (Topology-only, no edge text)	17.52	101.32	16.88	0.60	3.0	Spatial localization holds, but ambiguous on push vs. pull
Graph (Full GraphVid conditioning)	17.02	99.42	15.98	0.61	2.9	Full graph topology with edge semantics achieves best fidelity
Max Node Capacity \(N_{max}=10\)	17.25	102.04	15.56	0.60	-	Capacity too small, truncates entities in crowded scenes
Max Node Capacity \(N_{max}=30\)	17.02	99.42	15.98	0.61	-	Optimal structural balance without attention dilution
Max Node Capacity \(N_{max}=256\)	17.97	109.92	13.77	0.57	-	Excess zero-padded tokens inject noise and dilute attention
Backbone Swap: Wan 2.2 (5B)	17.29	100.01	15.70	0.60	3.1	Demonstrates backbone-agnostic transferability
LoRA Rank = 16	18.34	103.18	13.70	0.58	-	Lower capacity limits complex multi-subject adaptation
LoRA Rank = 128	17.02	99.42	15.98	0.61	-	Standard setting offering superior representation capacity

Key Findings

• Relational scene graphs outperform dense trajectory supervision: Compared to Motion-I2V, GraphVid cuts FID by 39.9% (28.32 \(\to\) 17.02) and FVD by 37.6% (159.32 \(\to\) 99.42), while reducing optical flow error from 3.9 to 2.9, despite training on \(370\times\) less data and utilizing half the parameters.
• Edge semantics resolve directional motion ambiguity: Conditioning on graph topology alone improves spatial binding but fails to disambiguate opposing actions (e.g., pushing vs. pulling). Incorporating text-embedded edge attributes eliminates this ambiguity, reducing FVD by nearly 2 points and yielding the lowest optical flow error (EPE 2.9).
• The attention dilution phenomenon: Expanding the maximum node capacity from 30 to 256 degrades FVD from 99.42 to 109.92. Because natural scenes typically contain 5–10 active entities, excessive padding introduces superfluous zero tokens that dilute transformer self-attention over active interactions.
• Inference efficiency: GraphVid achieves an inference latency of 200 seconds per video clip, significantly outpacing Wan-Move (1800s), Tora (1200s), and WISA (1000s) by omitting cumbersome trajectory tracking encoders.

Highlights & Insights

• Shifting from tedious trajectory drawing to semantic graph editing: Rather than requiring users to trace precise pixel paths across frames, GraphVid abstracts multi-object dynamics into intuitive relational tuples, allowing the pretrained generative prior to infer physically realistic motion trajectories.
• Direct message-passing injection of physical relations: By incorporating relational text embeddings directly into GINEConv neighbor aggregation, the model natively reasons about causal force propagation and object-to-object dynamics.
• Backbone-agnostic modularity: The graph reasoning module and adapters interface cleanly with both LTX-Video (2B) and Wan 2.2 (5B), demonstrating that interaction graphs serve as a universal, transferable conditioning modality for modern video DiTs.

Limitations & Future Work

• Author-admitted limitations: The model lacks fine-grained modeling for micro-scale physical dynamics, such as turbulent fluid viscosity, granular matter, and high-frequency fabric tearing.
• Dependency on upstream detection: Graph construction relies on Qwen3-VL; undetected or severely occluded objects fail to form graph nodes, preventing relational control over unsegmented background elements.
• Future directions: Integrating interactive human-in-the-loop graph correction tools, and lifting 2D scene graphs into 3D Gaussian or spatial point-cloud representations to enable joint camera viewpoint and 3D physical manipulation.

Related Work & Insights

• vs. Wan-Move / Motion-I2V (Trajectory-guided video generation): Wan-Move and Motion-I2V guide generation via point trajectories and optical flows, requiring millions of training clips (2M–10M) and large parameter updates. GraphVid replaces dense motion cues with high-level relational scene graphs, achieving competitive quality with only 27K clips and 0.6B trainable parameters.
• vs. WISA (Physics-conditioned video generation): WISA incorporates global physical concepts via text prompts and attribute categories, but lacks explicit graph topology for multi-object interaction. GraphVid explicitly grounds interactions to directed pairs, yielding a 34.5% FID improvement (25.98 \(\to\) 17.02) and superior multi-entity coordination.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Pioneers the use of directed interaction scene graphs as an interactive control modality for image-to-video generation]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive comparisons on MoveBench subsets, multi-object dynamics, backbone transfers, and capacity ablations]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, well-formulated graph equations, and thorough analysis of physical control mechanisms]
• Value: ⭐⭐⭐⭐⭐ [Provides a practical, parameter-efficient paradigm for interactive multi-subject video generation without dense trajectory overhead]