Skip to content

Event-Driven Video Generation

Conference: ECCV 2026
arXiv: 2603.13402
Code: Not publicly available; Project Page: https://evd-project-website.pages.dev
Area: Video Generation
Keywords: Text-to-Video, Diffusion Transformer, Event Gating, Interactive Dynamics, Flow Matching

TL;DR

EVD adds a lightweight token-level event head to pretrained video DiTs. It constrains the latent updates to occur only in actively interacting regions during training and inference using event gating. This reduces dynamic errors in video generation, such as objects moving prematurely, lack of contact, support relation jumps, and post-event drift.

Background & Motivation

Background: Text-to-video models have progressed rapidly in realism, resolution, and duration. DiTs, flow matching, temporal autoencoders, and efficient samplers have significantly improved single-frame quality. Many new benchmarks have also begun to evaluate appearance and dynamics separately, as visually appealing frames do not guarantee physically plausible interactions.

Limitations of Prior Work: Current models frequently suffer from causal inconsistencies in events rather than texture blur. Examples include plates moving before being touched, books stacking without placing actions, chairs continuing to drift after being pulled, or a completed state being generated without aligning the target platform. Such errors may be inconspicuous in single frames but are glaringly obvious within the causal chain of video events.

Key Challenge: Standard frame-first denoising/flow sampling updates all latent regions at every step, even when the prompt indicates that only localized contacts or state transitions should occur. Models lack an explicit "is there an event happening here and now" signal, meaning global latent updates easily lead to ghost motion, pre-contact motion, or post-event drift.

Goal: Without changing the backbone or solver, and without increasing the number of sampling steps, to add a small event branch to pretrained video DiTs. This notifies the generation process which tokens are undergoing a prompt-relevant event, thereby localizing and temporalizing the update field.

Key Insight: The authors model a video as a persistent latent state accompanied by a series of discrete interaction events. The state should not change everywhere at every step; instead, directional changes should occur only near the event regions when they are active. Once an event concludes, updates should decay to stabilize the outcome.

Core Idea: Token features from the final layer of the DiT are used to predict event activity. After spatial smoothing, soft gating, hysteresis, and an early-step schedule, this activity is applied to directly gate the post-CFG direction field. During training, realization, consistency, and ordering losses are employed to bind the event signals to latent state changes.

Method

Overall Architecture

EVD retains the latent flow matching interface of video DiTs: clean video \(x\) is encoded into latent \(z_1\) via a temporal autoencoder, and interpolated from noise \(z_0\) to obtain \(z_t=t z_1+(1-t)z_0\), while the base DiT predicts the velocity/direction field. The modifications in EVD are highly focused: an event head is attached to the DiT token features to output token-wise activity, which is then converted into a gate and multiplied with the backbone's update field. During sampling, CFG is performed as usual, except that the final direction field passed to the solver becomes an event-gated direction field.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Text prompt + noised latent video"] --> B["Pretrained video DiT<br/>Predicts base direction field"]
    B --> C["Token-level event head<br/>Predicts interaction activity"]
    C --> D["Soft gate + hysteresis<br/>Stabilizes event boundaries"]
    D --> E["Scheduled event gating<br/>Strong constraint early / relaxed late"]
    B --> F["CFG direction field"]
    E --> G["Event-gated solver update"]
    F --> G
    G --> H["More interaction-consistent video"]

Key Designs

1. Token-level event head: Providing DiTs with an explicit channel for "interaction occurring here"

EVD utilizes the token features \(s_t^{(L)}\) from the final layer of the DiT, appending a lightweight event head \(\pi_\psi\) to predict token-aligned event fields \(e_t\). In the main approach, the number of event channels is \(C_e=1\), with the first channel passed through a sigmoid function to obtain the activity probability \(a_t \in [0,1]^N\). To preserve the pretrained model, the event head is initialized to near-zero, which ensures it barely alters the base DiT's behavior in the early phases of fine-tuning.

This event activity is not an additionally annotated symbolic event graph, but rather a localized interaction signal learned from the model's own latent dynamics. Its function is to answer a question that the sampler originally cannot address: should the spatial-temporal patch corresponding to the current token undergo state changes? If the answer is close to 0, the update should be suppressed; if it is close to 1, the interactive region updates normally.

2. Soft activation + hysteresis: Preventing event gating from becoming a flickering hard-threshold mask

Directly thresholding event activity leads to two issues: boundary flickering and intensity discontinuity. EVD applies a \(3 \times 3\) spatial smoothing to the activity and then establishes a hysteresis band using distinct on/off thresholds. The soft gate is defined as:

\[g_{soft}=ฯƒ(ฮฒ(a_t-(ฯ„_{on}+ฯ„_{off})/2)),\]

where \(\beta\) controls boundary sharpness. The binary hysteresis gate opens when the activity exceeds \(\tau_{on}\) and closes when it falls below \(\tau_{off}\), retaining its state from the previous sampling step inside the intermediate interval. The final gate is \(g_t = g_{soft} \cdot g_{bin}\). This effectively adds a memory mechanism to the event boundary: once an event starts, it will not be abruptly shut down by a few noisy tokens; once it ends, it will not cause unnecessary movement in low-activity regions.

3. Event-gated direction field: Modifying only the update passed to the solver without altering the solver

Post-CFG, the base direction field can be written as \(v^{cfg}(z_t,y,t)\). EVD does not substitute the Euler/Heun/DPM-style solver, nor does it increase the number of forward passes. Instead, it patchifies the direction field, multiplies it by the gate token-wise, and then unpatchifies it back to the latent shape:

\[\tilde v_t = UnTok(g_t ยท Tok(v^{cfg}(z_t,y,t))).\]

Intuitively, EVD does not tell the model what objects to generate, but rather constrains where state changes are permitted. This precisely addresses the four types of failures: premature motion in Contact Stability is suppressed by low-activity tokens; post-event drift in State Persistence is weakened once the gate closes; and Support Relations and Spatial Accuracy rely on the alignment of active event regions with target contact areas.

4. Scheduled gating: Controlling dynamic structures early, refining textures late

The early sampling stage determines coarse motions and interaction structures, whereas the later stage focuses more on textures and localized details. Applying a strong gating mechanism throughout might keep dynamics metrics high, but it suppresses late-stage refinement, leading to a drop in human preference. Thus, EVD implements a schedule: full gating is applied when \(t \leq t^*\), after which the gate linearly anneals to all ones, progressively reverting to the base model in later stages.

This detail is highly important in the ablation study: the constant gate retains a high VBench Dynamics score, but human preference drops significantly, proving that "event constraint" should not become a blunt motion mask. EVD's schedule essentially divides labor temporally between coarse event formation and late-stage visual richness.

5. Event-grounded losses: Truly aligning event signals with latent state changes

Simply adding an inference-time mask is insufficient, because the event head must learn the actual geometric variations in the model's latents. EVD retains the base Flow Matching loss and adds three types of event losses. The Event Realization Loss penalizes large updates in low-activity regions, forcing the model to either activate the event or cease modifications; the Event Consistency Loss constrains the update directions within active event regions to remain consistent across similar diffusion times, reducing interaction jitter; and the Ordering / Termination Loss suppresses update energy before the event starts and after it concludes using \(\tau_{on}, \tau_{off}\).

The overall objective can be summarized as:

\[L=L_{base}+w(t)(ฮป_{real}L_{real}+ฮป_{cons}L_{cons}+ฮป_{order}L_{order}).\]

where \(w(t)\) emphasizes early diffusion/flow times since coarse dynamics are established early. In the paper, default settings are \(t^*_{loss}=0.60\), \(\kappa=6\), with \(\lambda_{real}=0.12\), \(\lambda_{cons}=0.08\), and \(\lambda_{order}=0.03\).

A Complete Example

Given the prompt "a plate is placed onto a dining table", ordinary DiT might slide the plate before the hand makes contact, or cause it to continue drifting after being set down. In early-stage EVD sampling, the event head predicts higher activity at the contact areas of the hand, plate, and dining table, allowing these tokens to be updated by the gate. Regions like the background, distant tabletop, and the already stabilized plate show lower activity, suppressing their direction fields. After the event concludes, the hysteresis gate closes, and the scheduled gate gradually relaxes to accommodate texture refinement, leading to a video that elegantly exhibits the causal sequence of "contact occurrence \(\rightarrow\) plate landing on table \(\rightarrow\) state stabilization".

Loss & Training

The main experiments execute additive modifications on DiT-4B and DiT-30B without altering the transformer blocks. All methods generate 128 frames at \(24\) fps with a base latent/decoder resolution of \(256 \times 256\) on EVD-Bench (upsampled to 720p for visualization), using the same solver, \(K=50\) steps, a CFG scale of \(w_{cfg}=4.0\), and fixed seeds. Only the first sample is evaluated. The event gating hyperparameters are set to \(\beta=12.0\), \(\tau_{on}=0.62\), \(\tau_{off}=0.38\), \(t^*=0.60\), and the event dropout is set to \(0.25\).

Key Experimental Results

Main Results

EVD-Bench contains 150 short-interaction prompts, covering four types of failures: State Persistence, Spatial Accuracy, Support Relations, and Contact Stability. The results report both VBench Appearance / Dynamics metrics and human 2AFC preferences.

Model Text Faithfulness Pref. Quality Pref. Dynamics Pref. VBench Appearance VBench Dynamics
DiT-4B - - - 75.4 78.9
DiT-4B + EVD 88.9 91.3 96.4 76.2 94.8
DiT-30B - - - 73.8 88.7
DiT-30B + EVD 89.7 92.4 97.1 78.1 95.7
Wan / Hunyuan Comparison - - EVD wins 68.4 / 65.7 - 95.7 vs 91.6 / 90.8

Ablation Study

The ablation study on DiT-4B shows that training losses, inference gating, and scheduling are all indispensable.

Config EVD Text Pref. EVD Quality Pref. EVD Dynamics Pref. VBench App. VBench Dyn. Explanation
Full EVD 88.9 91.3 96.4 76.2 94.8 Full training + gating + schedule
w/o event realization 65.2 69.4 78.8 75.9 91.2 Resurgence of missing/pre-contact issues
w/o event consistency 61.3 65.0 74.2 76.0 92.1 More jitter in interaction regions, unstable states
Training-only, no gating 63.0 66.8 77.1 76.1 90.5 Training only without gating, limited gain
Inference-only, no event losses 70.5 74.9 86.0 75.6 84.0 Inference-only masking is insufficient to align latent dynamics
No schedule, const gate 1.0 55.7 58.9 60.5 75.7 94.1 High auto-eval dynamics, but poor human preference

Key Findings

  • Very low overhead: DiT-4B adds only 6.5M parameters (0.16%), and DiT-30B adds 12.0M parameters (0.04%). Inference overhead is approximately \(1.02\times\). Under CFG, it still requires only 2 DiT evaluations per step with \(K=50\).
  • EVD primarily improves dynamics instead of trading off appearance quality. DiT-4B's Appearance rises from 75.4 to 76.2, while Dynamics surges from 78.9 to 94.8.
  • Motion mask baseline is insufficient: The appendix shows that a purely motion-magnitude-based gate easily confuses camera motion with contact motion, and cannot guarantee that results are generated via visible interactions. The advantage of EVD stems from its learned event representation.

Highlights & Insights

  • Attributing video generation failures to spatial-temporal control of the update field: Many dynamic errors occur not because the model cannot draw, but because "regions that should remain static are changing" during sampling. This perspective is highly insightful.
  • Strong compatibility: EVD does not modify the solver, decoder, NFE, or CFG structure, altering only the direction field. Hence, it is theoretically easy to plug into other DiT/flow video generators.
  • Honest scheduling ablation: While a constant strong gate improves automated dynamics metrics, it degrades human preference. This shows that the authors did not treat the gating as a magic mask and instead explicitly modeled the side-effects on late refinement.
  • Diagnosable event signals: As shown in Fig. 5, the activity and gate values concentrate on areas of placing, contacting, or material transferring, providing a visualization window to debug video dynamics.

Limitations & Future Work

  • EVD relies on the event head's ability to localize interactions at the given latent resolution; in scenarios with minor contacts, occlusions, multi-person/multi-object configurations, and crowded environments, events can be under-localized.
  • EVD-Bench consists of 150 short, atomic, single-event prompts, which work well for testing contact, support, and state retention, but do not cover long storylines, multi-event dependencies, or complex camera cinematography.
  • Training still requires massive resources for DiT fine-tuning; in the appendix, DiT-30B+EVD employs 256 GH200-level GPUs, posing a high entry barrier for ordinary academic labs.
  • The event activity is currently represented by a single channel, making it difficult to distinguish event phases like contact onset, state transition, and post-event stabilization. This could be extended in the future to multi-channel phase or object-centric event representations.
  • The correlation between human preference and VBench Dynamics still warrants caution; the constant gate baseline demonstrates that automated metrics might reward overly restrictive constraints that do not necessarily correspond to more natural videos.
  • vs VideoJAM / motion steering: These methods tend to focus on motion quality or inference steering, whereas EVD's core mechanism explicitly aligns prompt-relevant events with latent updates.
  • vs GEST / symbolic event control: GEST uses event graphs or formal specifications to control videos, whereas EVD does not require the user to provide symbolic event graphs, learning event activity internally within the DiT.
  • vs standard diffusion acceleration / masking: EVD is not designed to reduce computational cost but rather to align updates with causal chronology. Its gate is a semantic event gate, not simple motion-magnitude pruning.
  • Insights: Future video generation models could establish state, event, and object contact as first-class citizens in samplers. One might even let a planner first predict an event timeline prior to visual realization by the DiT.

Rating

  • Novelty: โ˜…โ˜…โ˜…โ˜…โ˜† The concept of event gating is intuitive and direct. Its real contribution lies in mapping this idea onto DiT latent updates, training losses, and scheduling.
  • Experimental Thoroughness: โ˜…โ˜…โ˜…โ˜…โ˜† Anchored by the dedicated EVD-Bench, human evaluations, automated metrics, overhead analyses, and ablation studies; coverage of long videos and complex multi-events is still limited.
  • Writing Quality: โ˜…โ˜…โ˜…โ˜…โ˜† The methodology is presented clearly, with a detailed appendix. Nonetheless, the human preference scores in the main tables require careful interpretation.
  • Value: โ˜…โ˜…โ˜…โ˜…โ˜† Significant practical value for resolving interaction dynamics in video generation, especially for prompts involving contact, support, and state retention.