Skip to content

SwitchCraft: Training-Free Multi-Event Video Generation with Attention Controls

Conference: CVPR 2026 arXiv: 2602.23956 Code: Available (coming soon) Area: Video Generation Keywords: Multi-event video generation, attention control, training-free framework, diffusion models, temporal alignment

TL;DR

This paper proposes SwitchCraft, a training-free multi-event video generation framework that achieves clear temporal transitions and scene consistency without modifying model weights, via Event-Aligned Query Steering (EAQS) to align frame-level attention to corresponding event prompts, and Auto-Balance Strength Solver (ABSS) to adaptively balance guidance strength.

Background & Motivation

State-of-the-art text-to-video (T2V) diffusion models (e.g., Wan 2.1) excel at single-event video generation but struggle with prompts describing multiple temporally ordered events. The root cause lies in the uniform injection of a shared text representation into all frames via cross-attention, causing the model to treat the entire description as a holistic context rather than an ordered event sequence. This leads to event aliasing, blurred transitions, or event omission.

Existing approaches exhibit two categories of limitations:

Training/fine-tuning methods (e.g., MinT): require densely annotated temporal data, incur high computational cost, and generalize poorly.

Stitching methods (e.g., MEVG, LongLive): generate and fuse segments sequentially, lacking global context; each segment cannot anticipate subsequent events, resulting in discontinuous transitions and temporal drift.

The core insight of SwitchCraft is that uniform prompt injection ignores the correspondence between events and frames, motivating a mechanism that directs each frame's attention precisely toward its associated event description.

Method

Overall Architecture

SwitchCraft builds upon a pretrained T2V diffusion Transformer (Wan 2.1 14B) and achieves multi-event control solely by modifying query vectors in cross-attention at inference time. The overall pipeline:

  1. An LLM extracts anchor phrases (discriminative keywords) for each event from the multi-event prompt.
  2. Events are assigned to contiguous frame intervals according to user-specified temporal weight ratios.
  3. Within each interval, EAQS modifies the queries to enhance responses to the target event and suppress responses to competing events.
  4. ABSS automatically computes enhancement/suppression strengths to avoid over- or under-guidance.

Key design choice: query editing is applied only during the first 20 of 50 denoising steps and the first 20 of 40 Transformer blocks, since early steps and shallow blocks establish scene layout and large-scale motion, while later steps refine texture and appearance.

Key Designs

1. Event-Aligned Query Steering (EAQS)

Function: Within the temporal window of each event, modifies frame query vectors to amplify their projection onto the target event key subspace and suppress projection onto competing event key subspaces in attention space.

Anchor extraction: An LLM (e.g., ChatGPT) extracts discriminative anchor phrases for each event from the multi-event prompt—e.g., setting descriptors ("sunny desert," "icy cave") for scene transitions, or action phrases ("walking forward," "reading a book") for action transitions. Anchor phrases are mapped to token index sets of the backbone tokenizer.

Temporal window assignment: Users specify relative duration weights per event, which are mapped to \(F'\) latent frames:

\[N_i \approx F' \cdot \frac{w_i}{\sum_{j=1}^{A} w_j}\]

Remainders after rounding are assigned to events with the largest fractional parts, ensuring full frame coverage. Each event \(i\) is mapped to a contiguous half-open frame index interval.

Key subspace projector construction: For a given cross-attention head, let \(K \in \mathbb{R}^{L_k \times D}\) denote the text key matrix and \(Q^* \in \mathbb{R}^{R \times D}\) the queries in the current event interval. Target event keys (\(K_{\text{tgt}}\)) and competing event keys (\(K_{\text{oth}}\)) are extracted from \(K\) by anchor indices, and a regularized right-projection operator is constructed:

\[P_{\text{tgt}} = K_{\text{tgt}}^\top (K_{\text{tgt}} K_{\text{tgt}}^\top + \epsilon I)^{-1} K_{\text{tgt}}\]

\(P_{\text{oth}}\) is constructed analogously. These projectors map queries onto the target event subspace \(\mathcal{T}\) and the competing event subspace \(\mathcal{O}\), respectively.

Query update: Queries are modulated with non-negative strengths \(\alpha\) and \(\beta\):

\[Q^* \leftarrow Q^* + \alpha \cdot Q^* P_{\text{tgt}} - \beta \cdot Q^* P_{\text{oth}}\]

The first term amplifies the query component along the target event subspace (increasing dot products with target keys); the second term suppresses the component along the competing event subspace (reducing event leakage). Row normalization is applied after editing to stabilize attention magnitudes.

Design motivation: Directly modifying post-softmax attention weights disrupts the pretrained model structure; modifying keys/values affects all frames globally. Modifying only queries is a frame-local operation that steers attention while preserving the model's learned priors. Operating in query space rather than weight space avoids abrupt discontinuities at event boundaries.

2. Auto-Balance Strength Solver (ABSS)

Function: Formalizes the selection of enhancement/suppression strengths \(\alpha\) and \(\beta\) as a convex optimization problem, solved automatically at each denoising step, eliminating manual hyperparameter tuning.

SVD direction compression: Direct token-level alignment score comparison is high-dimensional and sensitive to token count. ABSS applies SVD to the normalized key rows of each event to extract principal directions \(k_{\text{tgt}} \in \mathbb{R}^D\) and \(k_{\text{oth},j} \in \mathbb{R}^D\), compressing each event to a single representative direction for improved robustness.

Margin deficit computation: Alignment scores for query rows are computed as:

\[S_{\text{tgt}} = Q^* k_{\text{tgt}}, \quad S_{\text{oth}} = Q^* k_{\text{oth}}\]

The strongest competitor is \(S_{\text{oth}}^{\max} = \max_j S_{\text{oth},j}\), and the margin deficit is defined as:

\[d = S_{\text{oth}}^{\max} - S_{\text{tgt}} + \varepsilon\]

When \(d > 0\), the competing event dominates the current frame interval and guidance is required.

Convex optimization: Let \(x = [\alpha, \beta]^\top\), \(C = [S_{\text{tgt}} \; S_{\text{oth}}^{\max}]\), and construct the resistance matrix:

\[M = \text{diag}(\|S_{\text{tgt}}\|_2^2, \|S_{\text{oth}}^{\max}\|_2^2)\]

The objective is:

\[\min_{x \geq 0} \frac{1}{2} x^\top M x + \frac{1}{2} \|\max(0, d - Cx)\|_2^2\]

The diagonal entries of \(M\) measure the sensitivity of each direction to the margin, implementing scale-aware damping. The closed-form solution is:

\[(M + C^\top C) x = C^\top d, \quad x \leftarrow \max(x, 0)\]

When \(d \leq 0\) (target event already dominant), the optimal solution is \(x = 0\) and no editing is applied.

Design motivation: Excessively large \(\alpha\) causes appearance distortion and motion instability, while excessively small \(\alpha\) fails to overcome the model's global mixing bias. ABSS automatically determines strength by analyzing the current query-key alignment margin, completely eliminating manual hyperparameter tuning and operating robustly across diverse prompts and scenes.

3. Staged Execution Strategy: Early Guidance + Late Free Generation

Function: EAQS and ABSS are applied only during the first 20/50 denoising steps and the first 20/40 DiT blocks; the original model generates freely thereafter.

Design motivation: Diffusion Transformers exhibit hierarchical organization in both time and depth—early steps and shallow blocks establish scene layout and large-scale motion, while later steps and deeper blocks refine texture, identity, and appearance details. Guiding only in the early phase is sufficient to fix the temporal position of each event; the original model subsequently fills in high-frequency details, maximizing the effect-to-side-effect ratio.

Loss & Training

SwitchCraft is a fully training-free framework that modifies no model weights and requires no additional datasets or fine-tuning. All operations are performed at inference time: EAQS query editing and ABSS convex optimization are executed online within each denoising forward pass. The backbone uses Wan 2.1's original velocity prediction training objective; SwitchCraft introduces no loss functions of its own. Inference uses the UniPC sampler with 50 denoising steps and a guidance scale of 5.0.

Key Experimental Results

Main Results

Experiments use the Wan 2.1 T2V 14B backbone, generating 832×480 videos of 81 frames (5 seconds) on a single A100 GPU. Evaluation covers 60 multi-event prompts (2–4 events) spanning action transitions and scene transitions.

Method CLIP-T CLIP-F Visual Quality T2V Align. Physical Consist. Motion Smooth. Subject Consist. Background Consist. Aesthetic Imaging
MEVG 0.244 0.915 2.13 2.33 1.73 0.953 0.701 0.841 0.346 0.525
DiTCtrl 0.246 0.959 3.20 3.27 2.93 0.981 0.764 0.876 0.511 0.702
LongLive 0.252 0.984 4.27 3.13 3.97 0.984 0.898 0.908 0.627 0.725
Wan 2.1 0.256 0.980 4.30 3.47 4.12 0.987 0.947 0.924 0.645 0.738
Stitch 0.257 0.963 3.73 3.67 3.80 0.983 0.926 0.910 0.608 0.711
Ours 0.275 0.980 4.33 4.30 4.13 0.989 0.945 0.921 0.648 0.741

SwitchCraft achieves substantial gains in text alignment (CLIP-T +7.4%, T2V alignment +24%) while maintaining or exceeding backbone-level visual quality and temporal smoothness. CLIP-F does not reach the top because this metric rewards high inter-frame similarity; pose changes at event transitions naturally lower the score.

Ablation Study

Variant CLIP-T CLIP-F Visual Quality T2V Align. Physical Consist. Motion Smooth.
Full model 0.275 0.980 4.33 4.30 4.13 0.989
Random strength 0.253 0.974 4.15 3.62 3.98 0.987
Fixed strength=1 0.264 0.967 3.97 3.75 3.95 0.985
w/o SVD 0.255 0.978 4.30 3.67 4.08 0.988
Enhance only 0.262 0.980 4.35 3.78 4.13 0.989
Suppress only 0.261 0.978 4.28 3.73 4.05 0.986

Human evaluation (29 users, 5-point scale):

Method No Omission No Leakage Transition Smooth. Visual Quality
MEVG 1.41 1.38 1.38 1.28
DiTCtrl 1.66 1.48 1.48 1.59
LongLive 2.07 2.72 2.97 3.52
MinT 4.31 3.69 3.76 3.83
Wan 2.1 3.17 3.38 3.79 3.93
Stitch 2.62 2.07 2.14 2.45
Ours 4.21 4.04 3.93 4.24

Key Findings

  1. ABSS is critical: Random strength causes event omission/delay (T2V alignment only 3.62); fixed strength=1 causes over-guidance and appearance degradation (visual quality drops to 3.97); adaptive solving by ABSS substantially outperforms both.
  2. Both enhancement and suppression are necessary: Enhancement alone fails to isolate intervals when competing events are strong (subsequent events disappear); suppression alone cannot actively drive queries toward the target event (dominant actions persistently bleed in).
  3. SVD compression is effective: Removing SVD reduces event separation, with CLIP-T dropping from 0.275 to 0.255.
  4. Inference overhead is manageable: Runtime increases from 15.2 to 17.6 minutes for 2 events (+16%) and to 22.3 minutes for 4 events (+47%), with the overhead attributable mainly to SVD and convex optimization in ABSS.
  5. Creative occlusion transitions: SwitchCraft can produce creative transition effects by inserting an intermediate occlusion segment, with the occluder assigned a well-defined temporal window within a single diffusion trajectory.

Highlights & Insights

  1. Elegance of query-only editing: Keys and values are shared across all frames, so modifying them has global effects; queries are frame-local, enabling precise local attention steering without disrupting the global information flow.
  2. Subspace perspective of the projection operator: Event alignment is formulated as a subspace projection problem, with regularized pseudoinverse ensuring numerical stability and clear geometric intuition.
  3. Closed-form convex optimization in ABSS: A \(2 \times 2\) linear system with non-negative projection incurs minimal computational overhead yet achieves significant effect, completely eliminating manual hyperparameter tuning.
  4. Exploitation of the hierarchical generation structure of diffusion models: Early steps/shallow blocks establish layout; late steps/deep blocks refine details—intervention is applied only at the critical stage.
  5. Conceptual connection to Attend-and-Excite: The approach extends the scope of attention manipulation from the spatial dimension to the temporal dimension.

Limitations & Future Work

  1. Constrained by backbone capability: Complex motions the underlying model cannot generate (e.g., jumping jacks) degrade to approximations even with SwitchCraft.
  2. Absence of spatial constraints: In multi-subject scenes, the method cannot bind specific events to specific subjects' spatial locations, potentially causing action confusion across subjects.
  3. Assumes linear temporal structure: Parallel or complex nonlinear narrative structures are not supported.
  4. Inference overhead scales linearly with event count: Four events incur approximately 47% additional inference time relative to the baseline.

Rating

Dimension Score Note
Novelty ★★★★☆ First training-free multi-event video generation method based on query subspace projection
Technical Depth ★★★★☆ EAQS projection design and ABSS convex optimization are theoretically grounded
Experimental Thoroughness ★★★★☆ 6 baselines, 5 ablation variants, automatic metrics + human evaluation
Practicality ★★★★★ Training-free, generalizes to DiT architectures, manageable overhead
Writing Quality ★★★★☆ Clear structure, complete mathematical derivations, intuitive figures