ShotStream: Streaming Multi-Shot Video Generation for Interactive Storytelling¶

Conference: ECCV2026
Paper: ECCV Paper
Area: Video Generation
Keywords: multi-shot generation, interactive storytelling, dual caches, rotary position embeddings, distribution matching distillation

TL;DR¶

ShotStream reformulates multi-shot video generation as next-shot prediction conditioned on historical visuals, combining sparse conditioning, boundary-aware dual caches, and two-stage self-forcing distillation to achieve 15.95 FPS on one H200 and a transition-control score of 0.978.

Background & Motivation¶

Multi-shot storytelling requires more than smooth motion within a shot: identities, clothing, and scenes must remain recognizable after viewpoint changes. Existing bidirectional models generate multiple shots jointly, benefiting from global relationships but usually requiring all prompts upfront. That workflow becomes awkward when users decide the next shot only after viewing the current output; long-sequence attention and multi-step denoising also introduce substantial waiting time.

Causal video models can stream chunks, but primarily target continuous scenes. Cinematic editing permits abrupt visual changes, so a model must remember historical characters without treating previous shots as frames that should be continued directly. Accepting a new prompt alone is insufficient: cached information can carry the old scene's continuation bias into the new shot, while errors in generated history propagate across shots.

The paper therefore places the interaction boundary at the next shot and separates cross-shot memory from continuity within the current shot. Core Idea: first learn next-shot generation from sparse historical context, then turn that capability into low-latency streaming through context-separated causal dual caches and self-forcing training that progressively matches inference.

Method¶

Overall Architecture¶

Inputs comprise a global narrative description, local descriptions of historical shots, and the user's next-shot description; outputs are videos synthesized shot by shot and chunk by chunk within each shot. Training first constructs the Sparse-Context Teacher, introduces Dual Caches with RoPE Separation for the student, and then applies Two-Stage Self-Forcing Distillation. Inference uses only the distilled causal student, without rerunning the bidirectional teacher or requiring prompts for every future shot in advance.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    INPUT["Historical shots and captions"] --> TEACHER["Sparse-Context Teacher"]
    TEACHER --> CACHE["Dual Caches with RoPE Separation"]
    CACHE --> DISTILL["Two-Stage Self-Forcing Distillation"]
    DISTILL --> OUTPUT["Causal student streams chunks"]
    PROMPT["Runtime next-shot description"] --> OUTPUT

Key Designs¶

1. Sparse-Context Teacher: bind visuals to shot-specific text within a limited history budget

The teacher starts from Wan2.1-T2V-1.3B and learns to generate a target shot after reading historical visuals, rather than generating an isolated video from text alone. Past shots may contain hundreds of frames, making full-history retention redundant and expensive, so the model selects sparse context frames with an experimental budget of 6 frames. The budget is divided by the number of historical shots and rounded down to give each shot an equal base allocation; the remainder goes to the most recent shot. This uses the limited budget more effectively than taking only each shot's first frame and gives recent narrative content extra context.

Selected visuals are encoded by the base model's 3D VAE, then patchified separately from the noisy target latents and concatenated along the temporal dimension. Only the target shot receives noise; historical conditioning stays clean, allowing native 3D self-attention to relate history tokens to target tokens without new conditioning layers. Historical visuals do not all receive the target shot's caption: each shot's tokens attend to the global caption and their own local shot caption through cross-attention. Consequently, historical appearances and locations remain bound to their original visual content instead of being reinterpreted through the new shot description.

Training updates only the 3D spatial-temporal self-attention layers inside the DiT, leaving the remaining components frozen. The ablation favors this strategy over full-parameter fine-tuning, but does not establish that every larger backbone should freeze the same components. The teacher still requires approximately 50 denoising steps, making it a capability source rather than the final interactive generator.

2. Dual Caches with RoPE Separation: prevent identity memory from becoming a false continuation of the current shot

The student uses chunk-wise causality: the current chunk can access past information but cannot access ungenerated future chunks. A global cache stores sampled historical context frames for inter-shot subject and scene consistency; a local cache stores generated chunks of the current shot for intra-shot motion continuity. Experiments use 3 latent frames per chunk, a global cache of 2 chunks, and a local cache of 7 chunks. Latent frames and decoded video frames are different units, so cache chunk counts cannot directly determine output duration.

Querying both caches creates ambiguity: is a historical character close-up an identity reference or the immediately preceding visual state to continue? Instead of relying only on a learned marker, the authors introduce a shot-level discontinuity in the temporal phase of rotary position embeddings (RoPE). For latent \(t\) in shot \(k\), the main text gives the temporal rotation angle as:

\[ \Theta_t = \phi t + k\theta. \]

Here, \(\phi\) is the base temporal frequency and \(\theta\) is the shot-boundary phase offset. Temporal progression is preserved within each shot, whereas a discrete shift across shots helps distinguish historical reference from continuation of the current action. The main text does not specify the value of \(\theta\), so the equation should not be used to invent an implementation hyperparameter.

Before a new shot, global context is resampled from generated history; within that shot, generation proceeds chunk by chunk while KV caching reuses previous computation. This does not retain the entire story indefinitely: it preserves visual evidence for next-shot prediction within a fixed budget. Whether details from much earlier scenes remain retrievable in very long narratives therefore requires separate evaluation.

3. Two-Stage Self-Forcing Distillation: adapt first to predicted chunks, then to generated historical shots

Distilling next-shot generation directly into a causal student exposes two training-inference gaps: preceding chunks within the current shot are imperfect, and historical shots are imperfect too. Distribution Matching Distillation (DMD) compresses the multi-step teacher into a 4-step causal student, but historical conditioning is not replaced entirely with generated outputs from the outset. The first stage performs intra-shot self-forcing: global context comes from ground-truth historical shots, while the local cache contains chunks already generated by the student. The student learns to continue its own predictions within a shot while relying on reliable history to establish basic next-shot capabilities.

The second stage performs inter-shot self-forcing: the student generates the first shot from scratch and then conditions subsequent shots entirely on its own generated history. Each new shot still uses chunk-wise self-forcing, and DMD applies only to the newly generated shot at each iteration rather than repeatedly optimizing the entire history jointly. This nested rollout makes both local and global caches resemble inference-time distributions, removing the implicit assumption that historical shots are always correct. The second stage does not replace the first; using it alone performs worse, indicating that adaptation to imperfect long histories benefits from established generation capabilities.

A Worked Example¶

The following illustrates the mechanism rather than reporting another experiment: a story already contains 4 generated shots, and the user requests a close-up of the same character. With a historical budget of 6 frames, each shot first receives 1 frame and the remaining 2 go to the latest shot, producing an allocation of 1, 1, 1, and 3. Those context frames and their respective shot captions supply identity information, while the global caption provides narrative context.

At the new shot, the student uses the RoPE shot offset to separate historical references from current-shot state, generating chunks of 3 latent frames with 4 denoising steps. Subsequent chunks read already generated content to maintain close-up continuity; a later user description affects subsequent generation without requiring regeneration of the emitted history. During the first training stage, the historical visuals in this example come from the dataset; during the second stage and inference, they come from the student itself.

Loss & Training¶

DMD aligns the student's output distribution with the data distribution represented by the teacher, rather than merely copying an individual teacher denoising trajectory sample by sample. At randomly sampled noise levels, training compares scores for the real-data distribution and the student-generated distribution; the latter is learned through a denoising loss, and their difference guides student updates. The cached main text summarizes this mechanism and leaves the detailed objective to supplementary materials, so this note does not reconstruct unverified loss formulas or weights.

The teacher is trained on an internal dataset of 320K multi-shot videos; causal student adaptation begins with regression initialization using 5K teacher-sampled ODE solution pairs. The two distillation stages follow, with the second using captions from a subset of five-shot videos; output resolution is \(832\times480\). The main text does not provide the subset size, complete optimizer settings, or a detailed latency breakdown, all of which remain necessary for reproduction.

Key Experimental Results¶

Main Results¶

The evaluation uses 100 multi-shot prompts generated by Gemini 2.5 Pro, with text adapted to each baseline's expected input style. Speed is measured on one NVIDIA H200; the following selection comes from Table 1, with higher values preferred for all quality metrics and FPS measuring throughput.

Method	Architecture	FPS	Inter-Shot Subject Consistency	Inter-Shot Background Consistency	Transition Control	Text Alignment
EchoShot	Bidirectional	0.643	0.392	0.396	0.664	0.186
LongLive	Causal	16.55	0.594	0.565	0.693	0.216
Rolling Forcing	Causal	15.32	0.561	0.473	0.684	0.223
ShotStream	Causal	15.95	0.654	0.645	0.978	0.234

Inter-shot subject and background consistency use YOLOv11 and SAM to separate keyframe content before evaluation with DINOv2 features; these are not identity-recognition accuracies. Transition control uses Shot Cut Accuracy (SCA), measuring the accuracy of cut counts and temporal positions, with boundaries detected by TransNet V2. The main text does not give the complete SCA formula, so 0.978 should remain a transition-control score rather than being interpreted as 97.8% of videos having entirely correct edits.

Ablation Study¶

The following results come from Table 4, comparing cache distinction mechanisms and distillation stages; the first group changes the distinction strategy, while the second changes training.

Config	Inter-Shot Semantic Consistency	Inter-Shot Subject Consistency	Inter-Shot Background Consistency	Text Alignment	Aesthetic Quality
No indicator	0.728	0.507	0.465	0.203	0.549
Learnable embedding	0.737	0.518	0.588	0.204	0.523
RoPE offset (full configuration)	0.762	0.654	0.645	0.234	0.571
Stage 1 only	0.758	0.604	0.622	0.224	0.568
Stage 2 only	0.704	0.583	0.547	0.218	0.467
Two stages (full configuration)	0.762	0.654	0.645	0.234	0.571

Key Findings¶

Subject consistency rises from 0.507 without an indicator to 0.654 with the RoPE offset, supporting explicit separation of historical and current-shot context over simply mixing caches.
Subject consistency is 0.604 with the first stage alone and 0.654 with both; the second stage alone scores 0.583, supporting foundational training before adaptation to generated history.
The 15.95 FPS throughput does not exceed LongLive's 16.55 FPS; the advantage is stronger multi-shot control at similar causal throughput. Relative to EchoShot's 0.643 FPS, the exact table values imply approximately 24.81 times the throughput, so the text's claim of more than 25 times should not be applied indiscriminately to every bidirectional baseline.
The user study involves 54 participants and 24 prompts; selection rates are 87.69% for visual consistency, 76.15% for prompt following, and 83.08% for visual quality. Multiple selections are allowed, so these are not mutually exclusive pairwise win rates.

Highlights & Insights¶

Making the next shot the interaction unit lets users choose subsequent shots based on visible results. Interactivity begins with reformulating the conditioning task, not just reducing denoising steps.
Historical images must retain their own captions instead of sharing the target caption. This preserves the semantic meaning of remembered visuals and may transfer to editing tasks conditioned on visual history.
Self-forcing must cover both intra-shot and inter-shot recursion. Exposing only the local cache to generated errors does not automatically resolve distribution shift in global history.

Limitations & Future Work¶

The authors report visual artifacts and inconsistencies for complex scenes and prompts, attributing part of the problem to the relatively small backbone; benefits from scaling remain to be tested.
The authors suggest sparse attention and attention sinks for further acceleration. Single-H200 throughput and the abstract's sub-second latency claim do not replace consumer-hardware benchmarks or end-to-end interaction latency measurements.
The internal 320K-video dataset and supplementary details absent from the current cache constrain independent reproduction; the main text only promises code and model release, so this note does not invent a repository URL.
Full-model Dynamic Degrees is 63.56 in Table 1 but 63.06 in Tables 3 and 4, without an explanation in the main text; these values are not treated as interchangeable results from the same experiment.
A fixed 6-frame history budget and predominantly five-shot examples do not establish indefinitely persistent identity memory; future evaluation should vary narrative length, character reappearance intervals, and prompt complexity.

vs EchoShot / CineTrans: These bidirectional multi-shot methods emphasize joint modeling or transition control; ShotStream instead predicts the next shot from history, giving up joint access to future shots to support online shot planning.
vs Self Forcing: The method inherits the use of generated outputs to reduce exposure bias but extends self-forcing from within-shot generation to cross-shot history through progressive training.
vs LongLive / Infinity-RoPE: The former accepts runtime prompts through KV recaching, while the latter uses RoPE Cut for scene changes; ShotStream additionally retains sparse historical memory and trains the next-shot task, aiming to preserve characters and narrative relationships after a transition rather than merely change scenes.

Rating¶

Novelty: 4/5. Next-shot conditioning, dual-cache separation, and inter-shot self-forcing form a coherent system, while the underlying distillation techniques build on prior work.
Experimental Thoroughness: 4/5. Multiple baseline families, two groups of design ablations, and a user study support the claims, but evaluation is limited in scale and training data are not public.
Writing Quality: 4/5. The training-inference relationship is clear, while differing Dynamic Degrees values and the speed summary need more precise explanations.
Value: 4/5. The paper provides a concrete route toward online multi-shot storytelling, with complex-scene stability and longer historical context remaining important boundaries.