LiveEdit: Towards Real-Time Diffusion-Based Streaming Video Editing¶

Conference: ECCV 2026
arXiv: 2606.26740
Code: To be confirmed (Project page https://live-edit.github.io)
Area: Video Generation / Video Editing
Keywords: Streaming Video Editing, Diffusion Model Distillation, Causal Attention, Masked Caching, Real-Time Inference

TL;DR¶

LiveEdit transfers the editing capabilities of a bidirectional diffusion Transformer to a causal streaming editor (4-step inference) through a three-stage progressive distillation pipeline. Leveraging the high temporal redundancy of self-attention layers in background regions, it designs an AR mask cache to achieve real-time inference at 12.66 FPS while preserving the editing quality of the bidirectional model.

Background & Motivation¶

Video editing technologies have advanced rapidly in recent years. From text-guided local modifications to style transfer, offline bidirectional diffusion models can generate high-fidelity results. However, these models rely on global attention mechanisms, requiring the entire spatiotemporal context of the video to be processed before outputting the first frame. In scenarios demanding immediate response, such as augmented reality and live interactive streaming, this "process-then-edit" paradigm introduces unacceptable latency. The field has consequently shifted toward streaming video editing: causally processing videos frame-by-frame or chunk-by-chunk without relying on future frame information. However, this transition faces two core bottlenecks. The first is attention distribution shift: bidirectional models in the offline stage concentrate a large amount of attention weight on the nearest neighboring frames to obtain local structural consistency. Once simply truncated into a causal mask by removing the key/value of future frames, the attention distribution flattens sharply—weights become uniformly distributed across all historical frames. This causes the model to lose local structural priors, leading to severe flickering and content "forgetting". The second is spatiotemporal token redundancy: in streaming frame-by-frame inference, every arriving frame requires a full forward pass through the diffusion network. Even if only a small portion of pixels are edited, the remaining 90%+ background regions still bear 100% of the computational cost, rendering real-time deployment on edge devices out of reach.

Meanwhile, existing streaming video generation works (e.g., StreamDiffusion, Self-Forcing) are primarily tailored for text-to-video synthesis from scratch. They encounter fundamental mismatches when directly transferred to editing scenarios. Video generation only requires temporal and stylistic continuity between consecutive frames, whereas video editing must strictly preserve pixel-wise consistency of unedited regions from the source video (e.g., background textures, human identity). Existing caching schemes (DeepCache, VMem) mainly perform full-layer feature reuse for generation tasks, which causes high-frequency detail loss and structural drift in editing tasks. The key insight of LiveEdit is that the architectural gap and computational redundancy are two orthogonal issues that warrant targeted solutions: using three-stage distillation to perform a "soft landing" of bidirectional priors into a causal architecture, while designing a selective cache based on the high temporal redundancy of self-attention layers in background regions discovered through experiments.

Core Idea: Solve the bidirectional-to-causal attention distribution shift via a three-stage progressive distillation pipeline (Bidirectional Fine-Tuning \(\rightarrow\) Teacher Forcing Causal Alignment \(\rightarrow\) DMD 4-Step Distillation), while designing an AR mask cache based on the high temporal redundancy of the self-attention layer in unedited regions, achieving both strict background preservation and a real-time inference speed of 12.66 FPS in streaming editing.

Method¶

Overall Architecture¶

The overall pipeline of LiveEdit is to first leverage a bidirectional DiT to learn editing capabilities, and then transform it into an efficient causal streaming editor through three steps. Stage 1 (Foundation Tuning) performs editing fine-tuning on the homework bidirectional DiT, feeding the source video and noisy latents via channel concatenation. This avoids the quadratic complexity expansion of long-sequence attention while allowing the model to master complex editing mappings. Stage 2 (Teacher Forcing for Causal Initialization) introduces a chunk-level causal attention mask and employs the bidirectional model learned in Stage 1 as a teacher. This forces the output distribution of the causal DiT to align with the bidirectional distribution, preventing structural collapse after truncating future frames. Stage 3 (DMD for Streaming Editing) skips the expensive ODE initialization and directly uses the Stage 2 weights to initialize the DMD 4-step generator. Coupled with a frozen Real Score and a trainable Fake Score, it performs distribution matching distillation while removing CFG to compress a single forward pass from 200 NFEs to 4.

During inference, the AR mask cache computes the pixel-wise L2 distance between the edited output of the previous chunk and the source latent to generate an editing mask. This divides the spatial tokens of the current frame into two categories: "edited regions" (full computation of SA+CA+FFN) and "background regions" (skipping SA by reusing cached features, while CA and FFN are executed normally).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Source Video Latent + Noisy Latent<br/>Channel Concatenation"] --> B["Stage 1: Foundation Tuning<br/>Bidirectional DiT Editing Fine-Tuning"]
    B -->|"100 Steps + CFG<br/>MSE Loss"| C["Stage 2: Teacher Forcing<br/>Chunk-level Causal Attention<br/>Aligns with Bidirectional Dist."]
    C -->|"100 Steps + CFG<br/>MSE Loss"| D["Stage 3: DMD Distillation<br/>4-Step Generator<br/>Real Score ↔ Fake Score"]
    D -->|"4 Steps + No CFG<br/>MSE + DMD Gradient"| E["Streaming Inference<br/>chunk=3 frames"]
    E --> F{"Editing Mask M^k<br/>L2 Distance > Threshold?"}
    F -->|Yes (Edited Region)| G["Full Computation<br/>SA + CA + FFN"]
    F -->|No (Background Region ~70%)| H["SA: Reuse Cached Features<br/>CA + FFN: Executed Normally"]
    G --> I["Output Edited Frame"]
    H --> I

Key Designs¶

1. Three-Stage Progressive Distillation Pipeline: Soft Landing from Bidirectional Priors to Efficient Streaming Editing

Converting an offline bidirectional diffusion model into a causal streaming model is not a simple mask replacement. Removing the key/value of future frames causes the attention distribution to flatten dramatically, completely losing local structural priors. LiveEdit's three-stage design makes this transformation gradual, with each stage solving only one problem. Stage 1 merges the source video latent and the noisy latent into the bidirectional DiT using channel concatenation (rather than sequence concatenation), fine-tuning with a standard MSE noise matching loss to obtain a strong offline editing prior. Stage 2 introduces a chunk-level causal attention mask (each chunk has 3 frames, allowing attention only to the current and preceding chunks). The core technique is Teacher Forcing—using the output of the bidirectional model learned in Stage 1 as a teacher signal to force the causal DiT's attention distribution to align with the bidirectional distribution. Experiments show that directly truncating with a causal mask drops Text Alignment (TA) from 0.268 to below 0.210, whereas after Teacher Forcing, it yields only a slight drop to 0.264. Stage 3 goes further: since Stage 2 has already highly aligned the causal model's distribution, its weights can be directly used to initialize the 4-step generator of DMD, skipping the expensive ODE initialization required in methods like Self-Forcing. DMD distillation utilizes a frozen Real Score model \(\epsilon_{\phi}^{real}\) and a trainable Fake Score model \(\epsilon_{\psi}^{fake}\) to calculate the distribution matching gradient \(\nabla_\theta\mathcal{L}_{DMD} = \mathbb{E}_{z_T,c}[w(t)(\epsilon_{\phi}^{real}(z_t,t,c)-\epsilon_{\psi}^{fake}(z_t,t,c))\nabla_\theta G_\theta(z_T,c)]\), while using the MSE loss as an anchor. Jointly, the three stages compress inference from 200 NFEs (100 steps \(\times\) 2-way CFG) to 4, achieving a \(50\times\) speedup, and the Image Quality even improves from 0.716 (bidirectional baseline) to 0.720.

2. AR Mask Cache: Dynamic Computation Routing Based on Temporal Redundancy of Self-Attention Layers

In streaming videos, background regions of adjacent frames are highly similar, but different modules exhibit varying levels of redundancy—the self-attention layer captures global context dependencies, where features in unedited regions remain almost unchanged; meanwhile, the FFN contains high-frequency spatial information unique to each frame, and cross-attention is responsible for text condition alignment, so direct reuse causes severe degradation. Based on this observation, during inference, LiveEdit computes the spatial pixel-wise L2 distance between the edited output latent \(z_{edit}^{k-1}\) and the source latent \(z_{src}^{k-1}\) of the previous chunk and applies a threshold to obtain a binary editing mask \(M_{u,v}^k = \mathbb{I}(\|z_{edit,u,v}^{k-1} - z_{src,u,v}^{k-1}\|_2 > \tau)\). The threshold \(\tau\) is dynamically adjusted to ensure that approximately 70% of the tokens are pruned. For regions masked as "background", the self-attention computation is completely skipped, and intermediate features directly reuse the Token Cache from the previous chunk, while cross-attention and the FFN are still executed normally. Ablation experiments clearly support this differentiated design: when caching is applied to self-attention, BC (Background Consistency) is perfectly maintained at 0.956 and TA improves to 0.270; whereas when caching is applied to the FFN, TA plummets to 0.236 and BC drops to 0.841, confirming that FFN feature similarity distribution indeed lacks temporal redundancy. This mechanism further compresses the frame-by-frame inference latency to 79ms, achieving an additional \(\approx 2.5\times\) speedup on top of the 4-step distillation.

Loss & Training¶

Each of the three stages has different objectives and configurations. Both Stage 1 and Stage 2 use the standard MSE noise-matching loss, with Stage 2 optimized under the constraint of causal attention masks. Stage 3 jointly optimizes the MSE loss and the DMD gradient. The training data consists of 20K high-quality video pairs filtered from Ditto-1M, and the base model is Wan2.1-T2V-1.3B. The batch size for all three stages is 8, the learning rate is \(10^{-5}\), and training is conducted on 8 A100 GPUs for 9K / 20K / 10K steps, respectively. The chunk size for Stage 2 is set to 3 latent frames, and the time steps for Stage 3's 4-step sampling are set to [0, 250, 500, 750].

Key Experimental Results¶

Main Results¶

We compare our method against 6 baselines (3 offline bidirectional + 3 streaming generation) on a dedicated streaming editing benchmark containing 120 video pairs, evaluated using 6 automatic metrics.

Method	TA↑	BC↑	MS↑	DD↑	AQ↑	IQ↑
LucyEdit	0.253	0.943	0.990	0.266	0.529	0.707
VideoCoF	0.245	0.953	0.991	0.094	0.542	0.709
InsV2V	0.259	0.943	0.986	0.196	0.577	0.708
StreamDiffusion	0.239	0.886	0.975	0.239	0.590	0.717
StreamDiffusionV2	0.252	0.951	0.992	0.264	0.539	0.653
StreamV2V	0.244	0.934	0.989	0.153	0.548	0.712
LiveEdit (w/o Cache)	0.265	0.956	0.991	0.282	0.584	0.720
LiveEdit (w/ Cache)	0.270	0.956	0.992	0.256	0.581	0.708

LiveEdit achieves optimal performance on core metrics such as Text Alignment (TA) and Background Consistency (BC). Notably, the causal architecture actually surpasses all bidirectional models on TA (0.270 vs. 0.259 for InsV2V), the potential reasons for which are analyzed later. Incorporating the cache further improves TA while perfectly preserving BC.

Ablation Study¶

Configuration	TA↑	IQ↑	Latency (NFEs)	Description
Stage 1 Bidirectional Baseline	0.268	0.716	197.48s (100+CFG)	Offline model, cannot stream
+ Stage 2 Teacher Forcing	0.264	0.702	200.36s (100+CFG)	Achieves streaming but latency actually increases
+ Stage 3 DMD Distillation	0.265	0.720	7.89s (4, w/o CFG)	50× speedup, comparable quality
Cache on Self-Attention	0.270	0.708	79ms/frame	Optimal, almost no degradation
Cache on FFN	0.236	0.513	—	Severe degradation, unusable

Key Findings¶

The slight decreases in TA and IQ from Stage 1 to 2 (0.268 \(\rightarrow\) 0.264 / 0.716 \(\rightarrow\) 0.702) reveal that causal truncation indeed comes with a cost, but Teacher Forcing effectively suppresses this degradation. Stage 3 DMD distillation not only avoids further degradation but even beats the bidirectional baseline on IQ (0.720), proving the generative quality preservation capability of 4-step distillation.
The differentiated finding of SA Cache vs. FFN Cache is one of the most important insights of this work: the cosine similarity of self-attention layer features on the background regions of adjacent frames is close to 1.0, whereas FFN features show extremely low similarity between consecutive denoising steps. This indicates that the high-frequency spatial information of FFN is not temporally reusable. This quantitative explanation holds direct guiding value for other diffusion model caching works.
The causal architecture surprisingly outperforms the bidirectional model on TA (0.270 vs. 0.259): a plausible explanation is that because the causal model cannot see future frames, it is forced to decode the current frame's editing instructions more accurately for precise alignment, whereas the bidirectional model has more context tolerance and consequently relaxes on fine-grained text alignment.
In a 20-participant user study, LiveEdit achieved a 100% top-3 preference rate and the vast majority of "best" votes in instruction consistency, and dominated all baselines with 75% absolute "best" votes in background preservation (with LucyEdit receiving only 12.5%).

Highlights & Insights¶

Three-Stage Progressively Decoupled Distillation Design: Instead of converting bidirectional to causal in one giant step, this work decouples the pipeline into three independently researchable stages: "establishing editing capabilities \(\rightarrow\) aligning causal distribution \(\rightarrow\) compressing steps". In particular, utilizing Stage 2 causal weights directly as the initialization for DMD distillation bypasses the expensive ODE initialization phase, serving as a highly valuable engineering technique.
FFN and Self-Attention Caching Discrepancy Experiments: While many caching-related works treat FFN and SA indiscriminately for pruning or reuse, LiveEdit quantitatively demonstrates their intrinsic differences—SA focuses on contextual relationships (temporally stable backgrounds), whereas FFN focuses on spatial high-frequency details (differing across frames). This finding is generalizable to all cache-based acceleration methods.
Unexpected Domination of Causal Architecture in Text Alignment: This counter-intuitive experimental finding offers a new perspective on modeling pathways for streaming video editing: causal constraints may inherently favor fine-grained instruction comprehension, warranting further theoretical analysis and validation.

Limitations & Future Work¶

The editing mask depends on the L2 distance of the previous chunk. When an object suddenly enters the frame (motion not caused by editing instructions), it may be falsely identified as an "edited region", triggering full computation. Consequently, the cache hit rate declines in scenarios with rapid motion or large camera zooms.
The 70% pruning rate is empirically set and lacks an adaptive mechanism for different editing scenarios (large-area editing vs. minor tuning); ideally, it should dynamically adjust based on the proportion of the edited region.
Experiments are only conducted on Wan2.1-1.3B. Whether the distillation convergence and caching strategies remain robust when scaling to larger models (7B+) requires verification—FFN layers in larger models carry more complex spatial mappings, meaning the reuse window might need to be compressed.
The training data is limited to 20K pairs, all filtered from Ditto-1M. Data diversity and scale remain potential bottlenecks for further scaling the generalization capability.

vs. Self-Forcing: Self-Forcing uses self-generated conditions to bridge the training-inference distribution gap, targeting text-to-video generation; LiveEdit deals with video editing demanding strict preservation of source structures. Self-Forcing relies on the expensive overhead of ODE initialization, whereas LiveEdit bypasses this step through Stage 2 causal teacher guidance.
vs. FlashVSR: Both use three-stage distillation to achieve real-time streaming super-resolution. However, super-resolution is a low-level pixel mapping task, whereas editing involves complex semantic reconstruction. LiveEdit additionally introduces an AR mask cache to address the spatially selective redundancy problem unique to editing scenarios.
vs. StreamDiffusionV2: StreamDiffusionV2 utilizes sink-token-guided rolling KV caching to accelerate streaming generation, but such caching is designed for global resynthesis and disrupts the structure of unedited regions in editing tasks. LiveEdit performs spatially selective caching based on the editing mask, caching only the self-attention features of background regions.

Rating¶

Novelty: ⭐⭐⭐⭐ [The combined design of three-stage distillation + AR mask caching is proposed for the first time in the field of streaming video editing, and the experimental finding of SA vs. FFN caching difference holds general guiding value; however, the individual components (DMD, Teacher Forcing, staged distillation) are all existing techniques.]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [120-pair dedicated benchmark + 6 automatic metrics + 20-participant user study, with clear ablation designs for both the three stages and caching positions, providing a complete chain of evidence.]
Writing Quality: ⭐⭐⭐⭐ [The formulation of motives and visualization of motivation (attention distribution shift, token similarity distribution) are of high quality; however, the appendix section is relatively brief.]
Value: ⭐⭐⭐⭐⭐ [Achieving high-quality streaming video editing at 12.66 FPS directly fills the gap between offline bidirectional and real-time streaming editing, with direct driving value for AR and live streaming scenarios.]