StreamDiT: Real-Time Streaming Text-to-Video Generation

Conference: CVPR 2026 arXiv: 2507.03745 Code: https://cumulo-autumn.github.io/StreamDiT/ (project page) Area: Diffusion Models / Video Generation Keywords: Streaming video generation, Diffusion Transformer, real-time inference, sampling distillation, Flow Matching

TL;DR

StreamDiT presents a complete streaming video generation pipeline—covering training, modeling, and distillation—that introduces a sliding buffer with progressive denoising under Flow Matching, a mixed partitioning training strategy, a time-varying DiT architecture with windowed attention, and a customized multi-step distillation method. The resulting 4B-parameter model achieves real-time streaming video generation at 512p@16FPS on a single GPU.

Background & Motivation

1. Background: State-of-the-art text-to-video (T2V) models (e.g., MovieGen, Hunyuan, Step-Video) are built on Diffusion Transformer (DiT) architectures with bidirectional attention, capable of generating high-quality short clips. However, they only support offline generation of fixed-length segments and cannot serve interactive or real-time applications.

2. Limitations of Prior Work:

3. Increasing video length is prohibitively expensive due to the quadratic complexity of Transformers with respect to sequence length.
4. Autoregressive (AR) methods can generate long videos but rely on causal attention, yielding quality far inferior to bidirectional attention.
5. Existing training-free streaming approaches (StreamDiffusion, FIFO-Diffusion) lack training support, limiting their generation quality.

6. Sampling distillation methods (step distillation, consistency distillation) cannot be directly applied to the non-standard setting of streaming denoising.

7. Key Challenge: Low latency (streaming output), high throughput (batch processing), and high quality (bidirectional attention) are difficult to achieve simultaneously. AR methods offer low latency at the cost of quality; bidirectional diffusion offers high quality but cannot produce streaming output.

8. Goal: Design a complete streaming video generation framework that is both trainable and distillable, achieving quality and real-time performance simultaneously.

9. Key Insight: Inspired by the diagonal denoising of FIFO-Diffusion, frames in the buffer are assigned different noise levels. A trainable scheme and mixed partitioning strategy are then introduced to close the quality gap.

10. Core Idea: A unified frame partitioning scheme subsumes uniform noise and progressive diagonal noise as special cases within a single framework. Mixed training improves temporal consistency, and customized multi-step distillation enables real-time inference.

Method

Overall Architecture

Given a text prompt, the StreamDiT model continuously generates video frames via a frame buffer. The buffer holds \(B\) frames, each at a different noise level; after denoising, clean frames are popped from the buffer and output, while new noisy frames are pushed in. The overall pipeline comprises three layers: (1) a Buffered Flow Matching training framework; (2) an efficient model architecture with a time-varying DiT and windowed attention; and (3) customized multi-step distillation for real-time inference.

Key Designs

1. Buffered Flow Matching:

2. Function: Extends standard Flow Matching into a training framework that supports streaming generation.

3. Mechanism: Standard Flow Matching applies the same timestep \(t\) to all frames. StreamDiT instead assigns a monotonically increasing sequence of timesteps \(\tau = [\tau_1, \ldots, \tau_B]\) to frames in the buffer. Training samples are constructed as \(\mathbf{X}_\tau^i = \tau \circ \mathbf{X}_1^i + (1-(1-\sigma_{min})\tau) \circ \mathbf{X}_0\). During inference, the buffer slides along the frame dimension—clean frames are popped and noisy frames are pushed—enabling streaming output.

4. Design Motivation: Introducing the streaming mechanism directly within the Flow Matching framework ensures training–inference consistency and avoids the quality degradation inherent in training-free approaches.

5. Unified Partitioning Scheme:

6. Function: Provides a general framework that unifies different noise allocation strategies.

7. Mechanism: The buffer is divided into \(K\) reference frames and \(N\) chunks, each chunk containing \(c\) frames and \(s\) micro-steps. The total number of frames is \(B = K + N \times c\) and the total denoising steps are \(T = s \times N\). Setting \(c=B, s=1\) reduces to uniform noise (standard T2V); setting \(c=1, s=1\) reduces to diagonal noise (FIFO-Diffusion). Mixed training alternates among different partitioning schemes to prevent overfitting and enhance content consistency.

8. Design Motivation: A single partitioning scheme is prone to overfitting; mixed training leads to more generalizable denoising capabilities. Experiments confirm that mixing all chunk sizes (1, 2, 4, 8, 16) yields the best results.

9. Time-Varying DiT Architecture:

10. Function: Enables the model to handle different noise levels across frames within the buffer.

11. Mechanism: The standard adaLN DiT is modified so that time embeddings are separable along the frame dimension. The latent tensor is reshaped to \([F, H, W]\), and distinct time embeddings controlling scale and shift modulation are applied along the first dimension. Full attention is replaced with windowed attention: the 3D latent is partitioned into non-overlapping windows of size \([F_w, H_w, W_w]\), with alternating shifted windows every other layer to propagate global information. The computational cost is \(\frac{F_w H_w W_w}{FHW}\) of full attention.

12. Design Motivation: Frame-level time embeddings are a prerequisite for the StreamDiT training scheme; windowed attention substantially reduces computation and is critical for achieving real-time inference.

13. Customized Multi-Step Distillation:

14. Function: Reduces the number of sampling steps from 128 to 8 and eliminates CFG, enabling real-time inference.

15. Mechanism: A partitioning scheme with \(c=2, s=16, N=8\) is selected, giving the teacher model \(s \times N = 128\) steps. The Flow Matching trajectory is divided into \(N\) segments, and step distillation is performed independently within each segment. Step distillation and guidance distillation are conducted jointly—the teacher's multi-step CFG inference is distilled into the student's single-step unconditional forward pass. After distillation, the micro-steps \(s\) are reduced from 16 to 1, yielding only 8 total steps.
16. Design Motivation: Standard distillation methods (step distillation, consistency distillation) cannot be directly applied to the non-standard streaming denoising setting; distillation must follow the segment structure of the partitioning scheme.

Loss & Training

Training proceeds in three stages: (1) Task learning—3K high-quality videos, learning rate \(1e{-4}\), to adapt the model to the streaming task; (2) Task generalization—2.6M pre-training videos, learning rate \(1e{-5}\), to improve generalization; (3) Quality fine-tuning—high-quality data with a small learning rate for refinement. Each stage runs for 10K iterations on 128 H100 GPUs. Distillation is conducted on 64 H100 GPUs for 10K iterations.

Key Experimental Results

Main Results (VBench Quality Metrics)

Method	Subject Consistency	Background Consistency	Temporal Flickering	Motion Smoothness	Dynamic Degree	Aesthetic Quality	Quality Score
ReuseDiffuse	0.9501	0.9615	0.9838	0.9912	0.2900	0.5993	0.8019
FIFO-Diffusion	0.9412	0.9576	0.9796	0.9889	0.3094	0.6088	0.7981
StreamDiT (teacher)	0.9622	0.9625	0.9671	0.9861	0.5240	0.6026	0.8185
StreamDiT (distill)	0.9491	0.9555	0.9649	0.9831	0.7040	0.5940	0.8163

Ablation Study (Effect of Mixed Training)

Chunk Size Combination	Quality Score	Notes
[1]	0.8129	Diagonal noise only (Progressive AR Diffusion)
[1,2]	0.8100	Mix of 2 variants
[1,2,4]	0.8080	Mix of 3 variants
[1,2,4,8]	0.8076	Mix of 4 variants
[1,2,4,8,16]	0.8144	Full mix, best performance

Key Findings

• StreamDiT surpasses both ReuseDiffuse and FIFO-Diffusion on quality score and human evaluation (winning across all 4 dimensions).
• Baseline methods exhibit higher temporal consistency and motion smoothness, but their generated content is substantially more static (dynamic degree 0.29–0.31 vs. StreamDiT's 0.52–0.70).
• The distilled model achieves quality very close to the teacher (0.8163 vs. 0.8185) while reducing sampling steps from 128 to 8.
• Training with all chunk sizes mixed yields the best results, even when inference uses only chunk size 1.
• Real-time performance: the distilled model generates 2-frame latents (8 video frames) in 482ms on a single H100, achieving 16 FPS.

Highlights & Insights

• Elegant design of the unified partitioning scheme: The four parameters \((K, N, c, s)\) unify all schemes from standard diffusion to diagonal diffusion within a single framework, providing an exceptionally clean abstraction.
• Unexpected benefit of mixed training: Mixing all chunk sizes—including the non-streaming case of chunk=16—improves streaming generation quality (chunk=1), demonstrating a regularization effect from multi-task training.
• Segment-wise distillation: The idea of dividing the FM trajectory into segments according to the partitioning scheme and distilling each independently is transferable to other distillation scenarios involving non-standard sampling trajectories.

Limitations & Future Work

• The 4B-parameter model has limited capacity, and some generated videos exhibit artifacts (the authors confirm that a 30B model yields significantly improved quality).
• Short context length limits consistency—objects that leave the frame may reappear with altered appearances.
• Windowed attention, while efficient, may sacrifice global consistency.
• Future directions include integrating KV cache to extend context, scaling to larger models, and improving output resolution.

Related Work & Insights

• vs. FIFO-Diffusion: FIFO-Diffusion is a training-free diagonal denoising method; StreamDiT substantially improves quality through its training scheme and mixed partitioning strategy.
• vs. Self-Forcing: Self-Forcing is an AR video diffusion approach that generates one frame at a time with low latency but limited quality; StreamDiT achieves a better balance between quality and latency via bidirectional attention with streaming.
• vs. StreamingT2V: StreamingT2V is an AR approach using short-term and long-term memory blocks; StreamDiT's unified partitioning framework offers a more elegant design.

Rating

• Novelty: ⭐⭐⭐⭐ The unified partitioning scheme and customized distillation are original contributions; the systematic design is comprehensive.
• Experimental Thoroughness: ⭐⭐⭐⭐ Includes VBench quantitative evaluation, human evaluation, ablation studies, and demonstrations across multiple applications.
• Writing Quality: ⭐⭐⭐⭐⭐ The framework is presented with clear hierarchy, rigorous mathematical derivations, and well-crafted figures.
• Value: ⭐⭐⭐⭐⭐ Achieves real-time streaming video generation for the first time, with significant practical impact for interactive applications.

Related Papers

• [CVPR 2026] U-Mind: A Unified Framework for Real-Time Multimodal Interaction with Audiovisual Generation
• [ICLR 2026] MotionStream: Real-Time Video Generation with Interactive Motion Controls
• [NeurIPS 2025] Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generation
• [ICLR 2026] Streaming Autoregressive Video Generation via Diagonal Distillation
• [CVPR 2026] PAM: A Pose-Appearance-Motion Engine for Sim-to-Real HOI Video Generation