Skip to content

DiT as Real-Time Rerenderer: Streaming Video Stylization with Autoregressive Diffusion Transformer

Conference: ECCV2026
Paper: ECCV
Area: Video Generation
Keywords: video stylization, autoregressive DiT, real-time streaming generation, distribution matching distillation, KV cache

TL;DR

RTR-DiT first fine-tunes a bidirectional DiT teacher on a self-built text/reference-paired stylization dataset, then distills it into a 2-step causal autoregressive generator via Self Forcing and Distribution Matching Distillation, and pairs it with a reference-preserving KV cache update strategy so that a single H20 GPU can stream-stylize videos of arbitrary length while switching text prompts or reference images on the fly.

Background & Motivation

Diffusion-based video stylization has been studied for years along two main lines: one maps the video into latent space via DDIM inversion and injects style features, the other extracts structural and motion priors from the scene with frameworks like ControlNet and rerenders on top of those priors. To handle long videos, these methods typically stylize key frames and then propagate the result across the sequence. Both lines share the same weakness: they are built on LDM (U-Net) backbones, where frame-to-frame consistency has to be forced with extra constraints such as optical flow or feature correspondence, and style transfer stays at the level of local texture or geometric patterns; on top of that, the key-frame-plus-propagation pipeline is inherently offline, taking minutes per clip and therefore unable to enter real-time scenarios at all.

DiT changed the picture — both generation quality and temporal consistency are clearly better than LDM, and works such as StyleMaster already fine-tune DiT blocks for stylization while all-in-one editing frameworks like VACE treat stylization as a subtask. But the 3D full attention inside a DiT block is bidirectional along the temporal axis: the estimate for frame n depends on both past and future frames, meaning the whole video must be ready before computation can start. That directly conflicts with the three requirements of streaming, real-time, and arbitrary length. On the other side, a fair number of acceleration techniques already exist — DMD can distill a multi-step sampler into a one- or few-step generator, adding a causal mask yields an autoregressive DiT, and Diffusion Forcing and Self Forcing suppress error accumulation in long rollouts. Yet these techniques were mostly validated for video generation, and assembling them reliably for long-horizon streaming video-to-video translation is still hard: at inference time the reference condition gets pushed out of the rolling KV cache, and the style drifts with it.

This paper's angle is to explicitly reformulate video stylization as a streaming process with causal temporal modeling and efficient inference, and to give that process a dedicated condition-preservation mechanism. Core idea: turn a pretrained bidirectional DiT into a stylization teacher and distill it into a few-step causal autoregressive generator to obtain real-time behavior, while a KV cache update strategy that permanently anchors the reference token at the front simultaneously solves long-video drift and online style switching.

Method

The authors call this process rerendering — editing the appearance and style of the content while preserving the scene structure and intrinsic attributes of the original video. It covers two modes: text-guided video-to-video (TV2V) and reference-guided video-to-video (RV2V).

Overall Architecture

On the data side, the authors collect 5,000 internet videos (covering humans, animals, and natural scenes), use Qwen3-VL to generate stylized text prompts, and Kling to produce the corresponding stylized videos, forming paired training data. TV2V uses the Qwen3-VL prompts directly as the text condition; RV2V randomly takes one frame of the target video as the reference image and fixes the guiding text to "transfer the video style to match the reference image style."

On the model side, the backbone is the DiT-based Wan video model. During training the source video \(V_s\) and target video \(V_t\) are encoded by a VAE into latent representations for N frames, \(x_{1:N}\) and \(z^0_{1:N}\); a reference image, if present, yields its latent \(c_{\text{ref}}\), and the text prompt is encoded into \(c_{\text{text}}\). The clean latent \(z_0\) is perturbed at a continuous timestep \(t\) into \(z_t\), concatenated with the source video latent along the channel dimension, and then patchified into a token sequence. The reference latent passes through a lightweight adapter and is prepended to the sequence along the temporal dimension. All tokens go through the DiT blocks, where a temporal causal mask in 3D attention gives the model frame-by-frame autoregressive prediction.

Training proceeds in three stages: fine-tune the bidirectional teacher, initialize a few-step student with a fixed timestep sequence, then post-train with distillation plus adversarial training. Inference uses a rolling KV cache for streaming generation with the reference tokens anchored, and resets the cross-attention cache when conditions change to switch styles online.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["source · target video · reference<br/>VAE encoding"] --> B["channel-concatenated rerendering condition"]
    B --> C["causal 3D attention<br/>streaming autoregression"]
    C --> D["Self Forcing + DMD + adversarial<br/>few-step post-training"]
    D --> E["reference-preserving KV cache<br/>and live style switching"]
    E --> F["real-time stylized video"]

Key Designs

1. Channel-concatenated rerendering condition: separating "change the look" and "keep the structure" into two complementary inputs

The core difficulty of video stylization is changing the style without losing the content. If the model receives only a text or reference condition, it has no explicit pathway to the spatial layout and motion patterns of the original video, and on long videos it easily degenerates into "regenerating a video with similar content" rather than "reskinning this video." The design concatenates the noised target latent \(z_t\) with the source video latent \(x\) along the channel dimension before patchification: the source latent encodes the layout and motion cues of the original video, so every token to be denoised carries its corresponding source content, turning style transfer into a structure-constrained denoising problem. Note that this step changes neither the token count nor the attention structure, so it adds no overhead.

Style conditions are injected through two paths that share one token layout. Text is encoded (T5) into \(c_{\text{text}}\) and injected via cross attention; the reference image's VAE latent \(c_{\text{ref}}\) passes through a lightweight adapter and is prepended to the token sequence along the temporal dimension, effectively adding one latent frame before the first frame. This puts the reference style and the source video into the same spatiotemporal attention field, so the model can transfer style at the scene and object level rather than only matching local texture. In notation, TV2V uses \(c = \{\varnothing, c_{\text{text}}\}\) and RV2V uses \(c = \{c_{\text{ref}}, c_{\text{text}}\}\); both modes share the same structure and can therefore be trained jointly.

2. Causal 3D attention for streaming autoregression: replacing dense denoising with a fixed few-step schedule

Each teacher block uses full 3D spatiotemporal attention with bidirectional temporal modeling, which rules out streaming editing outright — training already requires seeing future frames. The fix is a temporal causal mask on 3D attention so that the tokens of the noised latent frame \(z_t^n\) can only attend to the current frame and preceding frames \(\{z_t^1, \dots, z_t^n\}\):

\[ \text{Attn}(Q,K,V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}} + M\right)V,\qquad M^{ij}=\begin{cases}0 & \lfloor j/k\rfloor \le \lfloor i/k\rfloor\\ -\infty & \text{otherwise}\end{cases} \]

where \(d\) is the attention head dimension and \(k\) the number of spatial tokens per latent frame; the integer comparison over \(\lfloor \cdot \rfloor\) lets tokens within a frame see each other while across frames only past-to-present attention is allowed (⚠️ this equation is garbled in the cache; it is restored here in the standard causal-mask form — refer to the original paper). Denoising no longer iterates densely over the whole time horizon either; it follows a fixed timestep sequence \(\{s_1, \dots, s_K\}\), yielding a few-step autoregressive generator \(G_\phi\). The experiments set \(K=2\), the best trade-off between quality and inference speed.

The two changes map onto "can stream" and "can be real-time": the causal mask restricts each frame's visibility to history, so frames can be emitted one by one without waiting for later frames; the fixed few-step schedule cuts per-frame model forwards from dozens down to 2. The cost is equally direct — after causalization and few-step compression the model quality collapses noticeably, and post-training has to recover it.

3. Self Forcing + DMD + adversarial few-step post-training: pulling the causal few-step model back to teacher level

Causalization plus few-step compression hurts the metrics badly (on TV2V, AQ drops from 0.6514 to 0.5446 and IQ from 0.7406 to 0.6380), and there is a subtler gap underneath: at training time each frame is conditioned on ground-truth history, while at inference it can only be conditioned on the model's own generated history, so exposure bias accumulates over long rollouts. Three ingredients address this together. First, Self Forcing conditions each training step on the model's own rolled-out history, so the conditional generation distribution factorizes frame by frame as \(p_\phi(\mathbf{z}_0^{1:N}\mid c)=\prod_{n=1}^{N} p_\phi(\hat{\mathbf{z}}_0^{n}\mid \hat{\mathbf{z}}_0^{<n}, c)\), aligning the source of conditioning between training and inference. Second, DMD distills the multi-step bidirectional teacher into the few-step \(G_\phi\): two score networks are initialized from the teacher's weights — a frozen \(N_{\text{data}}\) estimating the real-data score and a trainable \(N_{\text{gen}}\) estimating the generator-induced score — and the reverse KL between the two induced distributions is minimized, with a gradient that is simply the difference of the two scores times the upstream derivative:

\[ \nabla_\phi \mathcal{L}_{\text{DMD}} = \mathbb{E}\left[\Big(S_{\text{data}}\big(F(G_\phi(\mathbf{z}_t,\mathbf{x},c),t),t\big) - S_{\text{gen}}\big(F(G_\phi(\mathbf{z}_t,\mathbf{x},c),t),t\big)\Big)\frac{dG}{d\phi}\right] \]

where \(F\) is the forward diffusion process. For every update of \(G_\phi\), \(N_{\text{gen}}\) is updated five times to keep up with the current generator distribution. Third, adversarial post-training attaches a classification branch to \(N_{\text{gen}}\) as a discriminator \(D\) and optimizes a standard GAN objective (real samples \(\log D\), generated samples \(-\log D\)) for further perceptual quality.

The division of labor is clear: Self Forcing closes the train–test gap, DMD provides distribution-level supervision (better suited to few-step generation than per-sample regression), and the adversarial term adds perceptual quality. Together they let the 2-step causal model match or exceed a bidirectional teacher that takes 1.43 minutes, while inference stays in the 0.12-minute bracket.

4. Reference-preserving KV cache update and live style switching: keeping the reference token at the front forever

Long videos rely on a rolling KV cache: fixed-length caches are initialized for both 3D attention and cross attention, newly generated frames are appended, and once full the oldest frames are evicted. That works for TV2V but breaks RV2V — the reference image is prepended at the front of the sequence, so the moment the cache fills up it is evicted, and no subsequent frame can ever see the reference style again; the output gradually degrades and the style drifts away. The proposed reference-preserving update anchors the reference image tokens permanently at the front of the cache once it is full, while the remaining positions keep rolling forward with new frames. The reference turns from "a position in the sequence" into "a resident slot in the cache," so however long the video runs, every frame can still attend to the reference image.

The same mechanism incidentally enables online style switching. When a user supplies a new text prompt or reference image mid-stream, the cross-attention KV cache is reinitialized (discarding the old text condition) while the 3D attention cache retains the most recent frame as a bridge, so the transition between styles stays smooth instead of flickering; if the new condition is itself a reference image, it is anchored at the front as well. Retaining the last frame rather than clearing everything matters because an empty cache would force the model to "start over" with no history and hard-cut at the switch point, whereas keeping one frame is effectively a soft, conditioned transition.

Loss & Training

Three-stage training. ① Fine-tune the bidirectional teacher on the self-built stylization dataset for 10,000 steps with a flow matching objective. ② Initialize the few-step student with a fixed timestep sequence \(\{s_1, \dots, s_K\}\) and train it for 10,000 steps to obtain a strong initialization. ③ Post-training distillation (Self Forcing + DMD + adversarial) for 5,000 steps. All experiments use \(K=2\). TV2V and RV2V samples are trained jointly.

The teacher's flow matching objective and noising form are

\[ \mathcal{L}_{\text{FM}} = \mathbb{E}_{\mathbf{z}, c, t}\big\| v_\theta(\mathbf{z}_t, \mathbf{c}, t) - (\mathbf{z}_1 - \mathbf{z}_0)\big\|^2,\qquad \mathbf{z}_t = (1-t)\,\mathbf{z}_0 + t\,\mathbf{z}_1,\quad t \sim \mathcal{U}(0,1) \]

where \(\mathbf{z}_0\) is the clean latent encoded from data, \(\mathbf{z}_1 \sim \mathcal{N}(0, I)\) is standard Gaussian noise, and \(v_\theta\) is the predicted velocity field. The adversarial term takes the standard form \(\mathcal{L}_{\text{GAN}} = \mathbb{E}[\log D(\mathbf{z}_0, \mathbf{x}, c)] + \mathbb{E}[-\log D(F(G_\phi(\mathbf{z}_t, \mathbf{x}, c), t), \mathbf{x}, c)]\). ⚠️ Equations (1) and (2) are garbled in the cache (Eq. 2 is missing the \(\mathbf{z}_0\) term); they are restored here in the standard flow matching form — refer to the original paper.

Key Experimental Results

Main Results

The evaluation set consists of 50 videos collected from Pexels. For TV2V, ten diverse stylization prompts (e.g. ink wash painting, Minecraft) are randomly paired with the videos. For RV2V, the first frame of each video is style-transformed with Kling to serve as the reference. All videos are 5-second clips at 832×640 and 24 fps, and all tests except the commercial models run on a single H20 GPU. Metrics are CLIP-T (text–video similarity), CLIP-F (intra-frame similarity, a temporal consistency proxy), VBench's AestheticQuality (AQ) and ImagingQuality (IQ), CSD-Score (style similarity between reference image and video frames), and generation time per video.

Text-guided (TV2V) comparison:

Method CLIP-T ↑ CLIP-F ↑ AQ ↑ IQ ↑ Time (min) ↓
Rerender-A-Video 0.2272 0.9883 0.5730 0.7012 4.5
FRESCO 0.1970 0.9907 0.6060 0.6834 7.3
TokenWarping 0.2122 0.9857 0.6093 0.7324 3.2
RTR-DiT 0.2585 0.9914 0.6302 0.7279 0.12

Reference-guided (RV2V, CSD-Score replaces CLIP-T):

Method CSD-Score ↑ CLIP-F ↑ AQ ↑ IQ ↑ Time (min) ↓
VACE 0.4046 0.9905 0.6012 0.7165 27.6
StyleMaster 0.4275 0.9892 0.6536 0.7247 28.5
Gen-4 Aleph (commercial) 0.8312 0.9925 0.6336 0.7378 3.5
RTR-DiT 0.7958 0.9909 0.6714 0.7321 0.12

Note on the time metric: 0.12 min ≈ 7.2 s is the total generation time for a 5-second clip; converted to the 120 frames this is roughly 60 ms/frame (~16 fps), about 27× faster than the next-fastest TokenWarping (3.2 min). The paper's "real-time" claim therefore refers primarily to the streaming, low-latency deployment form (with a rolling cache, no need to wait for later frames); strictly speaking 7.2 s end-to-end for a 5-second clip is not faster-than-real-time playback. Per-frame latency is not reported in the table — this is a frame-count conversion (⚠️ refer to the original paper).

Ablation Study

Effect of post-training and causalization (same evaluation set; top TV2V, bottom RV2V):

Training stage CLIP-T / CSD ↑ CLIP-F ↑ AQ ↑ IQ ↑ Time (min) ↓
Bidirectional teacher (TV2V) 0.2463 0.9931 0.6514 0.7406 1.43
Causal, no post-training (TV2V) 0.2284 0.9948 0.5446 0.6380 0.12
RTR-DiT, post-trained (TV2V) 0.2585 0.9914 0.6302 0.7279 0.12
Bidirectional teacher (RV2V) 0.8050 0.9930 0.6583 0.7144 1.43
Causal, no post-training (RV2V) 0.6002 0.9956 0.5897 0.6021 0.12
RTR-DiT, post-trained (RV2V) 0.7958 0.9909 0.6714 0.7321 0.12

The reference-preserving (RP) KV cache strategy is evaluated only qualitatively: without anchoring, generated frames gradually degrade during inference and drift away from the reference style, whereas anchoring keeps stylization stable over long videos (real-world clips on the order of one minute). ⚠️ No quantitative metric accompanies this strategy, which is the weakest link in the paper's ablations.

Key Findings

  • Post-training is a necessity for causalization, not a bonus: removing post-training from the causal model costs 0.1069 AQ (0.6514 → 0.5446) and 0.1026 IQ (0.7406 → 0.6380) on TV2V, and 0.2050 CSD-Score (0.8050 → 0.6002) on RV2V — so the causal mask plus fixed few-step schedule alone damages quality substantially. Adding Self Forcing + DMD + adversarial training not only restores teacher-level metrics but surpasses the teacher on CLIP-T (0.2585 vs 0.2463) and on RV2V AQ (0.6714 vs 0.6583).
  • CLIP-F runs opposite to the quality metrics: the causal models (post-trained or not) score highest on CLIP-F (TV2V 0.9948, RV2V 0.9956), above the bidirectional teacher. Since CLIP-F measures intra-frame similarity, blurrier and flatter frames score higher, so this metric cannot alone support a temporal-consistency claim — it must be read together with AQ/IQ and the qualitative results.
  • The speed advantage is an order of magnitude, not a few percent: the three LDM/propagation baselines take 3.2–7.3 minutes on TV2V, and VACE/StyleMaster need 27.6/28.5 minutes on RV2V, whereas RTR-DiT is uniformly 0.12 minutes. The commercial Gen-4 Aleph shows 3.5 minutes in the table, but the paper notes practical use still involves queueing.
  • Style fidelity still trails the commercial model: RV2V CSD-Score of 0.7958 is clearly below Gen-4 Aleph's 0.8312, with AQ (0.6714 vs 0.6336) and IQ (0.7321 vs 0.7378) trading wins. The paper positions itself as best among open-source methods and competitive with commercial ones, not comprehensively superior.
  • The user study supports the same conclusion: 12 participants, 16 randomized comparison groups (8 per setting), reporting average rank (lower is better) on condition/style alignment, visual quality, and overall preference:
Setting Method Align. ↓ Quality ↓ Overall ↓
TV2V Rerender-A-Video / FRESCO / TokenWarping 2.78 / 3.25 / 2.83 3.05 / 2.48 / 3.15 2.91 / 2.74 / 3.10
TV2V RTR-DiT 1.14 1.32 1.25
RV2V VACE / StyleMaster / Gen-4 Aleph 3.88 / 3.07 / 1.76 3.20 / 3.58 / 1.60 3.48 / 3.45 / 1.64
RV2V RTR-DiT 1.29 1.61 1.44

RTR-DiT ranks first on all three TV2V criteria; on RV2V it leads on alignment and overall preference, while its visual quality (1.61) is essentially level with Gen-4 Aleph (1.60) — that is, commercial-model-level appearance under real-time inference.

Highlights & Insights

  • Promoting "the reference" from a sequence position to a resident cache slot: a rolling KV cache normally evicts all tokens alike by age, but a failed condition token costs far more than a failed history frame — losing one reference image permanently removes the style anchor for every later frame. Anchoring the reference at the front is a tiny change with a large payoff, and the idea transfers to any "long streaming + persistent condition" task (streaming audio-driven avatars, identity preservation in long video, system-prompt retention in streaming agents).
  • Keeping the most recent frame as a soft transition on a condition switch: style switches most easily fail by hard-cutting or flickering. Clearing the entire cache forces the model to restart a video, while retaining one frame leaves a conditioned transition point between the old and new styles. This is a practically useful trick that costs almost nothing.
  • The three training stages have cleanly separated roles: the bidirectional teacher sets the quality ceiling, the fixed few-step initialization gives the causal student a good starting point, and Self Forcing + DMD + adversarial training pulls that starting point back to the ceiling. The ablation (Tab. 2) confirms none of the three is dispensable — the "causal, no post-training" rows in particular show that a causal mask alone does not yield usable quality.
  • Channel concatenation replaces a structural control network: ControlNet-style methods need an extra branch and extra forwards, whereas here a single source latent concatenated along channels changes neither token count nor attention nor inference cost, yet delivers structure preservation. This "zero-structural-cost condition injection" is worth reusing in other video-to-video tasks.
  • Text and reference image share one token layout: treating the reference as an extra latent frame makes TV2V and RV2V structurally isomorphic, which enables joint training and lets both switch under the same KV cache mechanism. A good example of unifying multiple conditions in one representation.

Limitations & Future Work

  • The authors provide no quantitative ablation for the RP KV cache strategy, only a qualitative figure (Fig. 6b), so its gain over naive rolling cannot be measured — unfortunate given that this is one of the paper's core contributions.
  • The evaluation scale is small: 50 videos, only 10 prompts for TV2V, and RV2V references are produced by applying Kling transformations to each video's first frame (using one model's output as the style reference and evaluating on the same kind of style risks being same-source as the training data). This pairing simplifies standardized comparison but weakens the test of cross-domain reference images, which is the realistic use case.
  • The evidence for "real-time" is incomplete: the paper only reports total time per 5-second clip (0.12 min) and gives no per-frame latency, cache capacity, memory footprint, or throughput, whereas a real application cares about first-frame latency and stable frame rate. The frame-count conversion gives roughly 60 ms/frame (~16 fps), still short of the native 24 fps — ⚠️ this conversion is not confirmed by the paper.
  • Training data depends on a commercial model (Kling) to generate target videos, so the style distribution is bounded by the upstream model's capability and preferences; the dataset (5,000 source videos) is also small by video-generation standards.
  • Concrete improvement directions: ① add a quantitative ablation for the RP cache (e.g. fix the cache length and plot CSD-Score against frame index with and without anchoring); ② report per-frame and first-frame latency, and push inference past 24 fps; ③ generalize anchoring from a single reference image to multiple references (several style images, style interpolation) to enable continuous control of style strength; ④ condition switching currently requires resetting the cross-attention cache — finer-grained, region-wise local style switching is worth investigating.
  • vs Rerender-A-Video / FRESCO / TokenWarping (LDM propagation): they stylize key frames and propagate the result with optical flow or token warping, so a whole clip must be processed offline (3.2–7.3 min) and CLIP-T/AQ remain clearly below this work. This paper rerenders frame by frame with causal autoregression and delegates condition preservation to the KV cache instead of a propagation algorithm — the reason it achieves real-time behavior and quality simultaneously. The cost is a hard dependency on a DiT backbone pretrained on large-scale video, whereas propagation methods can be attached to off-the-shelf image stylization models with zero training.
  • vs VACE / StyleMaster (DiT editing): also DiT-based, but they keep bidirectional 3D attention and can therefore only run long videos offline (27.6/28.5 min); on RV2V, VACE merely adjusts brightness and fails to capture the reference style (CSD 0.4046). The key difference is the trio of causalization, few-step distillation, and anchored cache, which together trade "offline, high quality" for "streaming, near real-time, slightly lower quality."
  • vs Gen-4 Aleph (commercial): the commercial model is stronger on CSD-Score (0.8312 vs 0.7958), indicating headroom in reference style fidelity, but this work has higher AQ and no queueing. One direction is to combine the anchored-cache mechanism here with a stronger style encoder, or conversely to distill a stronger teacher with this streaming framework.
  • vs Self Forcing / DMD (autoregressive video generation): these were designed for generation; this paper uses Self Forcing to close the train–test gap caused by conditioning on ground-truth history, DMD to compress to 2 steps, and adds an adversarial term, making it a fairly complete example of porting existing acceleration techniques to video-to-video. The broader lesson: any conditional generation task that must be real-time can reuse the three-stage recipe of teacher fine-tuning → few-step initialization → Self Forcing + DMD + adversarial training; what genuinely has to be designed for the task is how the condition is preserved in the streaming state (here, anchoring).

Rating

  • Novelty: ⭐⭐⭐⭐ First DiT-based real-time video stylization framework; however causal masking, Self Forcing, DMD, and rolling KV caches are all existing components, so the novelty lies mainly in the system-level combination and the reference-preserving cache strategy.
  • Experimental Thoroughness: ⭐⭐⭐ Main comparisons, a three-stage ablation, and a user study are all present, but the core RP cache strategy is only evaluated qualitatively; the benchmark has just 50 videos and TV2V uses only 10 prompts.
  • Writing Quality: ⭐⭐⭐⭐ Clear structure and a complete motivation chain; weaknesses are that AQ/IQ/CSD are not defined in the main text and that the time metric is easy to misread against the "real-time" claim.
  • Value: ⭐⭐⭐⭐ Real-time long-video stylization with live style switching is directly valuable for VR/AR, live streaming, and digital content creation, and has already been deployed as a real application with a smartphone front end and a server backend.