Direct Autoregressive Diffusion Distillation via Error-aware Causal Pretraining¶
Conference: ECCV 2026
Paper: ECCV Official
Area: Video Generation
Keywords: Autoregressive Video Diffusion, Diffusion Distillation, Error Forcing, Causal Pretraining, Exposure Bias
TL;DR¶
To resolve training collapse and prohibitive trajectory-matching overhead in autoregressive video diffusion distillation caused by architectural mismatch, this paper introduces Error Forcingโa parallelizable causal pretraining scheme that injects empirical prediction residuals into the historical context to mitigate exposure bias, enabling the causal student to directly undergo few-step distribution matching distillation without ODE trajectory matching.
Background & Motivation¶
The emerging vision of interactive world modeling and real-time content creation demands high-quality, low-latency autoregressive (AR) video diffusion models. Unlike conventional full-sequence bidirectional diffusion architectures, AR video models must sequentially predict future frames conditioned on past context while maintaining minimal per-step inference latency. To achieve both generation fidelity and streaming speed, state-of-the-art workflows distill powerful bidirectional foundation models (such as Wan2.1) into few-step causal student models. However, naively initializing a causal student with pretrained bidirectional weights causes severe training instability or outright collapse, as the architectural divergence between bidirectional attention and causal constraints violates frame-level injectivity along the probability flow.
To stabilize distillation, existing approaches (e.g., CausVid, Self Forcing, Causal Forcing) insert an intermediate trajectory-matching phase, forcing the student to regress multi-step ordinary differential equation (ODE) trajectories simulated by a teacher model before performing distribution matching. Unfortunately, generating and storing high-fidelity ODE trajectories across massive datasets incurs prohibitive computational costs; moreover, cross-architecture ODE imitation violates frame-level injectivity, injecting cumulative errors throughout the generation pipeline. Simultaneously, standard pretraining paradigms suffer from exposure bias: Teacher Forcing trains on pristine ground-truth frames, and Diffusion Forcing operates on Gaussian-corrupted context, whereas real-time inference relies on self-generated histories containing accumulated prediction errors. While Self Forcing tackles this bias via sequential rollout, its lack of parallelizability restricts training throughput.
The core insight of this paper is that distillation collapse arises from the entangled optimization of causal structural alignment and few-step sampling adaptation. By decoupling these two challenges through an error-aware causal pretraining phase, the student can safely bypass forced trajectory matching and directly optimize few-step generation. Core idea: propose Error Forcing, an error-aware and fully parallelizable autoregressive pretraining framework that injects empirical prediction residuals into the historical context to reflect inference-time degradation, establishing a robust causal initialization that enables direct few-step distribution matching distillation (DMD) without intermediate ODE matching.
Method¶
Overall Architecture¶
The framework comprises two consecutive stages: Error Forcing causal pretraining and direct few-step DMD distillation from the causal initialization. In the first stage, a bidirectional foundation model is adapted into a causal architecture; during training, an online FIFO buffer collects realistic prediction residuals from the model's early outputs and injects scaled residuals into the ground-truth conditioning context. This equips the causal network with robustness against history drift while preserving full sequence-parallel training throughput. In the second stage, the resulting causal weights initialize the student generator, which directly performs distribution matching distillation (DMD) with a multi-step teacher and critic under self-rollout, completely bypassing ODE trajectory matching.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Bidirectional Base Model Weights<br/>Wan2.1-T2V-1.3B"] --> B["Residual Error Collection & Empirical Distribution<br/>Convert flow to x0 estimate and record residual ฮด"]
B --> C["Context Error Injection & Causal Pretraining<br/>Perturb context with empirical residuals in parallel"]
C --> D["Robust Causal Model Initialization<br/>Structurally aligned and resilient to rollout drift"]
D --> E["Direct Few-step Distillation from Causal Init<br/>Bypass ODE trajectory matching and execute direct DMD"]
E --> F["4-Step Real-time Autoregressive Video Model"]
Key Designs¶
1. Residual Error Collection & Empirical Distribution: Capturing Structured Prediction Residuals
Prior attempts to alleviate exposure bias commonly injected isotropic Gaussian noise into the history, which fails to mimic real autoregressive inference errors such as local motion blur, texture distortion, and semantic drift. To construct an accurate proxy for inference-time degradation, the authors introduce an online error collection mechanism. During causal pretraining, the model's instantaneous velocity output is integrated via Euler estimation to produce a clean frame prediction \(\hat{x}^i\) (the implied \(x_0\) prediction). The frame residual is computed against the ground-truth frame \(x^i\): $$ \delta^i = \hat{x}^i - x^i $$ All residuals are stored in a fixed-capacity FIFO buffer \(\mathcal{B}\) of size \(V = 3,000\), defining an empirical error distribution \(\mathcal{D}_{\text{err}} \approx \text{Uniform}(\mathcal{B})\). Utilizing a FIFO buffer ensures that sampled residuals continuously reflect the model's most recent prediction errors throughout training, preventing stale, large initialization errors from poisoning later stages.
2. Context Error Injection & Causal Pretraining: Train-Test Consistency with Parallel Throughput
With the empirical error distribution \(\mathcal{D}_{\text{err}}\) established, the model injects structured perturbations into the training context to match the inference distribution without sacrificing parallel computation. During pretraining, a residual sequence \(\delta^{1:N}\) is sampled from \(\mathcal{D}_{\text{err}}\), constructing the perturbed historical context: $$ \tilde{x}^i = x^i + \gamma \delta^i $$ where \(\gamma\) is an error scale sampled randomly from \(\{0, 0.1, 1\}\). The causal flow matching model predicts the velocity field of the \(i\)-th frame conditioned on the perturbed history \(\tilde{x}^{<i}\) rather than pristine ground truth \(x^{<i}\). This encourages the model to learn a robust conditional mapping satisfying \(p(x_t^i \mid \tilde{x}^{<i}) \approx p(x_t^i \mid \hat{x}^{<i})\). Because error injection is implemented by simply adding pre-sampled tensors onto ground-truth sequences, all sequence frames can be processed concurrently under causal attention masks, maintaining the high throughput of standard Teacher Forcing. To prevent instability from large initial errors, a warm-up phase of \(T_{\text{warm}} = 500\) steps collects residuals without injecting them.
3. Direct Few-step Distillation from Causal Init: Decoupling Causal Alignment from Few-step Adaptation
Having obtained robust causal weights via Error Forcing, the student model is already adapted to causal temporal dependencies and resilient to imperfect conditioning histories. This allows the framework to eliminate the computationally expensive and injectivity-violating ODE trajectory-matching phase. During few-step distillation (e.g., 4 steps), the student model generates rollouts via self-rollout and is optimized directly using Distribution Matching Distillation (DMD). The DMD gradient is formulated as: $$ \nabla_\phi \mathcal{L}{\text{DMD}} \approx -\mathbb{E} \right] $$ The authors investigate two conditioning variants: an external bidirectional teacher/critic (Wan2.1-14B) providing global guidance (}^i} \left[ \left( s_T(x_t^i \mid c_T^i, t) - s_F(x_t^i \mid c_S^i, t) \right) \frac{\partial x_t^i}{\partial \phi\(c_T^i = c_S^i = x^{1:N}\)), and an "Internal DMD" configuration where the teacher and critic share the identical causal initialization as the student and evaluate scores under the autoregressive context (\(c_T^i = c_S^i = \hat{x}^{<i}\)). Internal DMD eliminates cross-distribution drift between the student and critic, providing an internally consistent gradient landscape.
Loss & Training¶
During pretraining, the flow matching objective minimizes the mean squared error against the target velocity \(v = \epsilon^i - x^i\): $$ \mathcal{L}{\text{EF}} = \frac{1}{N} \sum}^N \mathbb{E{x, t, \epsilon, \delta} \left[ \left| (\epsilon^i - x^i) - v\theta(x_t^i, t \mid \tilde{x}^{<i}) \right|_2^2 \right] $$ where timesteps follow \(t \sim \text{LogitNormal}(0, 1)\) and noisy interpolations follow \(x_t^i = (1-t) \tilde{x}^i + t \epsilon^i\). After each backward step, generated predictions \(\hat{x}^{1:N}\) are obtained without gradients and residuals are appended to \(\mathcal{B}\). In the distillation stage, the student model is fine-tuned for 2,700 steps using 4-step sampling under the DMD objective across 32 NVIDIA A100 GPUs.
Key Experimental Results¶
Main Results¶
The base model is Wan2.1-T2V-1.3B, generating 81-frame videos at 832ร480 resolution. Quantitative evaluations are conducted on both standard VBench prompts and the challenging TA-Hard prompt set against bidirectional baselines, native autoregressive models, and distilled autoregressive models.
| Model | Params | VBench Total โ | VBench Quality โ | VBench Semantic โ | VisionReward Overall โ | Detail Quality โ | Motion Realism โ | Text Alignment โ |
|---|---|---|---|---|---|---|---|---|
| Wan2.1 (Bidirectional) | 1.3B | 75.53 | 73.86 | 67.17 | 3.94 | 12.26 | -40.49 | 39.18 |
| SkyReels-V2 | 1.3B | 78.48 | 81.37 | 66.93 | 7.71 | 29.32 | -31.87 | 47.57 |
| MAGI-1 | 4.5B | 78.15 | 80.50 | 68.77 | 7.18 | 28.40 | -40.43 | 49.47 |
| CausVid | 1.3B | 79.12 | 81.91 | 68.00 | 10.33 | 40.10 | -24.47 | 56.73 |
| Self Forcing | 1.3B | 80.79 | 83.38 | 70.46 | 11.33 | 41.86 | -10.20 | 64.76 |
| Longlive | 1.3B | 80.25 | 82.94 | 69.48 | 11.35 | 42.71 | -18.71 | 64.27 |
| Causal Forcing | 1.3B | 80.52 | 83.11 | 70.15 | 10.74 | 41.01 | -6.50 | 62.79 |
| Ours (EF + DMD) | 1.3B | 81.35 | 83.92 | 71.03 | 12.00 | 45.74 | -6.98 | 64.83 |
On the demanding TA-Hard benchmark emphasizing complex semantic adherence:
| Model | VBench Total โ | VBench Quality โ | VBench Semantic โ | VisionReward Overall โ | Detail Quality โ | Motion Realism โ | Text Alignment โ |
|---|---|---|---|---|---|---|---|
| Wan2.1 | 64.45 | 75.62 | 19.78 | -6.96 | -1.85 | -86.81 | -31.48 |
| CausVid | 63.09 | 74.84 | 16.09 | 1.20 | 33.33 | -79.17 | 12.96 |
| Self Forcing | 67.69 | 78.98 | 22.56 | 1.43 | 31.48 | -79.86 | 21.30 |
| Causal Forcing | 69.42 | 81.03 | 22.99 | 0.77 | 27.78 | -81.94 | 24.07 |
| Ours (EF + DMD) | 71.22 | 80.96 | 32.24 | 2.70 | 34.26 | -69.44 | 21.30 |
Ablation Study¶
1. Comparison of Pretraining and Distillation Paradigms
Evaluating multi-step (50 steps) generation and few-step (4 steps) distillation under Teacher Forcing (TF), Diffusion Forcing (DF), and Error Forcing (EF):
| Paradigm | Steps | Quality Score โ | Semantic Score โ | Total Score โ | Note |
|---|---|---|---|---|---|
| Diffusion Forcing (DF) | 50 steps | 80.98 | 67.16 | 78.22 | Multi-noise conditioning leads to overly smoothed textures |
| Teacher Forcing (TF) | 50 steps | 81.34 | 68.19 | 79.15 | Clean ground truth leads to exposure bias during rollout |
| Error Forcing (EF) | 50 steps | 81.89 | 68.95 | 79.64 | Injected structured residuals yield highest multi-step quality |
| DF + DMD | 4 steps | 83.13 | 68.39 | 80.19 | 4-step distillation from DF causal init |
| TF + DMD | 4 steps | 83.37 | 70.34 | 80.77 | 4-step distillation from TF causal init |
| EF + DMD (Ours) | 4 steps | 83.92 | 71.03 | 81.35 | Full model: optimal visual quality and semantic alignment |
2. Error Injection Granularity and Strategy Ablations
| Ablation Dimension | Configuration | Quality Score โ | Semantic Score โ | Total Score โ | Note |
|---|---|---|---|---|---|
| Storage Granularity | Frame-wise | 81.89 | 68.95 | 79.64 | Storing residuals per latent frame yields finest adaptation |
| Chunk-wise | 81.31 | 68.92 | 79.08 | Storing residuals across 3-frame chunks | |
| Window-wise | 80.19 | 66.15 | 77.38 | Storing across 21-frame windows smears error specificity | |
| Sampling Strategy | Timestep-aware | 81.48 | 69.05 | 79.32 | Retrieving residuals matching timestep \(t\) |
| Random | 81.89 | 68.95 | 79.64 | Uniform random sampling from FIFO queue offers better diversity | |
| Injection Region | Context only | 79.91 | 67.15 | 77.36 | Injecting residuals solely into past context frames |
| Context + Target (independent) | 80.49 | 61.45 | 77.40 | Independent residual sampling damages temporal coherence | |
| Context + Target (shared) | 81.89 | 68.95 | 79.64 | Shared residual across context and target preserves continuity |
3. Initialization Strategy vs. Direct Distillation Stability
| Initialization Source | Pre-distillation Total | Post-distillation Total (4-step) | Training Stability & Behavior |
|---|---|---|---|
| Bidirectional Pretrain | 55.37 | 76.25 | Direct DMD causes severe collapse due to structural gap |
| Bidirectional + ODE Distillation | 61.36 | 80.79 | Requires costly ODE trajectory matching to achieve stability |
| Error-Aware Causal Pretrain (EF) | 54.14 | 81.35 | Bypasses ODE distillation completely and converges stably |
| Causal Pretrain + Causal ODE | 75.38 | 80.99 | Intermediate ODE matching causes visual artifacts and drops peak score |
| Causal Pretrain + Internal DMD | 78.70 | 81.49 | Homogeneous causal critic; slight gain with doubled training cost |
Key Findings¶
- Robust Causal Initialization Unlocks Direct Distillation: Distilling a causal student directly from bidirectional weights causes optimization divergence and blurred outputs (76.25 VBench total). In contrast, initializing with Error Forcing causal weights enables direct DMD optimization, surpassing conventional ODE-initialized pipelines (81.35 vs. 80.79) while dramatically slashing training overhead.
- Shared Residual Injection Preserves Temporal Manifolds: Injecting shared residuals simultaneously into both context and target latents significantly outperforms injecting independent errors (79.64 vs. 77.40). Preserving cross-frame correlation within injected perturbations prevents unnatural temporal disconnections during training.
- Superior Semantic Adherence on Complex Prompts: On the TA-Hard benchmark, the proposed method achieves a semantic score of 32.24, outperforming Causal Forcing (22.99) and Self Forcing (22.56) by over 40%, proving that exposure-bias mitigation fundamentally prevents semantic drift over extended rollouts.
Highlights & Insights¶
- Decoupling Architecture Alignment from Few-step Distillation: By replacing costly, injectivity-violating ODE trajectory matching with an error-aware pretraining stage, the framework demonstrates that student models only require a robust causal inductive bias to learn distribution matching directly.
- Empirical Residuals vs. Synthetic Noise: Collecting model-generated residuals into an online FIFO buffer provides a realistic, structured error distribution (capturing blur and motion misalignment) that far exceeds Gaussian noise in preparing the model for inference-time exposure bias.
- Internal DMD for Fully Causal Ecosystems: Demonstrating that a causal teacher and critic can supervise a causal student within the identical autoregressive conditioning space establishes an attractive paradigm for self-improving autoregressive foundation models.
Limitations & Future Work¶
- Author-Admitted Limitations: During exceptionally long autoregressive rollouts, if accumulated errors exceed the perturbation envelope established by the static scaling factor \(\gamma\), subtle visual degradations and geometric inconsistencies may still surface.
- Future Directions: Developing an adaptive error scheduling mechanism that dynamically adjusts perturbation magnitudes based on rollout duration, and extending Error Forcing to interactive closed-loop world models conditioned on continuous action streams.
Related Work & Insights¶
- vs. CausVid & Self Forcing: CausVid relies on computationally heavy ODE matching across architectures; Self Forcing applies sequential rollout during pretraining, sacrificing parallelization. Error Forcing injects empirical residuals into ground-truth frames, achieving full training parallelizability while bypassing the ODE stage entirely.
- vs. Causal Forcing: Causal Forcing introduces causal ODE distillation, which still requires simulating and storing trajectories from an AR teacher. This work proves that an error-aware causal pretraining initialization renders ODE trajectory regression redundant, reaching higher quality with lower training expense.
Rating¶
- Novelty: โญโญโญโญโญ Elegant strategy capturing empirical residuals for exposure-bias correction while successfully bypassing ODE trajectory matching.
- Experimental Thoroughness: โญโญโญโญโญ Comprehensive evaluations covering multi-step and few-step regimes, standard and hard benchmarks, and thorough ablations on granularity, region, and initialization.
- Writing Quality: โญโญโญโญโญ Exceptionally clear mathematical formulation and well-structured motivation explaining the failure modes of architectural mismatch.
- Value: โญโญโญโญโญ Provides a practical, computationally efficient blueprint for scaling real-time autoregressive video generation models and interactive world simulators.