End-to-End Training for Autoregressive Video Diffusion via Self-Resampling¶

Conference: ECCV2026
Paper: ECCV Paper
Authors: Yuwei Guo, Ceyuan Yang, Hao He, Yang Zhao, Meng Wei, Zhenheng Yang, Weilin Huang, Dahua Lin
Area: Video Generation
Keywords: autoregressive video diffusion, exposure bias, self-resampling, causal attention, history routing

TL;DR¶

Resampling Forcing trains a causal diffusion predictor on histories containing its own resampled model errors while retaining clean video targets, removing auxiliary teachers and discriminators and improving long-video temporal quality through native 15-second training.

Background & Motivation¶

Autoregressive video diffusion separates generation into temporal progression and denoising within the current frame: each current image can depend only on the generated past. This structure suits interactive content and world simulation, but its training inputs are often much cleaner than those available during deployment. Teacher Forcing supplies real histories during training, whereas inference supplies generated histories, allowing small texture, color, and object-state errors to grow across frames. Adding Gaussian noise to histories simulates one kind of perturbation, but need not reproduce denoising errors or the process by which an earlier error affects the next frame.

Methods such as Self Forcing address this mismatch through autoregressive training rollouts followed by distribution matching on generated outputs. However, a bidirectional teacher or online discriminator adds training components, and supervision from a short-video teacher does not inherently cover long-term dependencies. The paper asks whether direct frame-level diffusion regression can be retained while conditioning on histories containing errors that the current model would make. The goal is not to make a finite-capacity model error-free, but to teach the predictor to recover toward clean targets from imperfect histories instead of repeatedly reproducing their defects.

The authors resample only the latter part of the denoising trajectory of a real video, retaining its content while perturbing it with current model errors. This avoids generating an entire training video from pure noise or requiring another network to define a desirable video distribution. Content-dependent history routing addresses the cost of long histories without truncating memory to a fixed set of recent frames. Core Idea: let the online model autoregressively construct recognizable but imperfect training histories, then predict original clean targets from these detached conditions to learn error robustness directly in a causal video diffusion model.

Method¶

Overall Architecture¶

The inputs are a clean training video and condition \(c\); the backbone uses the Wan2.1-1.3B architecture with causal attention and predicts the velocity field for the current autoregressive unit. For simplicity, the paper also calls a generated chunk a “frame”; each autoregressive step actually generates 3 latent frames, so not every frame count in the method denotes pixel-space video frames. Training first autoregressively resamples histories with online weights, then uses the resampled video as context while independently noising the original video and computing parallel frame-level diffusion losses. Optional history routing selects historical context inside attention without changing the principle that prediction targets come from the original clean video.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    SOURCE["Clean video + condition"] --> RESAMPLE["Autoregressive Self-Resampling"]
    SOURCE -->|Independently noised current chunks| CAUSAL["Parallel Causal Learning"]
    RESAMPLE -->|Detached histories| CAUSAL
    CAUSAL -->|Optional attention implementation| ROUTE["History Routing"]
    CAUSAL -->|Dense attention| OUTPUT["Velocity predictions"]
    ROUTE --> OUTPUT
    OUTPUT --> LOSS["Frame-level diffusion loss"]
    SOURCE -.->|Clean targets and sampled noise| LOSS

The diagram shows training data flow; loss gradients do not pass backward through the self-resampling branch. At inference, there is no real video to resample and no training loss: the model generates the current chunk from noise, adds it to history, and proceeds to the next chunk. The causal mask and optional history routing remain active, but the training history-perturbation procedure is not an additional inference module.

Key Designs¶

1. Autoregressive Self-Resampling: construct history errors resembling current inference errors

The method first noises a real chunk to an intermediate time \(t_s\), then uses the online model to complete denoising from that time to the clean endpoint. Starting near real data usually retains the video's semantics and layout, while completing the remaining trajectory with the model introduces its own generation errors. The authors particularly target high-frequency detail errors in later denoising rather than treating every error as independent pixelwise Gaussian noise. Chunks are not resampled independently in parallel: each later chunk conditions on previously resampled histories, exposing errors propagated across chunks as well. Online weights ensure that the error distribution evolves with model capability instead of keeping training centered on a fixed older model's weaknesses. The whole procedure is detached to prevent shortcuts through the condition-construction process; its resulting histories serve only as inputs to the subsequent learning stage.

Resampling strength controls a concrete tension: overly weak perturbations leave histories nearly perfect, resembling Teacher Forcing; overly strong perturbations cause content drift and weaken correspondence with clean targets. The authors sample an initial time from LogitNormal(0, 1), then shift it, using \(s=0.6\) in the experiments. The following expression rewrites Algorithm 1 with separate notation before and after shifting, rather than copying the extraction-damaged Eq. (7):

\[ u\sim\operatorname{LogitNormal}(0,1),\qquad t_s=\frac{s u}{1+(s-1)u}. \]

\(s<1\) emphasizes lower noise levels; stronger models can generally use weaker perturbations because their own errors are already smaller. The paper qualitatively compares \(s=0.1,0.6,3.0\): very weak perturbations allow error accumulation, very strong perturbations increase content drift, and an intermediate strength works better. Generated histories are not treated as correct answers; inputs are deliberately imperfect while a separate clean supervision path is preserved.

2. Parallel Causal Learning: separate degraded conditions from clean targets

After resampling histories, the method samples a training noise time for each original video chunk to construct the noisy prediction inputs. These noisy chunks and the resampled histories are distinct sequences: the former define the regression tasks, while the latter provide temporal conditions. A sparse causal mask permits interactions within the current chunk, but temporal context comes only from completed past chunks, excluding future histories and the current chunk's clean answer. Consequently, supervised predictions for all chunks can run in one parallel forward pass without also making gradient computation follow an entire sequential rollout. “Clean history” here denotes a denoised endpoint representation, not exact ground-truth data; the resampled history is explicitly a denoised result containing model errors. Corrupting conditions without corrupting supervision teaches tolerance to historical perturbations rather than turning those errors into new targets.

This separation explains why the method remains direct diffusion regression rather than video-level distillation or adversarial training. Self-resampling exposes the input distribution the model may encounter, and causal prediction maps those inputs back toward the next state associated with the original data. “End-to-end” refers here to training the target model without auxiliary teachers or discriminators, not to differentiating through every history-generation step. Training also requires Teacher Forcing warmup: without basic generation ability, early resampled histories may contain only uninformative perturbations. The paper emphasizes support for training from scratch; its experiments use the Wan2.1-1.3B architecture, but this alone does not establish that every component, weight, and dataset was built from zero.

3. History Routing: select relevant near and distant histories instead of truncating memory

Historical KV representations grow with video length, forcing dense causal attention to read increasingly many historical tokens. The router mean-pools each historical chunk's keys into a descriptor, then selects the top-\(k\) historical chunks using dot products between the current query and those descriptors. Fine-grained attention subsequently reads keys and values from the selected chunks, rather than attending only to their pooled descriptors. Routing adds no learnable parameters and makes selections separately for each attention head and query token, not once for the entire frame. Different spatial locations and heads can retrieve different past content, yielding a broad collective receptive field even when each query selects few histories. The frequency analysis shows a preference for initial and recent frames, with intermediate histories increasingly included as \(k\) grows.

Implementation separates current-chunk attention from sparse historical attention and merges them through globally aligned log-sum-exp terms. This is equivalent to a single softmax over the union of both sets of keys, rather than averaging two independently normalized outputs. With \(L\) historical chunks and \(k\) selected chunks, historical attention sparsity is \(1-k/L\); selecting 5 out of 20 gives 75%. The stated reduction from \(O(L)\) to \(O(k)\) concerns fine-grained attention over selected histories, not a guarantee of constant total runtime or KV storage. Descriptor scoring still processes candidate histories, and the paper does not provide complete long-duration throughput or memory curves proving fixed costs at unlimited lengths. Routing therefore answers which histories to read, not automatically which histories no longer need storage.

A Worked Example¶

Consider a training video of milk being poured continuously; the following uses 3 consecutive generation chunks to explain computation, not as an additional experiment. The first real chunk is lightly noised and denoised into a resampled version with a small texture error. Resampling the second chunk conditions on this version, and the third conditions on both earlier versions, so the resulting history includes temporally propagated model errors. Next, all 3 original chunks are independently noised and their velocity fields predicted in parallel; the third prediction reads the first two resampled histories, not future information. Its target still comes from the real third chunk and its sampled noise, so historical defects need not be carried unchanged into the next chunk. Actual inference instead generates and caches chunks sequentially from noise, without real video or a constructed resampling starting point. Once history contains 20 chunks with \(k=5\), different queries can retrieve the initial container appearance and recent liquid state, rather than being restricted to the latest 5 chunks. Such memory selection can support consistency, but cannot by itself guarantee liquid conservation or other physical laws.

Loss & Training¶

Algorithm 1 uses mean-squared Flow Matching velocity regression; the expression below follows its intact pseudocode to avoid ambiguity in the damaged equation extraction. Here \(x^i\) is an original clean chunk, \(\tilde{x}^{<i}\) is detached resampled history, \(\epsilon^i\) is training noise, and \(t_i\) is a per-chunk sampled time.

\[ x^i_{t_i}=(1-t_i)x^i+t_i\epsilon^i,\qquad \mathcal{L}=\mathbb{E}\left[\left\|v_\theta(x^i_{t_i},\tilde{x}^{<i},t_i,c)-(\epsilon^i-x^i)\right\|_2^2\right]. \]

The perturbed part is the conditioning history; the regression target \(\epsilon^i-x^i\) does not replace real chunks with resampled ones. Training begins with 10K Teacher Forcing steps on 5-second videos, followed by 15K and 5K Resampling Forcing steps on 5-second and 15-second videos, respectively. The 15-second training videos contain 249 pixel-space frames; history routing is then enabled for another 1.5K fine-tuning iterations on 15-second videos. The optimizer is AdamW with learning rate \(5\times10^{-5}\) and batch size 64; history routing uses \(k=5\). Training history resampling uses just 1 Euler step to obtain model-specific conditioning errors cheaply, rather than to produce a fully sampled high-quality video. Inference uses 32 Euler steps, a timestep shifting factor of 5.0, and classifier-free guidance scale 5.0, identically across frames. The training resampling setting \(s=0.6\) and inference shift 5.0 belong to different procedures; likewise, 1-step history resampling does not mean 1-step video generation.

Key Experimental Results¶

Main Results¶

Table 1 of the paper, on page 12, evaluates generated 15-second videos separately over 0–5, 5–10, and 10–15 seconds. The selection below retains the initial and final intervals for major 1.3B-parameter comparisons; Temp. denotes temporal quality, Visual denotes visual quality, and Text denotes text alignment, with higher values better as reported. The paper references VBench but does not specify the full component metrics and weights of these aggregate columns in this table, so they should not be called overall VBench scores or physical-correctness rates.

Method	Teacher	0–5 s Temp.	0–5 s Visual	0–5 s Text	10–15 s Temp.	10–15 s Visual	10–15 s Text
CausVid	Wan-14B (5 s)	89.35	65.80	23.95	87.14	64.90	22.81
Self Forcing	Wan-14B (5 s)	90.03	67.12	25.02	84.26	63.04	24.29
LongLive	Wan-14B (5 s)	81.84	66.56	24.41	84.57	67.17	24.44
Ours, 75% sparse	None	90.18	63.95	24.12	87.03	61.01	23.35
Ours, dense	None	91.20	64.72	25.79	89.74	63.99	24.39

The dense model exceeds Self Forcing by 5.48 points in final-interval temporal quality, but its final visual quality remains below LongLive's 67.17, ruling out blanket superiority across dimensions. Its temporal quality changes from 91.20 to 89.74, showing that degradation still occurs, although less than for some comparisons. Training durations and teacher conditions differ across methods; this is a full-system comparison rather than a single-variable experiment controlling only the training loss.

Ablation Study¶

Table 2, on page 13, compares error simulation strategies over the full 0–15-second interval; these values should not be compared cell by cell with the segmented intervals above.

History error simulation strategy	Temp.	Visual	Text
Gaussian noise augmentation	87.15	61.90	21.44
Parallel resampling	88.01	62.51	24.51
Autoregressive resampling	90.46	64.25	25.26

Autoregressive resampling improves temporal quality by 2.45 points and visual quality by 1.74 points over parallel resampling, supporting the need to simulate temporally propagated errors. Compared with Gaussian augmentation, text alignment increases from 21.44 to 25.26, suggesting that the perturbation distribution also affects content retention rather than only local image quality. The table reports no repeated-run variance or confidence intervals; these differences are point estimates, not statistical-significance conclusions.

Key Findings¶

In Table 1, 75% sparsity reduces final-interval visual quality from 63.99 for the dense model to 61.01, a 2.98-point cost; although the authors characterize the loss as small, a quality–sparsity trade-off remains.
Figure 7 on page 13 supports intermediate resampling strength qualitatively, without a complete numerical score curve for different \(s\) values.
Figures 8–9 on page 14 show advantages over equally sparse local windows and a preference for initial and recent frames; they do not quantify minute-scale stability.

Highlights & Insights¶

Separate condition quality from target quality. Training need not demand both perfect inputs and correct outputs; targeted input corruption can make frame-level supervision better reflect deployment.
Include temporal dependence in error simulation. Parallel resampling is already closer to model errors than Gaussian noise, but autoregressive resampling also models how errors propagate, making this ablation particularly informative.
Retrieve histories by content. Routing does not prescribe an initial-plus-recent memory rule; that pattern emerges from query–history matching, offering a basis for more adaptive video memory research.

Limitations & Future Work¶

Causal attention is not physical causal correctness. The milk-pouring example supports more plausible temporal evolution, but does not establish that distillation necessarily causes physical errors or that this method guarantees conservation laws.
Long-horizon evidence is limited. Core quantitative results cover 15 seconds; effective native long-video training does not imply drift-free generation at unlimited lengths.
Efficiency lacks system-level accounting. Resampling adds training forward passes and routing incurs selection costs; the examined text provides no complete training-compute, throughput, or memory comparison.
Reproduction details remain incomplete. The text supplies major optimization settings, but the examined experimental section does not clearly specify dataset scale, full evaluation aggregation details, or every initialization source.
A testable next step. Compare direct training and distillation under matched backbones, data, and duration budgets, then report longer-horizon drift curves, routing retrieval costs, and physical-state errors.

vs Self Forcing / LongLive: both aim to reduce the train–test gap; this method creates conditioning errors with the current model and regresses clean targets directly, whereas the comparisons use teacher distribution matching, so benefits and costs should be assessed alongside training budgets.
vs Diffusion Forcing: the latter supports different noise levels across frames; this method also uses per-frame noise, but its key addition is autoregressively generated error histories rather than only a different timestep distribution.
vs Scheduled Sampling / Self-Conditioning: all use the model's predictions; this method specifically starts from intermediate denoising states of real videos to retain content correspondence while simulating inter-frame errors.
vs MoBA / MoC / VMoBA: the method adapts parameter-free block routing to causal video histories; a transferable research question is how to select memory for reading rather than merely expanding context length.

This note checks the assigned text's title, authors, method, Algorithm 1, and Tables 1–2; some extracted display equations are damaged, so the two formulas above follow intact pseudocode rather than guessing missing characters in the original equations. The official paper identity is ECCV2026 poster 4177; no unconfirmed arXiv or code links are supplied.

Rating¶

Novelty: 4/5. Autoregressive conditioning-error simulation and direct diffusion supervision form a clear combination; history routing largely builds on existing sparse-attention ideas.
Experimental Thoroughness: 3/5. Multiple baselines, simulation ablations, and routing analyses are included, but longer horizons, system costs, and statistical stability remain insufficiently evaluated.
Writing Quality: 4/5. The workflow and pseudocode are clear, while claims about causality and negligible losses require cautious interpretation against the evidence.
Value: 4/5. A reusable approach to teacher-independent autoregressive video training, not a solution to physical consistency in general-purpose world models.