HiAR: Efficient Autoregressive Long Video Generation via Hierarchical Denoising

Conference: ECCV2026
Paper: ECCV Paper
Project: HiAR
Area: Video Generation
Keywords: autoregressive diffusion, hierarchical denoising, noisy context, distribution matching distillation, motion diversity

TL;DR

HiAR conditions each video block on earlier blocks at the current denoising step's output level, rather than fully clean predictions, and uses bidirectional teacher-trajectory regularization to prevent motion collapse, achieving a VBench Total of 0.821, Drift of 0.257, and reported throughput of 30 fps on 20-second videos.

Background & Motivation

Autoregressive video diffusion partitions a long video into blocks and extends the sequence using previously generated context, avoiding bidirectional attention over the entire video. Self-Forcing additionally trains with student-generated context to reduce exposure bias between real and generated histories. However, using generated history during both training and inference does not remove its errors: if an earlier block becomes oversaturated, oversharpened, or semantically distorted, the next block can inherit these mistakes as reliable conditioning.

The issue centers on finishing one block completely before starting the next. While the current block is still highly noisy, its history is already a fully denoised, definite prediction. HiAR draws on fixed-window bidirectional diffusion, where frames can develop coherent motion even before they become clean. The authors therefore ask how much cleaner the history needs to be to support causal continuation without transmitting prediction errors at maximum confidence.

Changing the generation order requires retraining the model to continue from noisy histories; longer self-rollout training then exposes the tendency of reverse-KL distillation to favor low-motion outputs. Core Idea: keep history only as clean as the current step's output noise level, reduce error propagation through hierarchical denoising, and preserve motion diversity with bidirectional teacher-trajectory regularization instead of obtaining low drift through nearly static videos.

Method

Overall Architecture

The inputs are text conditioning and initial noise for each video block; the output is a video that can be extended block by block. Matched-Noise Context determines the history noise level, and Hierarchical Denoising implements the corresponding causal order. Training adds Bidirectional Trajectory Regularization, while deployment uses Pipelined Parallelism to execute the same dependency graph.

Here, matching means that history uses the current step's output noise level, not its input noise level. The diagram follows design construction, training, and deployment; the regularization branch is training-only and does not introduce teacher calls during online generation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Text conditioning and block noise"] --> B["Matched-Noise Context"]
    B --> C["Hierarchical Denoising"]
    C -->|Training| D["Bidirectional Trajectory Regularization"]
    T["Teacher denoising trajectories"] --> D
    D -->|Deployment after training| E["Pipelined Parallelism"]
    E --> F["Blockwise long-video output"]

Key Designs

1. Matched-Noise Context

Suppose the preceding block's prediction equals its true clean latent plus accumulated error. When represented as noisy context, useful signal and prediction error are multiplied by the same signal coefficient, with random noise added separately. Increasing noise therefore does not selectively remove mistakes: it attenuates both errors and useful information. Fully clean history preserves all prediction error, whereas excessively noisy history may fail to constrain subsequent motion. This is the concrete basis for selecting the noise level, rather than a generic claim that noise improves robustness.

The paper assumes that history must have at least the signal-to-noise ratio of the current block after the step to provide sufficient continuation information. Under this constraint, choosing the largest admissible noise level reduces the error term, giving the boundary choice below. Here, \(t_j\) is the step's input level, \(t_{j+1}\) its output level, and \(t_c\) the history level. This conclusion depends on the authors' SNR-sufficiency assumption; it is not a universal optimality proof for arbitrary video distributions.

\[ \operatorname{SNR}(t_c)\geq\operatorname{SNR}(t_{j+1}) \quad\Longrightarrow\quad t_c\leq t_{j+1}, \qquad t_c^*=t_{j+1}. \]

2. Hierarchical Denoising

Conventional execution completes every denoising step for block 1 before doing the same for block 2. HiAR exchanges the two loops: it processes blocks in temporal order within the first denoising step, then advances to the next step. When processing block \(n\), earlier blocks have already completed the current step and can supply states at \(t_{j+1}\) without becoming fully clean first. The important distinction is reusing actual intermediate states at the corresponding level, rather than arbitrarily adding noise back to completed clean history.

Algorithm 1 can be written as the update below, where \(x\) denotes video latents, \(v_\theta\) is the velocity prediction network, and \(\sigma_t\) is the noise-schedule coefficient. The condition \(x_{t_{j+1}}^{(<n)}\) preserves the dependency on earlier blocks having completed this step, so all blocks within a level cannot be generated independently. A KV cache stores the relevant history representations, and inference uses a fixed-duration sliding window to bound the retained history.

\[ x_{t_{j+1}}^{(n)}=x_{t_j}^{(n)}+ v_\theta\!\left(x_{t_j}^{(n)},t_j\mid x_{t_{j+1}}^{(<n)}\right) \left(\sigma_{t_{j+1}}-\sigma_{t_j}\right). \]

3. Bidirectional Trajectory Regularization

Hierarchical denoising requires retraining: otherwise, the old model is unfamiliar with these multi-level noisy histories and can lose continuity while reducing drift. Training retains causal self-rollout and Distribution Matching Distillation (DMD), but its reverse-KL-direction objective alone may favor low-motion modes that are easier to denoise and extend. The authors call this the low-motion shortcut. It is an observed failure of mode coverage during training, not a claim that the global optimum of reverse KL must be a static video.

To counteract this, dense teacher ODE trajectories provide samples, and adjacent checkpoints aligned with the student's schedule supervise single-step predictions, encouraging coverage of the teacher's motion. The paper interprets this as forward-KL-direction regularization; its implementation is trajectory distillation, not direct evaluation of an exact KL over full distributions. The regularizer is applied only in bidirectional-attention mode and only at the first denoising step, leaving the causal self-rollout DMD path unchanged. Shared parameters transfer the motion constraint to causal generation at deployment. Across training checkpoints without regularization, motion scores in the two modes have Pearson \(r=0.968\), providing empirical support for this proxy constraint rather than proving zero gradient interference.

4. Pipelined Parallelism

Step-first scanning does not require completing a level for the entire video before starting the next level. Arrange block indices and denoising-step indices into a grid: each position depends on the previous step for the same block and the current-step states of earlier blocks. Positions on the same anti-diagonal can therefore run concurrently. The implementation assigns a dedicated process to each denoising step, exchanges latents through asynchronous point-to-point communication, and traverses \(N+S-1\) anti-diagonals, where \(N\) is the block count and \(S\) the denoising-step count.

The authors also fuse writing the previous block's context KV and denoising the next block into one forward call. They concatenate the two latent segments along the frame dimension, assign timesteps \(t_{j+1}\) and \(t_j\) respectively, and use causal attention so the second segment reads freshly written history. Fewer separate calls and pipelining yield a reported approximately 1.8-fold wall-clock speedup in the 4-step setting. The text gives \(N+2\) calls per stage, but its listed components, one initial call, \(N-1\) fused calls, and one final call, sum to \(N+1\). This counting inconsistency needs implementation verification and should not support a precise compute-saving claim.

A Worked Example

Consider a text prompt describing continuous ocean waves, 3 video blocks, and 4 denoising steps. These numbers illustrate scheduling and are not an additional experiment.

1. The first level processes block 1 from \(t_1\) to \(t_2\). Its state is still noisy but already contains coarse scene structure.
2. When block 2 advances from \(t_1\) to \(t_2\), it reads block 1 at \(t_2\); block 3 likewise reads earlier blocks that have completed this step.
3. The next level advances blocks from \(t_2\) to \(t_3\), making history progressively cleaner within that level. This repeats through the final level instead of fixing all wave textures in block 1 first.
4. Pipelining overlaps the next level of block 1 with the current level of block 2. Bidirectional Trajectory Regularization has already constrained motion during training and is absent from this inference run.

The result preserves temporal dependencies between neighboring blocks without forcing every later block, while still highly noisy, to obey a fully committed and potentially erroneous historical image.

Loss & Training

The overall objective retains DMD distillation and adds trajectory regularization. By default, only the first \(K=1\) of \(S=4\) denoising steps is constrained.

\[ \mathcal{L}=\mathcal{L}_{\mathrm{DMD}}+ \lambda\mathcal{L}_{\mathrm{FKL}},\qquad \lambda=0.1. \]

The student backbone is Wan2.1-1.3B, initially fine-tuned with causal attention on 16k ODE solution pairs sampled from the base model. The teacher used for the DMD critic is Wan2.1-14B, whereas forward-regularization trajectories come from Wan2.1-1.3B: 20k trajectories with 50 ODE steps each. These are distinct teacher sources and should not be conflated.

Each chunk contains 3 latent frames, and the critic-to-generator update ratio is 5:1. Training uses 5-second clips, a learning rate of \(2\times10^{-6}\), total batch size 64, and 20k steps. Inference fixes the sliding KV attention window at 5 seconds. A bounded history window enables continued output but does not preserve all memories of an arbitrarily long video.

Key Experimental Results

Main Results

The following excerpt from Table 1 includes distilled autoregressive models generating 20-second videos. VBench scores use a 0–1 scale, throughput is in fps, and latency is in seconds per chunk. Models share the Wan2.1-1.3B backbone and 4-step schedule, but the experimental setup does not clearly specify enough hardware detail to verify resource fairness for pipelining.

Method	Throughput	Latency	Total	Quality	Semantic	Dynamic	Drift ↓
CausVid	17	0.69	0.764	0.771	0.740	0.621	0.842
Self-Forcing	17	0.69	0.805	0.829	0.708	0.542	0.355
Causal Forcing	17	0.69	0.810	0.837	0.701	0.672	0.615
HiAR	30	0.30	0.821	0.846	0.723	0.686	0.257

Drift is constructed by dividing each 20-second video into 5 equal temporal segments and calculating MUSIQ, CLIP-IQA, consecutive-frame DINOv2 cosine similarity, consecutive-frame LPIPS distance, mean HSV saturation, and Laplacian variance. A line is fitted to each statistic over time, its slope measures change, and the slopes are normalized and combined through a weighted sum; lower values indicate less drift. The paper does not specify exact weights or complete normalization details, so no fully reproducible formula is invented here.

The full source table also includes bidirectional models, but the evaluation setup explicitly states a 5-second generation length for them. They should not be treated as strictly equal-duration comparisons with 20-second autoregressive results; the table above therefore focuses on the more directly comparable distilled autoregressive group.

Ablation Study

Table 2 examines context noise. Unless otherwise stated, variants are retrained with their inference schedule. Smooth is VBench motion smoothness, not motion magnitude or diversity.

Context setting	Quality ↑	Semantic ↑	Smooth ↑	Drift ↓
\(t_c=t_j\), input noise level	0.799	0.692	0.978	0.184
\(t_c=t_{j+1}\), output noise level	0.846	0.723	0.988	0.257
\(t_c=0\), Self-Forcing	0.829	0.708	0.991	0.355

Table 3 examines regularization and the training path. The following configurations capture the central trade-offs.

Config	Quality ↑	Semantic ↑	Dynamic ↑	Drift ↓
Bidirectional attention + 1 step, default	0.846	0.723	0.686	0.257
Causal attention + 1 step	0.828	0.701	0.625	0.271
Bidirectional attention + 2 steps	0.835	0.708	0.693	0.296
Bidirectional attention + 4 steps	0.813	0.684	0.691	0.306
Without forward regularization	0.839	0.732	0.445	0.218
Without retraining	0.767	0.559	0.512	0.309

Key Findings

• Relative to Self-Forcing, Drift falls from 0.355 to 0.257, a reported relative reduction of 27.6%. Total rises from 0.805 to 0.821, but not every individual metric is best.
• Input-level noise produces lower Drift of 0.184 while sacrificing quality and smoothness. Low drift alone does not establish continuity or rich motion.
• Removing regularization reduces Dynamic from 0.686 to 0.445 while improving Drift to 0.218, directly illustrating the evaluation trap of stability through reduced motion.
• Constraining more steps slightly helps Dynamic but lowers Quality and raises Drift; constraining the earliest step provides the main benefit.

Highlights & Insights

• The reliability of historical context is itself a generation-schedule variable. The goal is not to make history clean as early as possible, but to provide sufficient causal information at each step without prematurely committing to errors.
• One dependency graph explains both quality and speed benefits. Exchanging the nesting of temporal blocks and denoising steps creates suitable historical states and exposes parallel anti-diagonals.
• Regularizing a bidirectional mode while deploying a causal mode demonstrates the value of constraints across execution modes with shared parameters. The transferable idea is to choose a training path better suited to teacher supervision, while revalidating correlation between modes.

Limitations & Future Work

• Main evidence covers 20-second generation with a 5-second attention window. Theoretical continued extension does not establish minute-scale identity retention, narrative memory, or long-term physical consistency.
• The matched-noise argument uses SNR as a proxy for sufficient continuation information. Targeted tests are needed for occlusion, fast motion, and long-range dependencies.
• Drift weights and normalization are underspecified, and more static output can score better. Motion, semantics, and longer-horizon quality should be evaluated jointly instead of optimizing drift alone.
• Dedicated pipeline processes, communication overhead, and hardware resources need verification. Approximately 1.8-fold is a reported implementation-level wall-clock benefit, not an established single-GPU or equal-total-compute speedup. The call-count expression also has the inconsistency noted above.
• Some equations in the cached text have extraction damage. This note only reconstructs relationships supported by adjacent prose and the algorithm, without inventing the norm used in trajectory regularization. Reproduction should verify the original typesetting and implementation.

Related Work & Insights

• vs Self-Forcing: Both use student self-rollout to reduce exposure bias. HiAR additionally changes history noise and execution order and addresses motion collapse during distillation, making it more than a sampler substitution.
• vs Diffusion Forcing / AR-Diffusion / PA-VDM: These methods already explore heterogeneous or increasing context noise. HiAR specifically targets output-level noise matching and corresponding retraining in DMD-distilled autoregressive generation; noisy history alone is not a new concept.
• vs FIFO-Diffusion: FIFO-Diffusion converts a pretrained bidirectional model into a training-free streaming generator. HiAR's ablations show that changing only the schedule still harms quality and that train-test alignment matters.
• Research direction: Under a fixed hardware budget, jointly evaluate noise level, history-window size, and motion constraints to locate the boundary between attenuating errors and losing important history. This is a proposed extension, not a result established by this paper.

Rating

• Novelty: 4/5. Output-level noise matching, scheduling changes, and cross-attention-mode regularization form a distinct combination, though noisy history has precedents.
• Experimental Thoroughness: 3/5. Core design ablations are informative, but longer-horizon evaluation, the Drift definition, and hardware comparability remain incomplete.
• Writing Quality: 4/5. The problem-to-mechanism connection is clear, while call counting and evaluation-duration descriptions need more precision.
• Value: 4/5. The work offers useful ideas for stable quality, motion preservation, and pipelined execution in few-step autoregressive video generation.