AnyFlow: Any-Step Video Diffusion Model with On-Policy Flow Map Distillation

Conference: ECCV2026
Paper: Official paper page · PDF
Code: https://github.com/NVLabs/AnyFlow
Area: Video Generation
Keywords: Flow maps, on-policy distillation, distribution matching, any-step sampling, causal video generation

TL;DR

AnyFlow replaces endpoint-only video distillation with transitions across arbitrary time intervals, then corrects student rollouts using three-segment shortcut simulation and distribution matching, reaching 84.05 VBench at 4 NFEs and 84.41 at 32 NFEs with its 14B causal model.

Background & Motivation

Video generation does not have a single natural inference budget. A quick preview should be inexpensive, while a final render may justify more computation. Conventional diffusion models can refine their integration trajectory through additional small steps, but are costly to run. Consistency distillation instead learns to jump from a noisy state to a clean endpoint. This is effective for few-step generation, yet additional sampling steps do not necessarily improve the result. The paper observes this limitation in consistency-based approaches such as rCM and Self-Forcing.

The proposed explanation concerns the trajectory, not simply model capacity. A consistency sampler predicts a clean endpoint, adds noise again, and repeats. That process differs from continuing along the original probability-flow ordinary differential equation (PF-ODE). Additional computation can therefore compound trajectory bias instead of refining the same flow. Causal video generation faces another mismatch: later chunks depend on imperfect history produced by the student itself. Errors in that history accumulate through exposure bias. Learning a longer denoising jump alone does not automatically solve either problem.

Flow maps provide a useful interface because they can describe long transitions while retaining a local velocity field over short intervals. Making that interface work for video post-training, however, requires both a stable conversion from a pretrained backbone and supervision on student-generated states. Core idea: first learn transitions between arbitrary times, then use a differentiable chain consisting of a prefix shortcut, one target transition, and a suffix shortcut to propagate teacher distribution supervision into transitions at different step sizes.

Method

Overall Architecture

The generator takes text conditioning and video-latent noise and produces a video. Built on Wan2.1, AnyFlow uses the same two-stage training structure for bidirectional and causal architectures. Forward training on teacher-generated videos first converts the backbone into a two-time flow map. The second stage retains the forward objective while adding on-policy distribution matching on student-generated trajectories. The causal setting must address errors in previously generated history as well as integration errors within generation.

The diagram separates the forward stage into Two-Time Flow Maps and Stable Forward Initialization, and the on-policy stage into Flow Map Backward Simulation and Joint Distribution Matching. At inference, only the trained student and an Euler scheduler are needed; neither the teacher nor the distribution-matching score networks are part of deployment.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Pretrained backbone<br/>Teacher-generated videos"] --> B["Two-Time Flow Maps"]
    B --> C["Stable Forward Initialization"]
    C --> D["Flow Map Backward Simulation<br/>Prefix shortcut → Target step → Suffix shortcut"]
    D --> E["Joint Distribution Matching"]
    T["Frozen teacher<br/>Student-distribution score network"] --> E
    C -->|Retain forward objective| E
    E -->|Update student| D
    E --> F["Student and Euler scheduler<br/>Generate at the selected budget"]

Key Designs

1. Two-Time Flow Maps: replace the fixed endpoint with a selectable target time

A velocity-field model describes how the current state should move locally, whereas a consistency model primarily describes how to reach the clean endpoint. AnyFlow learns a transition from the current time \(t\) to a target time \(r\). Larger times represent noisier states, so generation proceeds toward smaller times. The network predicts the average transport velocity across an interval rather than only an instantaneous velocity. Using the paper's MeanFlow parameterization:

\[ f_\theta(z_t,t,r)=z_t-(t-r)u_\theta(z_t,r,t). \]

Here, \(u_\theta\) is the average transport velocity, while \(f_\theta\) returns the transported latent state. Setting \(r=0\) gives an endpoint prediction. As \(r\) approaches \(t\), the average velocity should approach the local instantaneous velocity. At exactly \(r=t\), the state map itself is the identity; training at this boundary constrains the underlying velocity prediction. The identity map is not a generator by itself. Its importance is that the parameterization can preserve local flow information while also learning large transport steps.

This makes the inference budget a choice of time intervals rather than a commitment to one endpoint predictor. A few-step schedule uses large intervals, and a higher-budget schedule uses smaller ones. However, accepting arbitrary time pairs is only a representational capability. It does not establish that every interval has been learned accurately, which is why the initialization recipe and on-policy correction are essential.

2. Stable Forward Initialization: preserve the existing velocity field while learning long transitions

Adding a second time embedding to a pretrained video backbone is not automatically stable. The authors find that adding a new branch to the original embedding can produce excessively large embedding norms and oversaturated outputs. They instead interpolate the original-time and target-time embeddings, initialize the new branch from the pretrained one, and fix the original-time branch's coefficient to 0.25. At initialization, equal input and target times recover the pretrained conditioning. This coefficient controls embedding interpolation; it is not the classifier-free guidance scale.

Forward training uses the MeanFlow average-velocity regression objective. The stopped-gradient target is formed from the instantaneous velocity minus the interval length multiplied by the along-trajectory derivative of the average velocity. Computing that derivative exactly requires a Jacobian-vector product, which is difficult to scale under Fully Sharded Data Parallel (FSDP). AnyFlow follows the central finite-difference approach, approximating it using forward evaluations at two neighboring times. The latent states are also perturbed along the instantaneous velocity, so this is not a difference obtained by holding the latent fixed and changing only its time label.

The remaining training choices protect the pretrained field. Non-boundary time pairs are sampled uniformly and reordered, with an additional noise-time weighting. Half of each batch uses \(t=r\), and the average boundary regression loss provides the reference scale for losses on nonzero intervals. This prevents difficult long-interval regression from overwhelming the already useful local field. Classifier-free guidance is fused into the prediction: the conditional prediction is adjusted using a stopped-gradient unconditional prediction and normalized by the guidance scale. The deployed student thus does not need separate conditional and unconditional evaluations at every step. Several equations are damaged in the cached text extraction, so no uncertain weighting expression is reconstructed here.

3. Flow Map Backward Simulation: cover different step sizes through a three-segment chain

Forward training uses noisy states constructed from teacher-generated data, whereas inference uses states produced by earlier student transitions. Fully unrolling the student for every possible sampling budget would make training increasingly expensive. Consistency approaches often truncate gradients to reduce memory requirements, but still follow the endpoint-prediction-and-re-noising trajectory. AnyFlow instead exploits the approximate composition property of flow maps: a learned long transition can approximate the composition of shorter transitions, allowing parts of a trajectory to be skipped during simulation.

For a selected budget \(N\), training chooses one target interval with width \(T/N\). A single flow-map shortcut carries the initial noise to that interval's starting state. The student then executes the selected interval, and a second shortcut takes its output to the clean endpoint. The complete chain is \(T\to t\to r\to0\), with the middle segment representing the actual step size being trained. The prefix and suffix replace longer portions of the rollout. This is an approximation based on learned transitions, not a claim that three evaluations exactly reproduce every intermediate state of a full Euler rollout.

Endpoint supervision is backpropagated through all three segments rather than stopped before the target step. Varying the budget and target interval exposes the model to different step sizes without increasing the number of simulation segments. The constant-cost claim applies to this trajectory construction, not to all score-network computations or to arbitrarily long causal histories. This design is the main mechanism that turns the two-time interface into a practical way to train for multiple inference budgets.

4. Joint Distribution Matching: correct the distribution of the student's own endpoints

After generating an endpoint through the shortcut chain, DMD adds noise to it and compares the teacher's real-distribution score with a fake score estimating the student's generated distribution. Their difference supplies a reverse-KL-style distillation gradient. The teacher remains frozen, the fake score network represents the student distribution, and the generator receives supervision through the whole differentiable shortcut chain. This is distribution-level correction, not a requirement that each student video reproduce a paired teacher video pixel by pixel. The fake score is also not a scalar aesthetic reward.

Two uses of noise must be kept separate. The paper criticizes the repeated endpoint projection and re-noising inside consistency sampling. AnyFlow still re-noises a final student sample to estimate a DMD training gradient. That supervisory operation does not turn its inference procedure back into consistency sampling. Stage two also retains the forward flow-map objective, so the student continues learning interval transitions while improving its endpoint distribution. The main text does not fully specify the objective mixture weights or fake-score optimization details; those missing settings should not be replaced with an invented reproducible recipe.

A Worked Example

Consider generating a video for one text prompt; this is a conceptual walkthrough, not an additional measured example. During training, a target step is selected from a sampled budget. The student shortcuts from initial noise to the step's starting time, performs the selected transition, and shortcuts to the clean endpoint. The teacher and fake score network assess the distribution discrepancy after that endpoint is re-noised. The resulting gradient updates the student through all three transitions. Sampling a different budget changes the width of the middle segment without requiring a full rollout of that length.

At deployment, the same trained model can generate a preview at 4 NFEs and another output at 32 NFEs. Inference actually executes the selected sampling budget; it does not always use the three-segment training surrogate. Nor does the paper require the preview to become the starting state of the higher-budget run. The demonstrated capability is flexible budgeting with one model, not guaranteed pixelwise identity between outputs at different budgets.

Loss & Training

The synthetic training set contains 256K prompt-video pairs generated by Wan2.1-T2V-14B, with up to 81 frames per sample at \(480\times832\) resolution. Both stages use AdamW and rank-256 LoRA.

Stage one uses a learning rate of \(5\times10^{-5}\). The 1.3B model is trained for 6,000 iterations with a global batch size of 32, and the 14B model for 4,000 iterations with a batch size of 16. Stage two starts from the converged forward checkpoint and jointly optimizes the forward and on-policy objectives for 800 iterations at \(2\times10^{-6}\). The main text does not explicitly give a separate second-stage batch size, so the first-stage settings are not silently reused here.

This is post-training rather than foundation-model pretraining: it depends on pretrained weights, teacher-generated data, and teacher guidance during on-policy distillation. The authors also argue that preserving a fine-grained field permits continued adaptation, but the cached main paper does not provide quantitative downstream fine-tuning results.

Key Experimental Results

Main Results

The T2V results below are selected from Table 2. All listed models carry the paper's re-evaluation marker and use \(480\times832\) resolution. VBench aggregates 16 fine-grained dimensions into Quality, Semantic, and Total scores; higher is better. NFE counts network evaluations. The notation \(50\times2\) includes the two predictions needed for CFG and should not be converted directly into a measured latency speedup.

Model and architecture	Parameters	NFEs	Quality	Semantic	Total
Wan2.1, bidirectional	1.3B	50×2	84.99	76.23	83.24
rCM, bidirectional	1.3B	4	84.71	73.74	82.51
AnyFlow, bidirectional	1.3B	4	85.24	76.41	83.48
AnyFlow, bidirectional	1.3B	32	85.70	76.99	83.96
rCM, bidirectional	14B	4	85.47	76.72	83.73
AnyFlow, bidirectional	14B	4	85.70	77.38	84.04
AnyFlow, bidirectional	14B	32	85.76	77.44	84.10
Self-Forcing, causal	1.3B	4	85.23	76.01	83.39
AnyFlow, causal	1.3B	4	85.60	75.30	83.54
AnyFlow, causal	14B	4	85.82	76.97	84.05
AnyFlow, causal	14B	32	86.12	77.55	84.41

At 4 NFEs, the 14B bidirectional AnyFlow improves Total by 0.31 points over rCM. Increasing its own budget to 32 NFEs adds only 0.06 points, illustrating that positive scaling need not imply a large marginal return. The 1.3B causal model exceeds Self-Forcing by 0.15 Total points but trails it by 0.71 Semantic points. The aggregate improvement is not a win on every dimension.

Ablation Study

The following Overall scores come from Table 1, retaining the combinations that separate forward initialization from backward simulation. The consistency-baseline scores in Tables 1 and 2 differ; the ablation's combined recipe must not be treated as the same evaluation run as the main-table Self-Forcing result.

Forward training and on-policy simulation	Bidirectional, 4 NFEs	Bidirectional, 32 NFEs	Causal, 4 NFEs	Causal, 32 NFEs
Consistency ODE-Init, forward only	80.44	82.86	73.97	77.55
Flow map training, forward only	81.75	83.40	80.48	83.13
Consistency ODE-Init + consistency backward simulation	82.96	79.80	82.49	79.64
Flow map training + consistency backward simulation	83.55	82.96	82.99	83.49
Flow map training + flow map backward simulation	83.48	83.96	83.54	83.96

With flow-map initialization fixed, replacing consistency backward simulation raises bidirectional performance at 32 NFEs from 82.96 to 83.96. This supports the claim that the simulation trajectory matters for higher-budget generation. At 4 NFEs, however, the complete bidirectional method scores 83.48, slightly below the consistency-simulation variant's 83.55. The result is not an improvement in every setting. For causal generation at 4 NFEs, moving from forward-only training to the full method raises the score from 80.48 to 83.54, showing that the parameterization alone does not remove test-time errors.

Key Findings

• Both stages matter: flow-map initialization closes much of the gap, while flow-map backward simulation further preserves the benefit of larger sampling budgets.
• Image-to-video evaluation provides additional evidence. Table 3 reports Quality 80.39, I2V 95.35, and Total 87.87 for 14B causal AnyFlow at 4 NFEs, compared with 80.30, 95.12, and 87.71 for Wan2.1-I2V-14B at 50×2 NFEs. These VBench-I2V totals must not be ranked together with T2V totals.
• The main text says I2V reuses the causal generator with nonuniform chunk partitioning, but does not specify the full partition boundaries. That brief statement is insufficient to reconstruct a separate detailed algorithm.

Highlights & Insights

• The main insight is to match an any-step interface with supervision at different step sizes. Adding a second time variable is insufficient unless training also addresses the errors the student makes over those intervals.
• Three-segment shortcuts propagate endpoint distribution supervision into intermediate transitions without fully unrolling every budget. The reusable idea is to compress training rollouts using composable transitions, not to force all inference into three steps.
• The pretrained boundary velocity field is treated as a stabilizing reference rather than obsolete knowledge. Embedding interpolation and boundary-based loss scaling highlight the importance of preserving pretrained numerical scales during post-training.

Limitations & Future Work

• Evidence boundary: the main paper has no standalone limitations section. Time-sampling weights and embedding ablations are deferred to supplementary material absent from the current cache. Missing hyperparameters and ablation values should not be guessed.
• Approximate simulation: shortcut composition is learned, not exact. A useful next experiment would measure the discrepancy between shortcut states and full multi-step rollouts across budgets and video lengths.
• Uneven marginal returns: the 14B bidirectional model gains only 0.06 points when increasing from 4 to 32 NFEs, and the main table provides no variance or significance tests. Real latency, memory use, and human preference are needed to judge whether that extra computation is worthwhile.
• Limited scope: evaluation covers bidirectional and causal models, 1.3B and 14B scales, and T2V/I2V, but the core backbone remains Wan2.1 and training samples contain at most 81 frames. This does not establish stable gains for every backbone, unlimited-length generation, or every possible budget.
• Teacher dependence: synthetic data and teacher scores remain training requirements. Cheap inference does not imply cheap training, and continued downstream fine-tuning is argued for rather than quantitatively established in the cached main text.

Related Work & Insights

• MeanFlow and Transition Model: average-velocity parameterization and finite-difference derivative estimation originate in prior work. AnyFlow contributes a video post-training recipe and a matching on-policy simulation strategy, not the first flow-map formulation.
• rCM: both approaches can improve distillation through distribution matching, but rCM follows an endpoint-consistency formulation. AnyFlow retains composable interval transitions in simulation and inference, focusing on quality across budgets.
• Self-Forcing: both train on student-generated trajectories to mitigate causal exposure bias. AnyFlow replaces the consistency trajectory to accommodate both few-step and higher-budget sampling; on-policy training itself is not new to this setting.
• Research direction: holding the teacher and data fixed while independently controlling shortcut error, the distribution of training budgets, and gradient truncation would better isolate gains from transition parameterization versus student-state coverage.

Rating

• Novelty: 4/5. Flow maps and DMD are established, but three-segment backward simulation provides a distinct combination for any-step video distillation.
• Experimental Thoroughness: 4/5. Multiple architectures, scales, tasks, and recipe ablations are covered; latency, statistical uncertainty, and long-video evidence remain limited.
• Writing Quality: 4/5. The motivation and trajectory diagrams connect the method clearly, although some implementation details require supplementary material and some cached equations are corrupted.
• Value: 4/5. One model supporting previews and higher-budget generation is useful, but the marginal quality return should be evaluated separately for each architecture.