AR-CoPO: Align Autoregressive Video Generation with Contrastive Policy Optimization¶

Conference: ECCV 2026
arXiv: 2603.17461
Code: None
Area: Video Generation / RLHF Alignment
Keywords: Autoregressive Video Generation, Contrastive Policy Optimization, RLHF, Consistency Models, Chunk-level Forking Alignment

TL;DR¶

AR-CoPO proposes a chunk-level contrastive policy optimization framework to align few-step autoregressive video generators: by constructing initial noise neighborhood candidates through a forking mechanism at a randomly selected chunk, and utilizing local GRPO updates for credit assignment, it sidesteps the fundamental mismatch between SDE-style GRPO and the near-deterministic dynamics of consistency models; meanwhile, a semi-on-policy training paradigm is designed to leverage fixed reference replay and a ratio-clipping trust-region constraint to improve text-semantic alignment quality and prevent motion quality collapse caused by pure exploration, achieving joint improvements on both VBench and VideoAlign benchmarks based on Self-Forcing.

Background & Motivation¶

Background: Diffusion and flow matching models have achieved significant progress in image and video generation, but their bidirectional generation inference cost increases linearly with the number of sampling steps and video length. To support low-latency, variable-length, and streaming generation scenarios, a large body of recent work (such as CausVid, Self-Forcing, Causal-Forcing, LongLive) distills powerful bidirectional video models into causal generators operating in a patch-by-patch or chunk-by-chunk autoregressive manner. Combined with few-step distillation technologies like distribution matching distillation, the inference process can be compressed to a few steps of an ODE solver or a consistency model, further reducing costs with KV caching.

Limitations of Prior Work: However, the combination of streaming autoregressive architectures and few-step distillation poses significant challenges to post-training RLHF alignment. For flow-matching generators, post-training policy gradient (GRPO-like objectives) is a natural choice—treating sampling as policy rollout and optimizing the distribution using reward feedback. Yet, in practice, GRPO variants rely on a key implementation choice: converting deterministic ODE sampling into a stochastic SDE form to introduce a Markov Decision Process (e.g., DanceGRPO, FlowGRPO, BranchGRPO). When applying these SDE-style GRPO methods to few-step streaming generators, a fundamental mismatch arises: (1) few-step generators (distilled ODEs or consistency models) deviate from standard flow-matching ODEs and are difficult to train with continuous flow-matching SDE methods; (2) the sampling trajectories of these models are short with highly limited stochasticity, making them highly sensitive to initial noise and causing the outputs to be dominated by the initial noise—whereas SDE methods rely heavily on intermediate noise injection to guide exploration (typically freezing the initial noise), resulting in near-zero gradient signals.

Key Challenge: The fundamental contradiction lies between the near-deterministic dynamics of few-step autoregressive video generators (where outputs are dominated by initial noise and intermediate noise barely alters the output) and the massive intermediate noise exploration required by SDE-style GRPO.

Key Insight: Neighbor GRPO offers a brand-new perspective: it reinterprets SDE-GRPO as a distance-driven contrastive learning objective over neighborhood candidate trajectories. This avoids relying on stochastic SDE exploration during sampling, instead constructing initial noise neighborhoods and defining a soft-max distance surrogate distribution during training to achieve controllable exploration while maintaining deterministic ODE sampling during inference.

Core Idea: AR-CoPO adapts the contrastive perspective of Neighbor GRPO to streaming autoregressive video generation: (1) It introduces chunk-level alignment, where neighborhood candidates are constructed by forking only at a single randomly selected chunk, while other chunks share identical noise, allowing credit assignment to be naturally attributed to that specific chunk. (2) It shifts the distance metric from the intermediate latent \(x_t\) space to the clean prediction \(\hat{x}_0\) space for consistency models, ensuring contrastive signals operate in a semantically meaningful space. (3) It designs a semi-on-policy training paradigm using fixed reference replay and a ratio-clipping trust-region constraint, overcoming the limitations of pure exploration on global semantic rewards and preventing reward hacking.

Method¶

Overall Architecture¶

The training process of AR-CoPO consists of three main stages, with the overall architecture illustrated in the flowchart below:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Prompt"] --> B["Shared Context Generation<br/>Chunks 1 to p-1"]
    B --> C["Forking Alignment<br/>Perturb Chunk p<br/>Initial Noise → G Candidates"]
    C --> D["Complete Remaining Sequence<br/>Deterministic Generation of<br/>Chunks p+1 to L"]
    D --> E["Reward Scoring<br/>Each Branch Receives<br/>Sequence-level Reward r^(i)"]
    E --> F["Replay & GRPO Update<br/>Backpropagation Performed<br/>Only on Chunk p"]

Specifically, for a target sequence of length \(L\) chunks, each training iteration executes the following operations:

Rollout (Neighborhood Candidates Generation): A fork chunk index \(p \in \{1,\dots,L\}\) is randomly sampled. The model first generates the first \(p-1\) chunks to establish a shared context \(h_{p-1}\) (containing the cached KV states). At the \(p\)-th chunk, \(G\) perturbed neighborhood noises \(\{\epsilon_p^{(i)}\}\) are constructed based on the shared initial noise \(\epsilon^*_p\) (according to Eq. 1). Each branch \(i\) independently completes the \(T\)-step denoising generation for the \(p\)-th chunk using its corresponding noise, and the results are stored in a replay buffer. Subsequently, the remaining \(L-p\) chunks are completed deterministically, with all branches using the exact same shared noise sequence in this phase, ensuring that the only difference originates from the forking chunk.
Reward (Sequence-level Reward Scoring): After each branch generates the complete video, a sequence-level reward \(r^{(i)}\) (comprising three dimensions: text alignment TA, video quality VQ, and motion quality MQ) is computed by the reward model.
Replay & Update (Replay & GRPO Update): The trajectory of the \(p\)-th chunk is retrieved from the buffer. The current policy is used to replay the forking chunk, and the Euclidean distance between the anchor and the candidates is computed in the \(\hat{x}_0\) prediction space. Based on this, a surrogate policy ratio is constructed to perform the clipped GRPO update, with backpropagation strictly confined within the \(T\) steps of the forking chunk.

Key Designs¶

1. Forked Chunk-level Alignment: Sequence-level Credit Assignment via Local Noise Perturbation

Applying full-sequence GRPO directly to AR generators is prohibitively expensive (\(O(L \times G)\) backpropagation) and makes credit assignment difficult—when the global reward is poor, it is hard to trace which chunk went wrong. The core innovation of AR-CoPO is to strictly restrict alignment operations to a single randomly selected chunk. Specifically, the model first generates the first \(p-1\) chunks as the shared context \(h_{p-1}\). At the \(p\)-th chunk, \(G\) neighborhood noises are constructed around the shared initial noise \(\epsilon^*_p\) via the perturbation method of Neighbor GRPO: \(\epsilon^{(i)} = \sqrt{1-\sigma^2} \cdot \epsilon^* + \sigma \cdot \delta^{(i)}\). Each branch independently denoises this chunk using a different \(\epsilon^{(i)}\). All branches then share the identical subsequent noise (including the initial noises of non-forked chunks and the solver noise of all denoising steps) to deterministically complete the remaining \(L-p\) chunks.

This noise-sharing control is the masterstroke of the design: it ensures that the reward difference between any two branches \(r^{(i)} - r^{(j)}\) is uniquely attributed to the choice of the initial noise at the \(p\)-th chunk, without any confounding randomness from subsequent stages. During the update phase, backpropagation is strictly confined within the \(T\) steps of the forked chunk, reducing the computational cost from \(O(L \times G)\) to \(O(T \times G)\). The forking position \(p\) is randomly sampled at each step, ensuring all chunks have an opportunity to be aligned during training.

2. \(\hat{x}_0\) Distance Contrastive Optimization: Adapting Surrogate Policy Definitions for Consistency Models

The original formulation of Neighbor GRPO constructs contrastive surrogate policies based on the Euclidean distance of intermediate latents \(x_t\) from ODE solvers, which is natural for flow-matching models with continuous-time velocity fields. However, for consistency models (such as Self-Forcing), the situation differs: the key operation of CM is to directly map from a fancy noisy latent to the clean prediction \(\hat{x}_0\) in a single step, bypassing standard DDIM/ODE trajectories. Measuring distances in the intermediate \(x_t\) space confounds noise scales with semantic content and is uninformative.

AR-CoPO shifts the distance metric to the \(\hat{x}_0\) prediction space, leveraging the single-step prediction of CM \(\hat{x}_{0,t} = F_\theta(x_t, h_{t-1}, t)\):

\[d_{0,t}^{(i)} = \| \hat{x}_{0,t}^{(i)} - \hat{x}_{0,t}^{(\theta)} \|_2^2, \qquad \pi_\theta(i \mid s_t) = \frac{\exp(-d_{0,t}^{(i)} / \tau_0)}{\sum_{k=1}^G \exp(-d_{0,t}^{(k)} / \tau_0)}\]

where \(\hat{x}_{0,t}^{(i)}\) is the prediction of the old parameters on the candidate inputs, and \(\hat{x}_{0,t}^{(\theta)}\) is the prediction of the current parameters on the anchor. This design ensures that distances are measured in a semantically meaningful, clean prediction space (closely matched \(\hat{x}_0\) means similar visual content, while distant ones denote major semantic differences), reflecting the true differences among candidates more accurately than the \(x_t\) space. Substituting this distance-driven soft-max policy into the standard GRPO objective naturally pulls the anchor toward high-reward candidates and pushes it away from low-reward ones.

3. Semi-On-Policy Alignment Training: Improving Semantic Alignment with Fixed Reference Replay and Trust-Region Constraints

Pure on-policy exploration performs poorly on text alignment (TA) rewards. TA is a global semantic-level reward that measures whether a video faithfully reflects the prompt, and local noise perturbations rarely alter the overall video semantics. Experiments confirm (Table 2) that although pure on-policy TA optimization improves TA scores, it causes motion quality (MQ) to collapse from 1.68 to 0.25, and VBench Total to drop from 82.15 to 79.26—a clear case of severe reward hacking where the generated videos display motion discontinuity and inter-frame inconsistency.

AR-CoPO addresses this by training two independent LoRA adapters complementarily and merging them at the end: (a) On-policy adapter—driven by initial noise perturbation exploration to boost VQ and MQ, showing a steadily rising training curve. (b) Semi-on-policy adapter—rollouts are fixed to the reference policy (initial checkpoint), and \(K=100\) groups of reference candidates are pre-collected and stored in a replay buffer. During training, the current policy re-predicts \(\hat{x}_0\) on the anchor, computes the distance to the old \(\hat{x}_0\) of the \(G\) candidates in the buffer, and upweights high-reward candidates while suppressing low-reward ones through the contrastive objective, without relying on random exploration to discover new patterns. The key is to retain the ratio clipping mechanism in the GRPO objective, which exerts an implicit trust-region constraint around \(\pi_{ref}\) to prevent the policy from collapsing by drifting too far from the reference distribution. Ablation studies confirm that removing clipping in off-policy mode causes all metrics to collapse (VBench Total drops to 67.99).

The two adapters are linearly merged with a scaling ratio. The paper proposes a "dual-improvement criterion" to select the scale: it must simultaneously improve both the in-domain VideoAlign Overall rating and the out-of-domain independent benchmark VBench Total. A scale of 0.8 satisfies this condition: VideoAlign rises from 7.76 to 8.22, while VBench Total maintains 82.15 \(\rightarrow\) 82.17. Going beyond this value (e.g., 1.0) improves VideoAlign but degrades VBench Total, which is identified as over-optimization rather than genuine quality improvement.

A Full Example of a Training Iteration with \(L=8, p=4, G=12\)¶

Assuming the model generates a video of 8 chunks: (1) A chunk index \(p=4\) is randomly sampled, and the model first generates chunks 1-3 to establish the shared context \(h_3\); (2) At the 4-th chunk, 12 neighborhood noises \(\{\epsilon_4^{(1)}\dots\epsilon_4^{(12)}\}\) are constructed based on \(\epsilon^*_4\), and each branch independently denoises chunk 4 (\(T=4\) steps of CM sampling); (3) The 12 branches complete the deterministic generation of chunks 5-8 using the exact same subsequent noise seeds, yielding 12 complete videos; (4) The reward model provides 12 sequence-level rewards \(r^{(1)}\dots r^{(12)}\), and advantages \(A^{(i)}\) are computed using their mean and standard deviation; (5) Replay phase: the saved trajectory of chunk 4 is retrieved, the current policy is used to re-predict \(\hat{x}_0\), and distances to the old \(\hat{x}_0\) of the 12 candidates are computed to construct the surrogate policy \(\pi_\theta(i)\). This is substituted into the GRPO objective to update \(\theta\) via backpropagation strictly over the \(T\) steps of chunk 4. In this process, the candidate with the highest reward among the 12 branches has the shortest distance and is upweighted, while the lowest-reward candidate has the closest distance and is pushed away—aligning and assigning credit precisely to the specific frames that affect video quality.

Loss & Training¶

The optimization objective of AR-CoPO inherits the contrastive GRPO form of Neighbor GRPO. Given the advantages \(A^{(i)} = (r^{(i)} - \bar{r}) / \sigma_r\) of \(G\) candidates and the distance-based surrogate policy \(\pi_\theta(i)\) in the \(\hat{x}_0\) space (Eq. 4), the objective is formulated as:

\[J(\theta) = \frac{1}{G} \sum_{i=1}^{G} \min\left( \frac{\pi_\theta(i)}{\pi_{\text{old}}(i)} A^{(i)},\; \operatorname{clip}\!\left( \frac{\pi_\theta(i)}{\pi_{\text{old}}(i)},\, 1-\epsilon,\, 1+\epsilon \right) A^{(i)} \right)\]

LoRA (rank=64, \(\alpha=128\)) is used for training, with a learning rate of \(1 \times 10^{-5}\), group size \(G=12\), symmetric anchor sampling batch size 4 on 24 GPUs. The optimization objective jointly targets three reward dimensions (TA+MQ+VQ). The semi-on-policy replay buffer contains 100 groups of rollouts.

Key Experimental Results¶

Main Results¶

A comparison with three streaming AR baselines (Self-Forcing, Causal-Forcing, LongLive) on Self-Forcing is presented below. Rewards are jointly optimized using the VideoAlign suite (TA+VQ+MQ), and out-of-domain evaluation is done via VBench (Quality / Semantic / Total).

Method	VBench Q	VBench S	VBench Total	VQ	MQ	TA	Overall
Self-Forcing	84.87	71.27	82.15	3.80	1.68	2.28	7.76
Causal-Forcing	85.27	70.35	82.28	3.97	1.43	2.40	7.79
LongLive	85.10	71.16	82.31	3.87	1.76	2.43	8.06
SF + ours (semi)	85.15	71.68	82.45	3.70	1.60	2.30	7.61
SF + ours (on-policy)	84.81	70.71	81.99	4.15	2.06	2.30	8.51
SF + ours (merged)	85.07	70.55	82.17	4.00	1.86	2.36	8.22

The semi-on-policy variant alone achieves the best VBench Total (82.45), outperforming all baselines. The fully merged AR-CoPO (scale=0.8) improves the VideoAlign Overall score from 7.76 to 8.22 (+0.46) while maintaining VBench Total around 82.15 \(\rightarrow\) 82.17, satisfying the dual-improvement criterion and demonstrating a genuine quality improvement rather than in-domain reward hacking.

Ablation Study¶

Under the controlled setting of optimizing only the TA reward, three training strategies are compared:

Strategy	VBench Q	VBench S	VBench Total	VQ	MQ	TA	Overall
Self-Forcing baseline	84.87	71.27	82.15	3.80	1.68	2.28	7.76
On-policy	81.66	69.68	79.26	3.53	0.25	2.63	6.42
Off-policy (w/o ratio clipping)	69.78	60.84	67.99	2.22	-0.15	2.16	4.23
Semi-on-policy	85.15	71.68	82.45	3.70	1.60	2.30	7.61

Additionally, scanning the LoRA merge ratio shows that scale=0.8 is the optimal choice (the maximum value satisfying the dual-improvement criterion). Although scale=1.0 yields the highest VideoAlign rating (8.33), it is deemed over-optimized as VBench Total drops to 81.99.

Key Findings¶

Comparison with SDE-GRPO: SDE-style GRPO baselines (mimicking DanceGRPO / FlowGRPO) show flat, non-increasing VQ reward curves throughout training, whereas AR-CoPO rises steadily. The primary reason is directly verified in the noise replacement experiment in Figure 6—exchanging the initial noise drastically shifts the output, whereas exchanging intermediate CM solver noise causes almost no change. This demonstrates that few-step CM is nearly deterministic, and intermediate noise exploration provides virtually no information.
Semi-on-policy is critical for TA optimization: Pure on-policy TA optimization causes MQ to collapse from 1.68 to 0.25. Global semantic rewards cannot be optimized via local noise exploration, leading the model to take shortcuts (reward hacking) that improve TA at the expense of motion continuity. Semi-on-policy training with ratio clipping entirely prevents this collapse.
Ratio clipping is indispensable: Off-policy training without clipping collapses entirely (VBench Total drops from 82.15 to 67.99), validating the critical role of trust-region constraints in defending against distribution shift.
Distinct behaviors across reward dimensions: VQ and MQ can be effectively improved via on-policy exploration (see the on-policy row in Table 1: VQ 3.80 \(\rightarrow\) 4.15, MQ 1.68 \(\rightarrow\) 2.06), whereas TA must be improved through the exploitation path.

Highlights & Insights¶

Diagnostic value of the noise replacement experiment (Figure 6): By utilizing an elegant controlled-variable design—"freezing all random seeds and replacing only the initial noise / replacing only the CM noise at specific intermediate steps"—it elegantly verifies the reason for and extent to which few-step consistency models are near-deterministic. This diagnostic methodology can be generalized to evaluate the source of randomness in any few-step generative model.
Credit assignment design of the forking mechanism: All non-forked chunk noises are fully shared, allowing the reward differences to be cleanly attributed to the forked chunk. Backpropagation is strictly confined to that single chunk, reducing computational costs to \(O(T \times G)\) instead of \(O(L \times G)\). This design, which kills two birds with one stone by addressing both credit assignment and computational efficiency, is a testament to systematic engineering ingenuity.
Adaptation to \(\hat{x}_0\) space distance: A seemingly small change (shifting from \(x_t\) to \(\hat{x}_0\)) essentially transfers contrastive learning from the "noisy latent space" to the "semantic visual space," giving the distance metric a clear physical meaning. This approach of "adapting the metric space to the architecture" can be generalized to other non-standard sampling frameworks (such as rectified flow and distilled models).
Semi-on-policy + dual LoRA merging paradigm: By applying different optimization paradigms (exploration vs. exploitation) for different reward dimensions, training them separately with LoRA, and then merging them, the approach avoids conflicts under a single objective. This paradigm is valuable for various multidimensional alignment tasks (e.g., text + image + motion multi-rewards).
Preventing reward hacking via the dual-improvement criterion: Refusing to rely solely on a single in-domain metric for model selection, and instead enforcing a simultaneous check against an independent, out-of-domain benchmark, provides a simple yet effective standard for evaluating RLHF alignment.

Limitations & Future Work¶

The proposed method has only been fully validated on Self-Forcing, while the results on Causal-Forcing are only included in the Appendix. Its generalizability across other streaming AR generators (e.g., different architectures like CausVid, LongLive) remains to be further tested.
Semi-on-policy requires pre-collecting 100 groups of reference rollouts, introducing up-front overhead and extra hyperparameters (such as buffer size and the timing of reference policy selection). Additionally, the fork position is sampled entirely at random; exploring whether learning to select the optimal fork location can further improve efficiency is a promising direction.
The dual LoRA merge uses a single global scale. Different reward dimensions (TA vs. VQ vs. MQ) might benefit from distinct merge ratios, but the paper only presents the optimal choice along a single scanning dimension.
In the semi-on-policy phase, the reference policy remains static. As the on-policy adapter continues to evolve, the fixed reference might gradually become "outdated"—exploring periodic updates of the reference policy via EMA could be beneficial.

vs. DanceGRPO / FlowGRPO / BranchGRPO: These methods achieve exploration by converting deterministic ODEs to stochastic SDEs, which is effective for long-trajectory diffusion models but fails on near-deterministic short trajectories. AR-CoPO derives contrastive signals from the initial noise neighborhood rather than intermediate noise injection, fundamentally bypassing the near-deterministic limitation.
vs. Neighbor GRPO: AR-CoPO originates from the same contrastive perspective but introduces crucial adaptations: (1) reducing full-sequence alignment to chunk-level alignment to fit the AR streaming structure; (2) shifting \(x_t\) distance to \(\hat{x}_0\) distance to accommodate CM; (3) introducing a semi-on-policy paradigm to resolve the bottleneck of optimizing global semantic rewards.
vs. Diffusion-DPO / DR-Tune: Pairwise preference-based DPO approaches require manual annotation or synthetic preference data, whereas AR-CoPO utilizes reward model scoring, providing richer and more scalable training signals. Traditional RLHF for diffusion usually requires a critic network (e.g., value function), whereas the critic-free objective of the GRPO style reduces implementation complexity.

Rating¶

Novelty: ⭐⭐⭐⭐ Systematically adapting the contrastive perspective of Neighbor GRPO to streaming AR video generation is a methodologically effective innovation; the combination of chunk-level forking + \(\hat{x}_0\) space distance + semi-on-policy training represents a highly clever engineering effort, although the core contrastive view is inherited from Neighbor GRPO.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ The experimental design is rigorous and multi-layered: the main experiment establishes dual benchmarks (in-domain VideoAlign + out-of-domain VBench) to compare against three baselines; the ablation study directly compares three training strategies; the noise replacement diagnostic experiment yields direct evidence of near-determinism; the LoRA scale sweep quantitatively demonstrates trade-offs; and the comparison of training curves with SDE-GRPO provides a solid baseline reference. The chain of proof is fully complete.
Writing Quality: ⭐⭐⭐⭐⭐ The motivation of the problem is established with exceptional clarity: the training curves in Figure 2 and the noise replacement experiment in Figure 6 make it immediately obvious to readers "why SDE-GRPO does not work"; the algorithmic flow and diagrams in the method section are well-integrated; and the contrastive perspective (reinterpretation of Neighbor GRPO) runs consistently throughout the text, showing strong logical coherence.
Value: ⭐⭐⭐⭐ This work provides a pragmatic and reliable solution for the RLHF alignment of streaming video generators, carrying direct practical value in the context of community interest in both VideoAlign and VBench benchmarks; the design concept of leveraging exploitation (semi-on-policy) to resolve global semantic reward optimization is highly informative for broader alignment tasks.