Skip to content

title: >- [Paper Note] Reward Lightning: Fast Video Generation via Homologous Preference Distillation description: >- [ECCV 2026][video_generation][Preference Distillation] Aligns latent reward modeling with adversarial distillation under the homology principle, enabling 1 to 4-step high-fidelity aligned video generation. tags: - ECCV 2026 - video_generation - Preference Alignment - Video Distillation - Flow Matching date: 2026-09-19 content_hash: b99d6a6d309a7dfe

Reward Lightning: Fast Video Generation via Homologous Preference Distillation

Conference: ECCV 2026
Paper: ECCV Official
Project: https://reward-lightning.github.io
Area: Video Generation
Keywords: Video Diffusion Models, Preference Alignment, Distillation Acceleration, Homologous Preference Distillation, Latent Reward Model

TL;DR

To resolve gradient conflicts caused by representational mismatch in joint video acceleration and alignment, Reward Lightning proposes a homologous preference distillation framework that shares a latent feature backbone with time-modulated attention heads, achieving high-fidelity, well-aligned video synthesis in merely 1 to 4 sampling steps.

Background & Motivation

Video diffusion and flow matching models have achieved remarkable generation quality, yet their huge parameter footprint, temporal context length, and iterative numerical sampling incur prohibitive inference latency. To reduce sampling steps, existing approaches widely adopt progressive distillation, consistency models, or adversarial distillation. However, pure distillation primarily targets data distribution matching and inevitably degrades the generative priors essential for reflecting human aesthetic and semantic preferences. Conversely, directly applying reinforcement learning or preference optimization on accelerated models introduces severe optimization instability.

The core bottleneck of joint acceleration and alignment lies in structural and representational heterogeneity. Prior paradigms fall into two problematic regimes: sequential disjoint training, where preference alignment is followed by distillation, leading to catastrophic forgetting of the aligned distribution during acceleration; and cross-domain heterogeneous training, which linearly sums a pixel-space reward with a latent-space distillation objective. Because pixel decoding and latent velocity updates operate on disjoint manifolds, their gradients conflict severely in the high-noise regimes of few-step generation (often exhibiting negative cosine similarity), precipitating severe distribution collapse.

To eliminate this representational mismatch, this paper establishes the principle of homology—enforcing that preference scoring and adversarial distillation are evaluated on identical latent representations without decoding back to pixels. Core idea: build a Latent Reward Model (LRM) directly in the flow-matching latent manifold, reuse it as a shared feature backbone to drive parallel discriminator and reward heads (HPD), and incorporate an adaptive equilibrium-based weighting mechanism to eliminate gradient conflicts, enabling 1 to 4-step high-fidelity and human-aligned video generation.

Method

Overall Architecture

Reward Lightning comprises two tightly coupled components: first, a Latent Reward Model (LRM) trained to evaluate intermediate noisy flow states directly in latent space; second, Homologous Preference Distillation (HPD), which freezes the LRM feature extractor as a shared backbone and branches into parallel discriminator and reward heads to jointly optimize the few-step flow-matching generator.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-Margin Dataset with Few-Step Pairs"] --> B["Latent Reward Model LRM<br/>Time-Modulated Attention + EMA Margin Clipping"]
    B --> C["Shared Feature Backbone F_phi<br/>Unified Latent Manifold"]
    C --> D["Dual-Head HPD Optimization<br/>Discriminator Head D_psi + Frozen Reward Head R_phi"]
    E["Adaptive Preference Weight lambda<br/>Dynamic Gating via Discriminator Nash Deviation"] --> D
    D --> F["Few-Step Video Generator v_theta<br/>1-4 Step High-Fidelity Output"]

Key Designs

1. Multi-Margin Dataset Construction with Few-Step Augmentation: Conquering Out-of-Distribution Drift Training an effective latent reward model requires overcoming two distinct challenges: discriminating macroscopic quality gaps across different generative models, and detecting subtle motion and semantic differences within the same model's outputs. Crucially, few-step generation trajectories (1 to 4 NFEs) exhibit heavy structural blurring, forming out-of-distribution (OOD) data for standard multi-step evaluators. The authors construct a multi-margin dataset comprising 30,000 intra-model pairs across random seeds, 17,390 inter-model pairs from VisionRewardDB, and 12,000 real-synthetic pairs. To bridge the distillation domain gap, 5,000 specialized few-step augmented pairs are curated by setting high-fidelity 80-step outputs as preferred winners (\(x_w\)) and structurally degraded 1-4 step outputs as rejected samples (\(x_l\)). This allows the reward model to provide reliable, non-divergent gradient signals across truncated distillation trajectories.

2. Time-Modulated Attention Head and Dynamic Margin Clipping: Preventing Collapse and Reward Hacking Traditional reward heads relying on static learnable queries fail across varying noise levels. LRM introduces a time-modulated attention mechanism: for intermediate state \(z_t = (1-t)z_0 + t z_1\), the timestep embedding \(e_t\) is projected through weight matrices \(W_h\) into dynamic state queries \(s_{t,h}\), which cross-attend to spatial features \(F_t\) extracted by the shared backbone. This enables the head to shift focus dynamically—capturing macroscopic layout in high-noise regimes and fine textures in low-noise regimes. Furthermore, applying standard Bradley-Terry with Ties (BTT) to multi-margin data often leads to shortcut learning on trivial domain artifacts (real vs. synthetic), causing feature collapse on subtle intra-model variations. The framework introduces an EMA-guided dynamic margin-clipping mechanism tracking the average reward difference on narrow-margin pairs \(\mathcal{B}_{\text{narrow}}\): $$ \mu \leftarrow \rho \mu + (1-\rho) \frac{1}{|\mathcal{B}{\text{narrow}}|} \sum \Delta r_i $$ When reward differences on large-margin pairs }_{\text{narrow}}\(\Delta r_j\) exceed \(\mu\), they are softly bounded, zeroing out their gradient contribution. This adaptive regularization prevents reward hacking and forces optimization onto fine-grained human preference attributes.

3. Homologous Feature Routing and Dual-Head Adversarial Distillation: Transforming Gradient Conflict into Implicit Regularization While adversarial distillation \(L_{\text{ADV}}^G\) pulls the generated trajectory toward the real data manifold, the preference objective \(L_{\text{ReFL}}\) guides the model toward human-preferred high-reward sub-manifolds. Heterogeneous architectures evaluate these objectives on disparate representations, yielding conflicting gradient vectors. HPD resolves this by structural and space homology: the pre-trained LRM feature extractor \(F_\phi\) serves as a frozen shared backbone, and its weights directly initialize the discriminator head \(D_\psi^{\text{head}}\), which runs in parallel with the frozen reward head \(R_\phi^{\text{head}}\) over identical latent features \(F_t = F_\phi(\hat{z}_t, t)\). Constraining both gradients within the same semantic manifold serves as an implicit gradient regularizer, flipping the gradient cosine similarity from negative in prior baselines to positive, preventing trajectory divergence.

4. Adaptive Preference Weighting via Discriminator Equilibrium: Self-Paced Curriculum Distillation During early distillation stages, generator predictions deviate markedly from the true distribution, rendering preference feedback noisy and detrimental. The framework introduces an adaptive preference weight \(\lambda\) based on the deviation of the discriminator loss \(L_{\text{ADV}}^D\) from its theoretical Nash equilibrium value of 0.5: $$ \hat{\lambda} = \exp\left( -\text{ReLU}(\text{sg}(L_{\text{ADV}}^D) - 0.5)^2 \right) $$ coupled with momentum EMA smoothing (\(\beta = 0.99\)): \(\lambda \leftarrow \beta \lambda + (1-\beta)\hat{\lambda}\). When predictions exhibit noticeable structural artifacts (\(L_{\text{ADV}}^D \gg 0.5\)), \(\hat{\lambda} \to 0\) suppresses the preference gradient. As structural realism is established (\(L_{\text{ADV}}^D \to 0.5\)), \(\hat{\lambda} \to 1\) smoothly activates preference guidance, forming an automated, stable curriculum.

Loss & Training

The pipeline is trained in two distinct stages: 1. LRM Training: Built on Wan2.2-14B, trained with AdamW (learning rate \(1\times 10^{-5}\), weight decay \(1\times 10^{-3}\)) on 720P 5-second video pairs. Employs DeepSpeed Ulysses Sequence Parallelism (SP), Fully Sharded Data Parallel (FSDP), and CPU-GPU offloading on 8× H20 GPUs. 2. HPD Training: The backbone \(F_\phi\) and reward head \(R_\phi^{\text{head}}\) are frozen. The discriminator head is updated with Hinge loss \(\mathcal{L}_{\text{ADV}}^D\). The generator \(\theta\) minimizes: $$ \mathcal{L}{\text{HPD}}(\theta) = \mathcal{L}}}^G + \lambda \cdot \mathcal{L{\text{ReFL}} = - D\psi^{\text{head}}(F_t^{\text{fake}}, t) - \lambda \cdot R_\phi^{\text{head}}(F_t^{\text{fake}}, t) $$ Trained with AdamW (betas 0.5/0.999, learning rate \(5\times 10^{-7}\)) with a batch size of 64 for 2,000 steps to reach convergence.

Key Experimental Results

Main Results

The models were evaluated on out-of-distribution reward benchmarks (Table 1) and comprehensive generative quality benchmarks via VBench (Table 2). All generative baselines were built on the Wan2.2-14B foundation.

Table 1: Preference accuracy and online reward feedback efficiency (from original Table 1, 8× H20 GPUs)

Paradigm Method VideoGen-RewardBench w/ Ties (%) VideoGen-RewardBench w/o Ties (%) GenAI-Bench w/ Ties (%) Training Memory (GB) Training Latency (ms)
Pixel Reward LiFT 39.08 57.26 37.06 87.81 213.2
Pixel Reward VisionReward 56.77 67.59 51.56 82.15 185.3
Pixel Reward VideoAlign 61.26 73.59 49.41 87.46 217.9
Latent Reward PRFL 57.59 69.20 53.40 77.32 156.8
Latent Reward (Ours) LRM (Ours) 72.24 86.63 59.85 76.24 154.2

Table 2: Quantitative comparison on VBench for T2V and I2V (from original Table 2)

NFEs Method Training Type T2V Text Alignment T2V Motion Quality T2V Visual Quality T2V Overall Score (↑) I2V Overall Score (↑)
80-NFEs Wan2.2 Baseline Pretrain 97.34 79.59 69.48 84.23 92.91
80-NFEs PRFL Latent RL 97.83 81.56 70.45 85.72 94.87
4-NFEs TurboDiffusion Distill (rCM) 94.53 75.62 63.28 83.51 92.45
4-NFEs DMDR Pixel RL & Distill 96.14 71.85 68.57 84.63 93.28
4-NFEs FlashDMD Pixel RL & Distill 88.19 68.23 67.84 82.36 92.59
4-NFEs (Ours) HPD (Ours) Homologous Joint 97.26 77.48 68.71 85.83 95.34
1-NFE APT Distill (Adv) 85.37 65.42 64.19 78.53 84.87
1-NFE (Ours) HPD (Ours) Homologous Joint 92.54 70.16 62.83 82.42 91.92

Ablation Study

The paper validates key components across dataset composition, head architecture, and initialization strategies (Table 3).

Table 3: Ablation study on key components (from original Table 3)

Ablation Category Setting Preference Accuracy (%) Overall Quality Score (Score) Finding
Dataset Design Full Configuration 72.24 95.34 Optimal performance across both metrics
Dataset Design w/o Few-Step Augmentation 68.47 93.81 Lack of few-step blur priors degrades few-step guidance
Dataset Design w/o Real-Synthetic Pairs 65.12 92.45 Quality ceiling drops without real-world anchors
Head Architecture Self-Attention 64.38 91.87 Cannot dynamically focus across flow timesteps
Head Architecture Static Trainable Query 69.85 94.12 Struggles to adapt to severe noise level transitions
Head Architecture Time-Modulated Query 72.24 95.34 Successfully extracts timestep-specific semantic features
Discriminator Init Random Initialization - 88.26 Adversarial training destabilizes, causing visual artifacts
Discriminator Init Pre-trained Backbone Init - 93.48 Moderate performance; lacks preference prior
Discriminator Init LRM Head Weight Inheritance - 95.34 Provides strong inductive bias and stable convergence

Key Findings

  • Resolution of Gradient Conflict: Measuring cosine similarity between \(\nabla_\theta \mathcal{L}_{\text{ADV}}^G\) and \(\nabla_\theta \mathcal{L}_{\text{ReFL}}\) over 1,000 iterations (Table 4-a) reveals that pixel reward + distillation (\(-0.14\)) and separate-backbone latent reward + distillation (\(-0.06\)) actively fight each other. In contrast, HPD with a shared homologous backbone achieves +0.41, validating that representational homology acts as an implicit gradient regularizer.
  • Superior Single-Step Generation: At 1-NFE, HPD achieves a VBench score of 82.42, outperforming the previous distillation champion APT (78.53) by +3.89 points, notably improving Text Alignment by 7.17 points (92.54 vs. 85.37).
  • Reduced Training Overhead: By eliminating high-resolution pixel decoding and pixel-space backpropagation, LRM reduces training memory from 87.46 GB to 76.24 GB and latency from 217.9 ms to 154.2 ms (an ~29% speedup).

Highlights & Insights

  • The Homology Principle for Multi-Objective Post-Training: Pinpoints representation mismatch as the root cause of gradient conflicts in few-step distillation, replacing cumbersome explicit multi-objective optimization with an elegant shared-backbone formulation.
  • Few-Step Degraded Outputs as Asset: Repurposes structurally blurred 1-4 step outputs as negative preference anchors, systematically eliminating the evaluator's out-of-distribution blind spot.
  • Nash-Equilibrium Gating Function: Employs a half-Gaussian squash of discriminator loss deviation from 0.5 to dynamically modulate preference gradients, establishing an automated self-paced curriculum for adversarial reinforcement learning.

Limitations & Future Work

  • Sequence Length and Resolution Ceiling: While thoroughly benchmarked at 720P 5-second videos, extending this framework to 4K multi-minute videos will impose severe inter-node communication overhead on Ulysses sequence parallelism.
  • Heuristic Tie Boundary: While BTT outperforms standard BT loss, identifying delicate ties among closely competing few-step outputs still relies on empirical margin hyperparameters.
  • Backbone Coupling: Although LRM boosts external distillation algorithms (rCM, DMD2), the feature extractor and discriminator heads remain coupled to Wan2.2, necessitating re-adaptation when switching to alternative DiT backbones.
  • vs DMDR / FlashDMD: DMDR and FlashDMD optimize pixel rewards against latent distillation, suffering from negative gradient cosine similarity; Reward Lightning operates entirely within a shared latent manifold, achieving positive gradient alignment (+0.41).
  • vs PRFL: PRFL pioneers latent reward evaluation for video diffusion models, but targets standard multi-step (80-step) RLHF; Reward Lightning extends latent evaluation to few-step distillation via time-modulated queries and homologous multi-task routing.
  • vs APT / TurboDiffusion: Pure distillation methods minimize sampling steps while disregarding human preference alignment; Reward Lightning matches or exceeds the visual alignment of the 80-step teacher model within 1 to 4 steps.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ [Pioneering diagnosis and solution of representational mismatch in joint video acceleration and alignment via homology]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across OOD benchmarks, VBench, gradient conflict measurements, and comprehensive ablations]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical formulation, cohesive narrative, and insightful architectural illustrations]
  • Value: ⭐⭐⭐⭐⭐ [Directly facilitates real-time high-fidelity video generation deployments and provides an effective template for DiT post-training]