Aligning Human Sense: Calibrated Distributional Reward Learning for Video Generation

Conference: ECCV2026
Paper: ECCV Official Page
PDF: Full Paper
Code: https://github.com/alignhs26/ahs
Area: Video Generation
Keywords: Preference calibration, multidimensional rewards, Wasserstein distance, GRPO, human feedback alignment

TL;DR

The paper connects consensus-based annotation calibration, multidimensional reward learning, and Wasserstein-constrained GRPO, improving action pairwise accuracy excluding ties from 80.30% to 83.33% and the reported aggregate scores of three video generators, although not every individual metric improves.

Background & Motivation

Visually plausible video is not necessarily video that matches human judgment. Viewers consider whether the action follows the prompt, whether motion looks natural, and whether the subject remains consistent. These dimensions can conflict, and different annotators can assign different scores to the same clip. Averaging raw scores or turning them into winner-loser pairs can therefore introduce ambiguous supervision into a reward model, with subsequent reinforcement learning amplifying its errors.

The authors argue that changing the policy optimizer alone is insufficient: annotation reliability, the information retained by the reward representation, and the space in which policy deviations are constrained all matter. They also question the commonly used per-denoising-step KL constraint in video GRPO. Matching local policy densities to a reference does not necessarily improve the final video's semantics, motion, or temporal consistency. This is a critique of the particular alignment setup, not a proof that KL can represent only local information in every setting.

The proposed response is to learn a scoring proxy from consistent annotations, use it to revise less consistent data, and carry the resulting supervision into both reward learning and generator adaptation. Core idea: improve the reliability of preference supervision first, then add Wasserstein constraints in multidimensional reward space and policy output space so that the learned preferences and the generator's optimization form a connected pipeline.

Method

Overall Architecture

Training starts with videos, text prompts, and dimension-specific scores from multiple annotators. Elite-Guided Preference Calibration separates high- and low-consistency samples. Preference-Aware Reward Modeling learns a multidimensional quality representation from the calibrated supervision. Preference-Guided Policy Optimization then generates a group of candidate videos for each prompt and updates the generator using their relative reward advantages.

The two Wasserstein terms have different operands. Reward learning compares a predicted reward distribution with a target constructed from calibrated labels; policy learning compares the output distributions of the current and reference policies. They should not be collapsed into a single loss, and the reference policy should not be equated directly with the original distribution of human ratings.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    inputs["Videos, prompts,<br/>multi-annotator scores"] --> calibration["Elite-Guided<br/>Preference Calibration"]
    calibration --> reward["Preference-Aware<br/>Reward Modeling"]
    prompts["Prompts and candidate<br/>video groups"] --> policy["Preference-Guided<br/>Policy Optimization"]
    reward --> policy
    reference["Reference policy"] --> policy
    policy --> output["Aligned video generator"]

Key Designs

1. Elite-Guided Preference Calibration: learn stable judgments before revisiting disagreement

Here, elite means high annotation consistency, not a high-quality video or a new panel of expert annotators. A sample qualifies when all annotators give the same score, or when at most one rating differs from the mode by no more than one rating level. The remaining samples form the low-consistency subset. This provides a starting supervision set that is closer to reproducible consensus and reduces the risk of fitting contradictory labels from the outset.

An Elite Scorer is trained on the high-consistency subset to regress aggregated human ratings from the video and prompt. It then assigns revised scores to low-consistency samples, which are combined with the original high-consistency subset. The experiments use Qwen2-VL-2B and initially average the original annotator scores. The experimental setup also mentions confidence-based reweighting and hard filtering of low-scoring outliers, but does not specify the confidence definition or filtering thresholds. Those omissions do not justify inventing a precise executable filtering procedure.

This is not simply a proposal to discard every disputed example. The scorer first learns from cleaner supervision and then attempts to recover the training value of the remaining data, increasing the effective supervision scale. However, the mechanism treats disagreement as something to correct. If disagreement expresses genuine differences in taste, the proxy may replace minority preferences with majority consensus. The later distributional objective does not automatically remove this selection bias.

2. Preference-Aware Reward Modeling: retain dimensions and constrain distributions beyond regression

The reward model still receives a video and its prompt, but does not consider only one undifferentiated quality score. Calibrated labels are represented as multidimensional vectors, and a conditional reward distribution is learned with both MSE regression and Wasserstein alignment. Although the source dataset has nine assessment dimensions, the central reward experiments focus on Action and Motion Effects. Results for these two dimensions should not be expanded into a claim that full nine-dimensional joint modeling has been validated.

The essential relationship in the paper's equations (3)-(5) is:

\[ \mathcal{L}_{\mathrm{total}} =\mathcal{L}_{\mathrm{MSE}} +\lambda W_p\!\left(p_\theta(\mathbf{z}\mid v,p),\delta_{\mathbf{y}}\right). \]

MSE encourages the predicted reward vector to match the calibrated label. The second term measures the transport distance between the predicted reward distribution and the label location, weighted by a positive coefficient \(\lambda\). Wasserstein distance uses distances in score space rather than merely checking whether densities coincide. The authors intend this to strengthen constraints involving semantic and motion-related quality attributes; it supplements regression rather than replacing sample-level supervision.

An important qualification is that the per-sample empirical target is written as \(\delta_{\mathbf{y}}\), a point mass at one calibrated label vector. It is not a multimodal distribution retaining every annotator's score. Consequently, distributional reward learning here should not automatically be interpreted as recovering population-level uncertainty or correlation structure. The cached text also does not sufficiently specify the predicted distribution's parameterization or the transport estimator. This note therefore does not assume Gaussian outputs, covariance prediction, or a particular Sinkhorn implementation.

3. Preference-Guided Policy Optimization: relative rewards set the direction, distributional constraints limit deviation

For a given text prompt, the old policy samples a group of candidate generation trajectories. The reward model assigns a score to each trajectory. Subtracting the group mean and dividing by the group standard deviation produces a relative advantage. Candidates scoring above the group average encourage the corresponding generation behavior, while lower-scoring candidates push in the opposite direction. The group-relative comparison reduces the influence of reward-scale differences across prompts.

The update retains the basic GRPO/PPO mechanism: compute the probability ratio between new and old policies at each trajectory step, clip that ratio to limit update magnitude, and average over timesteps and group members. WGRPO changes the reference constraint, replacing the usual KL regularizer with a Wasserstein distance between current and reference policy output distributions. The paper gives the following minimization objective:

\[ \mathcal{L}_{\mathrm{WGRPO}}(\theta) =-\mathcal{L}_{\mathrm{CLIP}}(\theta) +\beta W\!\left(\pi_\theta,\pi_{\mathrm{ref}}\right). \]

The first term seeks higher group-relative rewards, while the second constrains output-distribution deviation from the reference policy, with strength controlled by \(\beta\). The intended advantage is to constrain overall video behavior more directly than per-step density matching. Nevertheless, this depends on the ground metric used in output space: choosing Wasserstein distance does not itself make that metric semantic or perceptual. The text does not fully specify the implemented ground metric, transport estimation procedure, or how multidimensional rewards become scalar trajectory scores. These remain implementation details to verify for reproduction.

Loss & Training

The scorer, reward model, and generator are trained in sequence rather than jointly end to end. Video-Bench contains 8,799 unique videos, each assessed by four annotators across nine dimensions, with outputs from four open-source and three commercial generators. Policy training separately uses 50,000 prompts selected from VidProM.

The reward model uses Qwen2-VL-2B with LoRA rank 8, batch size 32, and learning rate \(2\times10^{-6}\). Policy optimization uses 20 sampling timesteps, 16 candidates per group, and LoRA rank 32 with alpha 64. All experiments use 8 H100 GPUs. Other hyperparameters follow VideoAlign; the unreported values of \(\lambda\), \(\beta\), and filtering thresholds should not be treated as known constants.

Reward evaluation uses a held-out set of 100 high-fidelity video pairs and reports pairwise accuracy both including and excluding ties. For generation evaluation, GPT-4o rewrites each original test prompt into three variants to mitigate prompt leakage. This introduces variation in wording but does not strictly establish semantic independence from the training data.

Key Experimental Results

Main Results

The following results come from Table 2. Values are percentages, with higher being better. Including and excluding ties are different evaluation protocols and should not be compared as equally difficult tasks. Equal denotes a matched-scale comparison; Full denotes the full raw-data baseline.

Reward Training Data	Action With Ties	Action Without Ties	Motion Effects With Ties	Motion Effects Without Ties
Raw (Equal)	54.65	74.24	67.68	80.30
Curated (Equal)	59.60	78.79	68.69	84.85
Raw (Full)	58.59	80.30	65.66	86.36
Calibrated	60.61	83.33	71.72	87.88

Curating data already helps at matched scale, so training-set size alone does not explain the gains. Against Raw (Full), calibration improves Action accuracy without ties by 3.03 percentage points and Motion Effects accuracy with ties by 6.06 points. This is an overall data-and-reward comparison, not the isolated contribution of the Wasserstein loss.

On generation, Table 1 reports Avg. increasing from 68.16 to 69.58 for Wan2.1-T2V-1.3B, from 66.39 to 68.32 for CogVideoX-2B, and from 63.86 to 65.92 for ModelScope. However, Subject Consistency decreases from 87.35 to 86.45 for CogVideoX and from 82.12 to 80.25 for ModelScope. Avg. is quoted as reported rather than reinterpreted as the simple arithmetic mean of the seven displayed submetrics.

Ablation Study

Table 3 isolates the Wasserstein term during reward learning. Values are percentages.

Reward Learning Configuration	Action With Ties	Action Without Ties	Motion Effects With Ties	Motion Effects Without Ties
Without Wasserstein loss	58.98	81.25	68.33	86.11
With Wasserstein loss	60.61	83.33	71.72	87.88

Motion Effects accuracy with ties improves by 3.39 percentage points, while Action accuracy with ties improves by 1.63 points. This supports adding a distributional constraint to regression, but does not attribute every gain to calibration or prove that the model captures the full distribution of human preferences.

The following five columns are selected from Table 4, comparing three policy regularization settings. These are the original reported scores, with higher being better.

Policy Constraint	Motion Smoothness	Dynamic Degree	Spatial Relation	Background Consistency	Reported Avg.
No regularizer	89.10	36.29	59.78	91.33	67.97
KL	88.89	36.50	59.40	91.45	68.02
Wasserstein	89.32	36.48	63.23	92.55	69.58

Relative to KL, Wasserstein improves reported Avg. by 1.56 points and Spatial Relation by 3.83 points, but Dynamic Degree is 0.02 points lower. The appropriate conclusion is an overall improvement across multiple metrics, not a win on every motion metric.

Key Findings

• Data quality and loss design have separate evidence: the matched-scale experiment tests curation, Table 3 tests the reward loss, and Table 4 tests the policy constraint. Their gains should not be conflated.
• Gains transfer across optimizers: Table 5 replaces the raw reward model with the calibrated model on Wan2.1-T2V-1.3B. DanceGRPO Avg. rises from 68.99 to 69.97, and FlowGRPO from 69.86 to 70.87.
• Figure 4b supports greater stability for pointwise than pairwise training under the paper's noisy conditions. Its individual plotted values are not sufficiently clear in the cache for reliable extraction, so no numerical curve values are invented here.

Highlights & Insights

• Reliable ratings are distinguished from excellent videos before preference learning. A consistently low-rated sample can therefore provide useful supervision rather than being excluded by a filter that keeps only highly rated videos.
• The two Wasserstein constraints have separate ablations. This makes it possible to distinguish stage-specific contributions instead of relying only on the performance of the complete pipeline.
• Matched-scale controls and cross-optimizer tests are useful experimental patterns. The former addresses data-volume confounding, while the latter tests dependence on a particular policy optimizer.

Limitations & Future Work

• Preference diversity may be suppressed by consensus filtering and proxy relabeling. This is a methodological risk identified in this note; retaining original rating distributions or calibrating separately for preference groups would help investigate it.
• Distributional definitions and implementation details are incomplete. A point-mass target does not directly represent multimodal disagreement, and the text does not sufficiently specify distribution parameterization, the ground metric, transport estimation, or reward aggregation. More precise definitions and matched comparisons against simple multidimensional regression are needed.
• Evaluation is limited in scale. Reward assessment uses only 100 video pairs, central results focus on two dimensions, and tables do not report repeated-run uncertainty. Larger independent human evaluations and confidence intervals would strengthen generalization claims and small reported gains.
• Quality dimensions trade off against one another. The paper acknowledges reduced subject consistency for some generators; follow-up work should report joint motion and subject-stability behavior and test dimension-specific constraints instead of optimizing only an aggregate score.
• Cost and evaluation independence need further verification. Experiments use 8 H100 GPUs without sufficiently quantifying the additional transport-computation cost, and prompt rewriting does not replace deduplication. Reporting compute overhead and semantically deduplicated evaluation would address both issues.

Related Work & Insights

• VideoScore / VisionReward also address fine-grained video quality and human preferences. This paper emphasizes consensus-based calibration connected to Wasserstein constraints at two training stages, rather than being the first to recognize multiple quality dimensions.
• VideoAlign / VideoDPO bring human feedback or direct preference optimization to video generation. The present work highlights the need to inspect label quality and reward representation before focusing on the choice of optimizer.
• FlowGRPO / DanceGRPO concern video policy optimization and serve here to test the transferability of calibrated rewards. Their gains in Table 5 demonstrate the independent value of reward-side improvements, not that these methods are universally inferior to WGRPO.
• A useful extension would separate annotation errors from genuine preference diversity, then compare point-mass targets, multimodal targets, and group-conditioned rewards. This would test the paper's distributional-learning claim more directly.

Rating

• Novelty: 4/5. The work connects consistency calibration, multidimensional rewards, and policy distribution constraints into a complete pipeline, while its components build on established ideas.
• Experimental Thoroughness: 3/5. Matched-scale controls, two-stage ablations, three generators, and cross-optimizer tests are useful, but the reward test set is small and uncertainty analysis is absent.
• Writing Quality: 3/5. The problem and pipeline are clear, but the definitions of distributional modeling and transport implementation are insufficient for reproduction from the text alone.
• Value: 4/5. The study foregrounds preference-data quality, a factor often obscured by optimizer choices, and offers testable training and evaluation designs.