Multi-dimensional Preference Alignment by Conditioning Reward Itself

Conference: ECCV2026
Paper: ECCV Paper
Area: Image Generation / Alignment & RLHF
Keywords: multi-dimensional preference alignment, conditional rewards, diffusion models, DPO, classifier-free guidance

TL;DR

MCDPO conditions a single diffusion model on per-axis preference outcomes so that conflicting image pairs provide directionally explicit supervision and enable inference-time control; on PickV2, the six-metric average win rate for SD1.5 increases from 66.74% with Diffusion-DPO to 81.47%.

Background & Motivation

Two candidates from a text-to-image model rarely consist of one universally good image and one universally bad image. One may look more appealing while omitting an object in the prompt; the other may faithfully implement the text despite an ordinary composition. Diffusion-DPO uses an overall preference label to adjust image probabilities relative to a reference model, without directly identifying which attributes motivated that preference. Collapsing several evaluation axes into a scalar can suppress useful local attributes of the globally losing sample along with its overall probability. This paper addresses such multi-dimensional preference conflicts, not the generation of better textual answers by language models.

Existing approaches can train separate experts and merge their parameters, or retain only pairs with a consistent winner across multiple axes. The former adds model management and combination costs, and parameter averaging does not reliably compose reward directions; the latter discards complementary attribute information. The conflict-free filtering regime discussed in the paper discards 82% of the training data, illustrating that consistent labels are not a cheap resource. More importantly, a conflict reveals that aesthetic quality and prompt fidelity are different concepts, so discarding conflicts also removes opportunities to distinguish them. The authors therefore seek to retain per-axis outcomes within pairwise training instead of repairing an aggregated reward afterward.

The key is to change what the model is conditioned on: rather than merely declaring one image better overall, specify the axes on which the comparison should be positive or negative. The same image pair receives opposite ranking constraints under opposite conditions, encouraging the network to change its implicit reward with the condition. This representation also gives training supervision and inference-time control a shared interface. Core Idea: condition a diffusion model on a vector of per-axis wins, losses, and ties, exploit conflicts through conditional DPO, and steer generation using differences between the resulting conditional distributions.

Method

Overall Architecture

Training takes two images for the same prompt, the original human preference label, and per-axis comparisons supplied by proxy reward models. The implemented configuration has 5 axes: human preference, Aesthetic, CLIP, HPSv2, and PickScore. The output is one diffusion model with a reward-conditioning branch, not 5 independent experts. The pipeline constructs preference outcome conditions, processes them with context-aware reward conditioning, and then trains with symmetric conditional DPO and dimensional dropout. At inference time, inputs become a prompt, random noise, and user-specified reward directions; neither candidate pairs nor external reward models are needed. Supervised initialization through MCSFT teaches the new branch to respond to conditions, but does not replace subsequent preference optimization.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    PAIR["Image pairs and per-axis labels"] --> OUTCOME["Preference outcome conditions"]
    OUTCOME --> CONTEXT["Context-aware reward conditioning"]
    INPUT["Prompt and noisy latent"] --> CONTEXT
    CONTEXT --> TRAIN["Symmetric conditional DPO<br/>and dimensional dropout"]
    TRAIN --> MODEL["Trained single model"]
    MODEL --> GUIDE["Inference-time reward guidance"]
    CONTROL["Prompt, noise,<br/>and target reward directions"] --> GUIDE
    GUIDE --> IMAGE["Generated image"]

Key Designs

1. Preference outcome conditions: encode conflicts instead of erasing them

The authors define a preference outcome vector for an image pair, with each entry representing a win, loss, or tie. Here, \(\rho\) denotes the paper's outcome vector and \(r_i\) denotes reward axis \(i\), avoiding ambiguous Greek characters in the extracted text. The key to Equation (9) is not to average independently predicted probabilities, but to correct the direction of each reward difference inside the sigmoid using its outcome. An image that wins aesthetically but loses semantically receives a vector containing both positive and negative entries. The first image is therefore preferred under that particular condition, not declared superior on every dimension.

\[ \rho_i(x^w,x^l)=\operatorname{sign}\!\left(r_i(x^w,c)-r_i(x^l,c)\right),\qquad p^*_{\mathrm{BT}}=\sigma\!\left(\sum_i w_i\rho_i\left[r_i(x^w,c)-r_i(x^l,c)\right]\right). \]

For nonnegative weights, each corrected contribution corresponds to the absolute reward difference; tied axes contribute no direction. However, constructing a conditional expert for every axis still requires separate likelihood-ratio evaluations, with computation increasing with dimensionality. MCDPO instead feeds the complete outcome vector into one policy, representing its combinations through a conditional implicit reward. This avoids a separate training computation for every axis, but does not imply that a sample pair needs only one network call. The authors' optimal reward-decomposition argument assumes sufficient capacity and expressiveness; approximate decomposition in a finite network still requires empirical validation.

2. Context-aware reward conditioning: let reward labels observe the image-text content they evaluate

The outcome vector is first encoded as natural-language tokens, such as win tie for two axes, and passed through the text encoder to obtain an initial reward embedding. The words indicating better semantic alignment cannot determine a denoising direction alone: the target text and the current image's deficiencies depend on the example. The context-aware reward module first uses attention to combine the reward embedding with the prompt, then attends to the current image latent, and finally applies an MLP. The resulting reward representation is therefore not a fixed user-preference vector, but a contextual representation that changes with the prompt and noisy image state. It particularly supports axes such as CLIP that require joint understanding of text and image.

The U-Net retains its original prompt cross-attention and adds a parallel reward cross-attention branch, summing their outputs. The new attention branch copies the pretrained cross-attention structure, and its weighting coefficient is set to 1 in the experiments. The paper emphasizes zero initialization of the new modules to preserve pretrained knowledge, so reward conditions initially have no effect. MCSFT first freezes the U-Net and trains the conditioning modules to reconstruct preferred images under winning conditions, providing a usable initialization for the interface. MCDPO follows afterward; appendix-level training hyperparameters are absent from the supplied main-text cache, so learning rates and training steps cannot be supplied reliably.

3. Symmetric conditional DPO and dimensional dropout: learn both rankings without letting easy axes dominate

Each image pair participates in two corresponding tasks: increase the first image's relative probability under \(\rho\), and increase the second image's under \(-\rho\). Unlike uniformly suppressing the globally losing image, this gives its useful attributes positive supervision under the appropriate condition. To present Equation (17) compactly, let \(\Delta_\theta\) denote the difference between the images' conditional policy/reference log ratios, with \(\beta\) controlling the preference-optimization scale. The reference policy receives only the prompt, whereas the training policy also receives the outcome vector; reversing the images must also reverse the condition. With this notation, the two-direction objective is:

\[ \begin{aligned} \Delta_\theta(\rho) &=\beta\left[\log\frac{p_\theta(x^w\mid c,\rho)}{p_{\mathrm{ref}}(x^w\mid c)} -\log\frac{p_\theta(x^l\mid c,\rho)}{p_{\mathrm{ref}}(x^l\mid c)}\right],\\ \mathcal L_{\mathrm{MC}} &=-\mathbb E\left[\log\sigma\!\left(\Delta_\theta(\rho)\right) +\log\sigma\!\left(-\Delta_\theta(-\rho)\right)\right]. \end{aligned} \]

The diffusion implementation follows Diffusion-DPO by expressing policy/reference comparisons through denoising errors on noisy samples, rather than explicitly evaluating a tractable full-image density. Conditioning resolves ranking-direction ambiguity, but does not automatically balance optimization: easily learned aesthetic features can saturate the aggregated sigmoid first. The authors also identify domination by dimensions with large gradients, leaving inadequate updates for semantic dimensions that have not yet been learned well. Dimensional dropout randomly sets some outcome entries to 0 during training, preventing the network from relying on the same easy axis in every comparison. Here, 0 can indicate either a genuine tie or random masking; it means that the current condition does not request optimization of that direction, not that image quality is zero. The interpretation that masking removes an axis's contribution relies on the paper's reward-decomposition model, rather than guaranteeing independent gradients in an arbitrary neural network.

4. Inference-time reward guidance: control sampling through winning and losing conditional distributions

After training, the same network produces different denoising predictions under different preference conditions, corresponding to the scores of different conditional distributions. For the same prompt and image state, the score difference between winning and losing conditions can be interpreted as the gradient direction of an implicit reward difference. Equation (20) gives the key relationship below, where \(r^w\) and \(r^l\) are implicit rewards under the two conditions, not human labels for an image yet to be generated.

\[ \nabla_x\log p_\theta(x\mid c,\rho^w) -\nabla_x\log p_\theta(x\mid c,\rho^l) =\frac{1}{\beta}\nabla_x\left(r^w(x,c)-r^l(x,c)\right). \]

The authors incorporate this difference into classifier-free guidance (CFG), moving sampling away from undesired reward directions and toward desired ones. Users can emphasize aesthetics, semantics, or a combination without running an external reward scorer on images at inference time. This does not imply unchanged inference cost, since comparing conditions still involves different denoising predictions. Conditions can also vary across timesteps: Aes-CLIP Control sets aesthetics to a tie for the first 25 steps and switches it to a win for the remaining 25 steps. The rationale is to prioritize semantic structure early and enhance details later, rather than imposing identical aesthetic pressure at every denoising stage.

A Worked Example

This is an illustrative mechanism example, not an additional quantitative experiment from the paper: a prompt requests a red chair; candidate A has attractive composition but the wrong chair color, while candidate B has the correct color but ordinary composition. Considering only the aesthetic and semantic axes, A versus B has outcome \([+1,-1]\), and the reverse comparison has \([-1,+1]\). A is not treated as a correct-color example merely because it wins overall, and B does not lose all positive supervision merely because it loses overall. The reward module combines the prompt and current latent to learn different preference directions under these conditions, while the symmetric objective requires both rankings to hold. Masking the aesthetic axis during training yields \([0,-1]\), so composition alone can no longer explain the current comparison. At generation time, the user provides a new prompt and desired directions such as \([+1,+1]\), and sampling starts from random noise. This example does not guarantee a perfect combination of composition and correct color; it explains why training can preserve complementary information from both candidates.

Loss & Training

The training dataset is Pick-a-Pic v2, originally containing approximately 1M human-preference image pairs. Scores from the four proxy axes are normalized on the training data and discretized into 100 bins, with scores in the same bin treated as ties; these outcomes join the human label in a 5-dimensional condition. Discretization reduces the treatment of tiny score differences as definite outcomes, but discards difference magnitudes and depends on reward calibration and binning. SD1.5 uses approximately 180K pairs, or 18% of the data; SDXL uses approximately 30K pairs, or 3%. The main text reports total training costs including MCSFT of 16 and 32 A100 GPU hours respectively, which should not be interpreted as end-to-end annotation and preprocessing costs. The main-table baselines use the full dataset, so the sample-efficiency comparison includes different data budgets; Table 3 separately compares methods under an identical 18% budget. The conflict-data experiments in Table 3 provide directional evidence, but several rows are interleaved in the supplied extraction, so unclear complete rows are not reconstructed below.

Key Experimental Results

Main Results

The following selection comes from Table 1 on page 9 and Table 2 on page 10; all results are PickV2 test-set win rates in %, with higher being better. Each column is the proportion of generations judged better than the corresponding original SD1.5 or SDXL baseline by that evaluator, not the direct pairwise win rate of MCDPO against DPO. Average is the paper's reported six-metric average; ImageReward and MPS were not directly used to construct training conditions.

Backbone	Method	PickScore	Aesthetic	HPSv2	CLIP	ImageReward	MPS	Average
SD1.5	Diffusion-DPO	75.52	65.08	70.28	57.80	64.44	67.33	66.74
SD1.5	MCDPO	86.20	91.88	93.44	57.64	82.92	76.75	81.47
SDXL	Diffusion-DPO	71.60	49.20	72.92	61.24	68.64	60.36	63.99
SDXL	MCDPO	74.92	63.40	85.84	58.16	77.72	64.31	70.72

Average win rate increases by 14.73 percentage points for SD1.5 and 6.73 percentage points for SDXL, but CLIP decreases by 0.16 and 3.08 percentage points respectively. The appropriate conclusion is a substantial improvement in aggregate preference metrics with a semantic-metric trade-off, not uniform superiority on every axis.

Ablation Study

The following dimensional-dropout comparison is selected from Table 8 on page 14, with the full-model metrics cross-checked against the ALL row in Table 6; the setting is SD1.5 on PickV2, and all values are win rates in %. Only two clearly identifiable rows are included; other ablation values interleaved with adjacent rows in the extraction are not guessed.

Config	PickScore	Aesthetic	HPSv2	CLIP	ImageReward	MPS	Average
MCDPO, paper default	86.2	91.8	93.4	57.6	82.9	76.7	81.1
reward dropout 0.0	87.3	96.0	92.1	49.2	79.2	77.1	80.1

Removing dropout raises aesthetics by 4.2 percentage points but reduces CLIP by 8.4 percentage points, supporting the interpretation that easier axes can dominate optimization. This evidence indicates that dropout helps balance axes, but two output metrics alone cannot establish strictly disentangled internal representations.

Table 6 on page 13 additionally compares inference-control settings for the same MCDPO model, again using SD1.5 and PickV2 win rates in %.

Inference target	PickScore	Aesthetic	HPSv2	CLIP	ImageReward	MPS	Average
ALL	86.2	91.8	93.4	57.6	82.9	76.7	81.1
Aesthetic only	77.3	95.8	79.1	49.3	73.1	68.8	73.8
CLIP only	76.4	76.5	85.1	59.0	75.0	67.8	73.3
Aes-CLIP Control	88.0	90.6	91.2	59.6	81.0	78.4	81.5

Key Findings

• Temporal scheduling raises CLIP from 57.6% under ALL to 59.6%, with the reported average increasing from 81.1% to 81.5%, but HPSv2 falls from 93.4% to 91.2%.
• Aesthetic-only guidance reaches 95.8% aesthetics but 49.3% CLIP, showing that controllability does not mean simultaneous improvement on all dimensions.
• The paper contains reporting inconsistencies: Table 1 gives an average of 81.47, while analysis Tables 6/8 give 81.1; the latter also differs from the arithmetic mean of its six entries, so the reported values are preserved rather than silently corrected.
• The page 14 discussion uses 95.8/49.3 for removing dropout, whereas the corresponding Table 8 row gives 96.0/49.2; this note uses the table values and explicitly distinguishes the two.

Highlights & Insights

• Conflicting labels can supervise distinguishable attributes rather than merely constitute noise to filter out. Symmetric conditional comparisons preserve valuable features in the globally losing sample.
• A shared conditioning interface connects training and inference-time control. Instead of a fixed policy producing a single compromise, the model represents selectable conditional distributions.
• The context-aware module connects desired quality to the actual image and text. It avoids treating semantic preference as a static label detached from the input.
• Timestep-dependent control offers a finer intervention than fixed reward weights. Early semantics followed by later aesthetics is an empirically supported schedule, not a cost-free improvement on every metric.

Limitations & Future Work

• The authors explicitly acknowledge dependence on proxy rewards: vector quality is limited by scorer accuracy, and additional scoring creates preprocessing overhead.
• The theoretical decomposition assumes adequate capacity and conditioning expressiveness; experiments with 5 axes do not guarantee stability with more axes, rare condition combinations, or strongly correlated rewards.
• Several training axes also serve as evaluation metrics; improvements on ImageReward/MPS provide additional evidence but do not replace independent large-scale human preference validation.
• The supplied cache contains the main paper and references but not the cited appendix, preventing verification of all hyperparameters, reward-agreement analyses, or additional temporal-control experiments.
• Reader recommendation: test conflict utilization with more independent human multi-attribute labels, and report proxy-labeling cost, guidance computation cost, and per-axis confidence intervals.

Related Work & Insights

• vs Diffusion-DPO: the baseline uses overall rankings, while this method conditions rankings on per-axis outcomes; both perform preference optimization relative to a reference policy.
• vs reward-expert merging and Conditional Language Policy: parameter composition places multi-objective control at model combination, whereas MCDPO puts it inside one diffusion network's conditioning interface; this paper provides no LLM experiments.
• vs conflict-free data filtering: filtering avoids inconsistent directions but loses complementary information; this method learns attribute differences from conflicts at the cost of per-axis labels and additional conditioning modules.
• vs Rewards-in-Context: both use reward conditions, but this work designs a symmetric objective around directional conflicts in pairwise DPO rather than simply inserting reward scores into SFT inputs.
• Classification recommendation: image_generation is more appropriate. The method operates on U-Nets, image latents, and diffusion sampling, and all experiments concern text-to-image generation; the assigned path is retained, leaving any paired move to the coordinating workflow.

Rating

• Novelty: 4/5. Connects per-axis outcome conditions, symmetric ranking, and inference control to address information compression in multi-preference DPO.
• Experimental Thoroughness: 4/5. Covers two diffusion backbones, three prompt test sets, and several ablations, but the missing appendix and inconsistent reporting limit verification.
• Writing Quality: 3/5. Clearly motivates the modules, but some table and prose values conflict and require careful separation.
• Value: 4/5. Useful for controllable image preference alignment; transfer to language models or broader reward axes remains to be validated.