Personalized Reward Modeling for Text-to-Image Generation¶

Conference: ECCV2026
Paper: ECCV Paper
Authors: Jeongeun Lee, Ryang Heo, Dongha Lee
Area: Image Generation
Keywords: personalized reward modeling, visual chain-of-thought, user context, adaptive evaluation dimensions, prompt optimization

TL;DR¶

PIGReward explains a user's historical image choices, induces personalized evaluation dimensions, and compares new images dimension by dimension, achieving 85.85% acc w/o tie on PIGBench while also using these preferences for prompt optimization.

Background & Motivation¶

Text-to-image models can generate images that match a prompt and look good overall, but broad appeal does not imply alignment with an individual user. Reward models such as PickScore and ImageReward primarily learn a shared judgment direction from aggregate human preferences. MPS and VisionReward decompose evaluation into multiple dimensions, but still use a predefined attribute space across users. For example, two people may like the same mountain image because one values composition and color while the other values realism and lighting. Recording only their identical final choice collapses these different reasons into a single label. Optimizing a shared criterion can therefore improve general aesthetics while moving away from details that matter to a particular user.

Existing personalized generation studies often evaluate outputs through similarity to previously liked images, but similarity can memorize subject matter rather than preference. Liking images of metal, vehicles, or space does not mean that the user wants the same large, smooth structures for every new prompt. Nearby image embeddings may reflect incidental appearance, whereas the stable preference concerns detailed texture, spatial organization, or symbolism. Simply feeding historical images to a vision-language model (VLM) is also insufficient: sparse pairwise records do not explain their causes, encouraging generic rationales. Per-user fine-tuning requires more feedback and is difficult to justify for users with short histories. The task is therefore to infer transferable comparison criteria from sparse choices, not to reproduce previously liked images.

The authors use language as an intermediate representation between historical records and reward judgments: explain known choices individually, then let the evaluator infer which dimensions matter. Users can differ both in their dimension sets and in how the model reasons about concrete visual evidence. This design does not require a separate parameter set for every user, but it still requires training a shared preference reasoner and reward model. Core Idea: convert sparse image preferences into textual user context, then induce evaluation dimensions and reason dimension by dimension, allowing the reward criteria themselves to vary across users.

Method¶

Overall Architecture¶

Inputs consist of historical prompts with preferred and rejected images, plus the current prompt and two candidate images. The preference reasoner \(\pi\) performs Preference Reasoning Bootstrapping on each historical record, producing a rationale for that choice and collecting the rationales into user context. The reward model \(\phi\) then performs Adaptive Dimension Evaluation, generating context-conditioned dimensions, explanations, and per-dimension scores before predicting a pairwise preference. Both components use the Qwen2-VL-7B backbone family, but serve different tasks and are trained separately. Personalized Prompt Optimization is a downstream extension: preferences over generated images are assigned back to their candidate prompts to train a prompt model. Basic evaluation inference requires historical context and target images, without updating model weights whenever a new user arrives.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    History["User preference history"] --> Reasoner["Preference Reasoning<br/>Bootstrapping"]
    Reasoner -->|Textual user context| Judge["Adaptive Dimension<br/>Evaluation"]
    Target["Prompt and candidate image pair"] --> Judge
    Judge --> Output["Dimension-wise rationales,<br/>scores, and pairwise preference"]
    Output -.->|Downstream training only: preference labels| Optimizer["Personalized Prompt<br/>Optimization"]
    Optimizer --> PromptModel["DPO training of prompt model"]

Key Designs¶

1. Preference Reasoning Bootstrapping: expand choice labels into transferable user context

A user's history is not a generic style label, but a collection of preferred and rejected images generated from the same prompt. The reasoner processes one record at a time and explains why the user selected the preferred image, instead of concatenating all historical images into a single multi-image input. Negative examples matter because a liked image alone cannot reveal whether an attribute caused the preference or was simply shared by both candidates. Pairwise contrast supplies evidence about differences, such as which image better expresses detail, emotion, or composition when both match the prompt. The resulting rationales are collected into textual context for the reward model. This context is a model-inferred explanation of choices, not a psychological profile explicitly confirmed by the user.

To reduce repetitive rationales from an off-the-shelf VLM, the authors construct opposing explanations through hint-driven sampling. For the same pair in a general preference dataset, one instruction requests a rationale supporting the genuinely preferred image, while another requests a rationale supporting the rejected image. The first rationale becomes the positive response and the second the negative response for direct preference optimization (DPO). A frozen reference model anchors the comparison, so optimization targets relative preferences over explanations rather than user-specific parameters. Bootstrapping here expands sparse choices into language; it does not create additional human-confirmed preference labels. The intended result is a more detailed and discriminative explanation, preventing every user's context from collapsing into generic statements about clarity, beauty, and prompt alignment. Processing images individually and then aggregating text also mitigates the limits of feeding many reference images into the reward model simultaneously.

2. Adaptive Dimension Evaluation: decide what matters before making a traceable pairwise judgment

The reward model receives user context, the current prompt, and two candidate images, without directly applying one universal dimension list. It first induces relevant evaluation dimensions from the context, then explains and scores the two images along those dimensions. Composition, lighting, and texture may matter for one user, while emotion, narrative, or symbolism may matter for another. These dimensions are part of the generated evaluation, not predictions from a separately trained fixed-category head. Dimension-wise rationales make the final choice traceable to comparison criteria, unlike a scalar reward or an unexplained preference for the first image. Scores are subsequently aggregated into a final image preference; Figure 2 illustrates summation across dimensions rather than a separately learned weight vector.

The following notation summarizes the summation rule in the prose, where \(s_{k,1}\) and \(s_{k,2}\) are the two image scores on dimension \(k\), and \(K\) is the number of generated dimensions:

\[ \Delta = \sum_{k=1}^{K}(s_{k,1}-s_{k,2}). \]

A positive difference means the first image has the higher total, and a negative difference favors the second; this is a compact description, not a reconstruction of damaged training equations. The problem formulation encodes preferences as \(+1\) and \(-1\), but the readable method does not adequately specify how equal totals are resolved. Consequently, the exact implementation of the two tie protocols in Table 2 cannot be inferred from this summation rule.

Reward-model training starts with user contexts and target image pairs carrying human preference labels. GPT-4o produces chain-of-thought (CoT) trajectories containing dimensions, dimension-wise rationales, score aggregation, and the final choice. Rule-based filtering discards malformed outputs, choices inconsistent with human labels, outputs with fewer than 3 dimensions, and inconsistent score sums. The resulting dataset contains 5K structured samples, which are distilled into \(\phi\) through supervised fine-tuning (SFT). This filtering establishes label and format consistency, but cannot guarantee that every rationale faithfully reflects the user's actual reasoning. In the reader's interpretation, adaptive dimensions prevent fixed criteria from overlooking preferences, while dimension-wise CoT applies those criteria to the current images; the two mechanisms serve distinct purposes.

3. Personalized Prompt Optimization: convert image preferences into prompt-training signals

The downstream prompt model expands an initial prompt into two candidates, and an image generator renders an image from each. PIGReward compares the resulting images using the corresponding user's historical context. The prompt producing the winning image is labeled chosen, the other rejected, and these prompt pairs support DPO training. The prompt model is conditioned on the initial prompt and user features, allowing the same request to be expanded toward different visual preferences. The reward does not demand maximal resemblance to historical favorites; it evaluates whether the new image meets criteria inferred from that history. Experiments use SDXL for rendering; the directly optimized component in this workflow is the prompt language model, not SDXL's weights.

A Worked Example¶

Figure 6 illustrates a challenging case: a user's liked images contain metallic close-ups, vehicles, and space-related visual elements. A similarity metric favors the second candidate because its large, high-contrast, smooth structure resembles historical appearances. Preference Reasoning Bootstrapping instead attempts to explain each preferred-versus-rejected comparison, rather than simply memorizing metal as a subject category. Adaptive Dimension Evaluation then attends to detail, composition, and symbolism, correctly selecting the first image in this example. The historical record changes from a list of visual elements to reproduce into criteria for evaluating new content. Figure 6 does not provide complete dimension-wise scores in the available text, so no example scores or verbatim rationales are invented here. For prompt optimization, this pairwise choice could also label a prompt-training pair, but that continuation illustrates the mechanism rather than an experiment reported in Figure 6.

Loss & Training¶

The preference reasoner uses DPO to favor correct over reversed explanations, while the reward model learns complete CoT outputs through an autoregressive supervised objective. The distinction is that the former compares relative probabilities of two rationales, whereas the latter imitates filtered structured answer sequences. The downstream prompt model separately uses DPO to learn preferred expansions, applying the same optimization framework to a different output object. The paper explicitly states that Pick-a-Pic splits for reasoner training, reward-model training, and evaluation are strictly disjoint. The available prose does not adequately specify whether this separation covers examples, prompts, or user identities, so it does not establish complete user-level non-overlap. The text extraction of Equations (3), (4), and (6) is damaged; guessed standard DPO or SFT equations are not presented as the authors' exact formulas. Training epochs, learning rates, and inference costs are also not fully specified in the readable prose, so a 7B backbone and 5K trajectories alone do not establish low deployment cost.

Key Experimental Results¶

Main Results¶

Table 2 on page 10 reports preference-prediction accuracy; the original acc w/ tie and acc w/o tie columns are retained below, both in %. Each dataset cell is ordered as "w/ tie / w/o tie"; the protocols are not mixed, and tie handling is not fully defined in the available prose.

Method	User references	Pick-a-Pic	PIP	PASTA	PIGBench
UnifiedReward-Think	No	59.92 / 59.92	35.28 / 35.21	65.71 / 65.71	66.04 / 66.04
Qwen2.5-VL-7B	Yes	61.52 / 61.52	49.95 / 49.95	50.34 / 50.34	54.72 / 54.72
GPT-4o	Yes	46.90 / 46.70	76.23 / 76.54	68.80 / 68.80	64.15 / 68.29
PIGReward	Yes	63.76 / 62.14	77.84 / 78.59	75.43 / 74.43	84.91 / 85.85

Table 1 on page 9 reports reference sizes of 8, 5, and 4 for Pick-a-Pic, PIP, and PASTA, with 10,000, 1,002, and 5,143 users, respectively. PIGBench starts from abstract prompts across 12 categories, expands each into 4 variants with GPT-4o, and renders them with SDXL; manual filtering retains 100 sets and 75 user records. Each annotator ranks 5–15 image sets, and evaluation compares the highest- and lowest-ranked images among the 4 candidates in each set. PIGBench accuracy of 85.85 versus GPT-4o's 68.29 represents a 17.56 percentage-point gain under the same w/o tie protocol, not accuracy on arbitrary image pairs or complete rankings.

Ablation Study¶

The following results come from Table 4 on page 12 and report accuracy (%); full-model values match Table 2's w/ tie column, although Table 4 does not itself label a tie protocol.

CoT	DPO-trained reasoner	SFT-trained reward model	Pick-a-Pic	PIGBench
No	Yes	No	44.43	39.62
Yes	No	No	48.94	56.94
Yes	Yes	No	45.63	70.83
Yes	No	Yes	49.98	58.49
Yes	Yes	Yes	63.76	84.91

With the trained reasoner and CoT fixed, adding reward-model SFT raises Pick-a-Pic from 45.63 to 63.76, a gain of 18.13 percentage points. However, adding reasoner DPO alone lowers Pick-a-Pic from 48.94 to 45.63 while raising PIGBench from 56.94 to 70.83, so each training stage does not improve every dataset monotonically. Table 3 on page 12 separately compares absent, fixed, and generated dimensions before reward-model SFT; the authors report the best result from generated dimensions with CoT, indicating that distillation alone does not explain the gains.

Key Findings¶

Page 13 reports 671 dimensions and 2,609 dimension combinations on Pick-a-Pic, versus 23 and 42 on PIGBench; lexical diversity does not automatically establish semantic independence.
The authors define explanation alignment as the fraction of generated evaluation dimensions grounded in users' written explanations, reporting 83.34%; this is not final image-choice accuracy.
Figure 10 on page 14 evaluates prompt optimization using both PIGReward and the original PIGBench annotators; readable win rates are absent from the extracted text, so only the authors' reported advantage is recorded without invented numbers.

Highlights & Insights¶

Personalization operates at the level of selecting evaluation criteria, not just scoring under fixed criteria. This provides an explicit representation for different reasons behind identical choices.
Converting historical image pairs to text individually separates visual comparison from long-history processing. It can reduce simultaneous multi-image input pressure, although a full latency benchmark is not provided.
Reward outputs can explain a judgment and supervise prompt selection. The transferable idea is history-explanation-driven candidate comparison, rather than a particular fixed list of aesthetic dimensions.

Limitations & Future Work¶

PIGBench contains only 75 user records and filters out sets unlikely to elicit personalized disagreement; highest-versus-lowest comparisons also emphasize clear preferences, leaving close candidates, additional generators, and long-term preference changes insufficiently tested.
The automatic evaluator for prompt optimization is the same PIGReward supplying training preferences, creating a self-evaluation risk; human judgments offer a separate signal source but use the same PIGBench annotators rather than entirely new users.
Language rationales can be post-hoc explanations, and filtering checks answers, format, and arithmetic without proving each visual or psychological attribution; the reader suggests user correction and rationale-intervention experiments.
Table 2 names the reference baseline Qwen2.5-VL-7B, whereas the settings name Qwen2-VL-7B; the proposed model follows the method section, and the table retains its original label rather than silently merging the versions.
The extracted project link lacks a credible complete domain, so no code or arXiv link is inferred; uncertainty in training equations, Figure 10 win rates, and tie implementation is retained as noted above.
The authors propose using these rationales as natural-language feedback for more interactive generation; this is future work, not a validated multi-turn system in the paper.

vs PickScore / ImageReward: general reward models estimate aggregate human preference, whereas PIGReward explicitly reads user history; reference-free and reference-conditioned methods have different information budgets, so the full gap cannot be attributed to backbone capability.
vs MPS / VisionReward / UnifiedReward-Think: these methods offer multi-dimensional or reasoning-based evaluation, while PIGReward additionally conditions dimensions on user context; pre-SFT ablations test this design more directly than overall rankings alone.
vs PIP / ViPer / PPD: related work emphasizes prompt rewriting, user-feature injection, or personalized generation, while this paper first builds an individual-preference evaluator and then connects it to prompt optimization.
Research insight: comparing user-confirmed dimensions with bootstrapped dimensions could test whether gains come from recovering true preferences or from language priors; this is a reader-proposed experiment, not a reported result.

Rating¶

Novelty: 4/5, adaptive dimensions combined with historical explanations address a specific problem, while the components reuse DPO, SFT, and CoT.
Experimental Thoroughness: 4/5, covering 4 datasets, component ablations, and human evaluation, with remaining limits in benchmark size and protocol clarity.
Writing Quality: 3/5, a clear central argument with inconsistent model naming and some underspecified protocols; equation-extraction damage is a separate limitation of the available reading material.
Value: 4/5, an interpretable reward interface for personalized generation, with deployment cost, robustness, and user-correction mechanisms still requiring validation.