PIPBench: A Profile-Inclusive Framework for Personalized Image Generation Evaluation

Conference: ECCV2026
Paper: ECCV Paper
Project: PIPBench
Area: Image Generation
Keywords: personalized generation, user profiles, synthetic agents, preference evaluation, conditioning fusion

TL;DR

PIPBench incorporates psychological and lifestyle profiles into preference data construction and evaluation, comparing personalized image generation on 1,369 test cases and finding that strong VLM-based textual conditioning fusion outperforms the tested direct-reference approaches while still trading instruction fidelity for preference alignment.

Background & Motivation

Conventional text-to-image models can follow what to depict without knowing how a particular user would like it depicted. Short prompts usually specify a subject or scene, leaving color, composition, mood, and style to the model. Even when users supply several favorite images, copying one image's style or content does not establish an understanding of recurring preferences across the collection. Those preferences may also depend on context rather than a single fixed style label.

Previous approaches explore per-user fine-tuning, reference-image conditioning, and rewriting based on historical prompts. Their evaluation often relies on isolated images, comments, or domain-specific interactions. Without broader user context, it is difficult to distinguish genuine preference learning from reproducing incidental objects in the references. Psychological characteristics and lived experience offer additional clues, but their relationship to taste should be treated as an associative prior rather than a deterministic rule about individuals.

The paper therefore combines real-user data with synthetic agents: the former supplies human-confirmed choices, while the latter broadens profile coverage and produces training examples. Core Idea: use profiles to organize diverse but coherent collections of preferred images, then evaluate whether generated outputs fit a specific person rather than merely resemble a particular reference.

Method

Overall Architecture

PIPBench primarily contributes a benchmark and a data construction protocol, not a new image generation backbone. It defines profiles, collects real users and samples synthetic agents, generates profile-conditioned candidate images, and forms test cases containing reference images, a short prompt, and a held-out target image.

Profiles are used during data construction and are available to persona-aware judges. However, the standard generation task receives a short prompt \(X\) and at most \(K\leq5\) preferred reference images \(\{I_k\}_{k=1}^{K}\), not the full questionnaire. The evaluated model generates an image \(Y\) that should satisfy both the requested content and implicit aesthetic preferences. Keeping these information pathways separate is essential to interpreting the term profile-inclusive.

The benchmark compares per-user test-time tuning, direct joint conditioning, VLM conditioning fusion, and separate conditioning. The authors also train prompt rewriters and preference compressors with synthetic data to test whether the construction pipeline improves performance on real users.

Key Designs

1. Profile definition and sampling: represent the sources of implicit preferences

Profiles combine psychological measures with personal context. The psychological component includes Openness from the Big Five, TIPI for estimating Openness, Schwartz values, EVT for experience-associated color preferences, and CMA for valence and arousal. Personal context includes academic background, living arrangements and lifestyle, digital content exposure, interests, and clothing preferences. These fields provide associative cues for candidate generation instead of restricting sampling to explicit labels such as warm colors or oil painting.

Real profiles are collected through a 19-item online questionnaire, including an image-upload item, yielding 134 valid responses after filtering. Synthetic profiles are randomly sampled from a unified schema and checked with hard and soft rules. Hard contradictions trigger rejection; soft conflicts accumulate penalties. Checks cover rating coherence, value antagonisms, cross-domain behavioral consistency, and compatibility between AI usage and visual consumption.

To avoid repeatedly accepting near-identical personas, Algorithm 1 compares each candidate's signature against the most recent 200 accepted signatures using Jaccard similarity. Its acceptance condition can be written following the original algorithm as:

\[ \operatorname{penalty}(x)\leq\tau_{\mathrm{conf}},\qquad \max_{s\in\operatorname{Sig}(D_{\mathrm{agents}})_{\mathrm{last}\,200}} \frac{|\operatorname{Sig}(x)\cap s|}{|\operatorname{Sig}(x)\cup s|} \leq\tau_{\mathrm{jac}}. \]

The signature is a set representation of profile attributes, not an image embedding. Figure 2 labels the similarity filter as 42%, but the main text does not fully specify signature encoding and all threshold settings. Reproduction still requires the supplementary material; this condition alone does not implement the entire generator.

2. Profile-conditioned preference sets: separate candidate generation from human or automatic selection

The LLM follows three reasoning stages: organize demographic and lifestyle attributes into a personal narrative, use psychological frameworks to infer cognitive and emotional tendencies, and generate a profile-conditioned space of image prompts. After parsing and deduplication, each user has up to 20 stylistically coherent enriched prompts, which are rendered using a diffusion model such as Qwen-Image. A user's preferences can consequently span different subjects rather than variations of one reference image.

Real participants select 6–8 favorite candidate images and may upload additional images, yielding 76 calibrated user records after quality control. Synthetic agents instead use aesthetic-score ranking to select images. These sources should not both be described as observed human preferences: synthetic personalization first enters through the profile-conditioned candidate pool, and subsequent aesthetic ranking is not equivalent to a person's own choice.

Each test case samples at most 5 references from a preference pool and holds out another image as \(Y_{\mathrm{gt}}\). A VLM produces a short prompt \(X\) describing only the target's primary visual content. Omitting detailed aesthetic instructions forces the evaluated model to infer preferences from the references. The target constructs the test case; it should not be mistaken for the sole comparison image in metrics such as CLS-R.

The complete benchmark contains 251 users or agents, 1,876 images, and 1,369 test cases. The synthetic subset contains 175 agents, 1,231 images, and 719 cases; the real-user subset contains 76 users, 645 images, and 650 cases. Test-case counts differ from image counts and should not be interpreted as numbers of independent users.

3. Hybrid evaluation: distinguish prompt fidelity, reference alignment, and personal taste

CLS-T measures CLIP similarity between the generated image and the short prompt, checking whether the requested content remains correct. CLS-R and DIS-R average CLIP and DINO similarities between the generated image and the reference set; LPIPS-R reports the corresponding average perceptual distance. CLS and DIS are displayed after multiplication by 100 and are higher-is-better; LPIPS is lower-is-better. These measures detect use of visual conditions but cannot independently establish that a user likes the output.

The real-user subset additionally uses persona-aware Elo. A judge receives the complete profile, adopts that user's perspective, and compares two generated images for aesthetic compatibility. Image order is randomly swapped to mitigate position bias. GPT-5, Gemini 2.5 Pro, and Qwen3-VL issue separate judgments, each entered as an independent pairwise observation rather than collapsed through majority voting.

The authors report approximately 91% agreement between sampled judgments and human annotations. They also sample 100 cases for people to choose the best output among the top four models in Table 1, finding broadly consistent rankings. The 91% figure measures judge agreement, not generation success; the 100-case study does not cover all 650 real-user test cases.

4. Preference encoding comparisons: distinguish textual rewriting from visual condition injection

The no-preference baseline directly uses Qwen-Image, while DreamBooth fine-tunes on each user's reference set at test time. Joint conditioning uses Qwen-Image-Edit with either 1 or 2 randomly sampled references. VLM conditioning fusion asks GPT-5, Gemini, or differently sized Qwen VLMs to interpret the short prompt alongside multiple references and express inferred preferences in an enriched prompt for the same Qwen-Image generator. GPT-5 acts as a rewriter here, not as the image generator.

Fabric represents separate conditioning but uses a weaker image backbone; the authors explicitly caution against interpreting its lower score as a failure of the entire paradigm. Among trainable approaches, instruction tuning learns to recover the original enriched prompt from reference images and the short prompt. Preference compression instead uses a VLM to extract compact preference tokens, enriches them with dense multi-image features through a lightweight cross-attention compressor, and injects them into DiT blocks through in-block adapters. To control the cost of VLM usage, this route does not additionally perform caption enrichment.

Loss & Training

Instruction tuning uses Qwen2.5-VL-7B and Qwen2.5-VL-32B. Its supervision is the original enriched prompt \(E\) used to generate the target image; inputs are the short prompt \(X\) and reference images. Training data comes entirely from synthetic agents, so real-user evaluation tests transfer between data sources rather than fitting real-user training examples.

Preference compression compares a frozen VLM with projection-side training against a learnable configuration that unfreezes the VLM. The main paper describes the architecture and training objective in words but delegates learning rates, training steps, optimizer, exact training-set size, and compressor details to supplementary Sections D and E. Those appendices are absent from the available full-text cache, so unverified hyperparameters or loss formulas are not supplied here.

Key Experimental Results

Main Results

The following selection reproduces real-user results from the original Table 1. Conditioning fusion and the main baselines are organized around Qwen-Image, with the separately discussed Fabric exception. Metric directions follow the paper.

Method	CLS-T ↑	LPIPS-R ↓	CLS-R ↑	DIS-R ↑	Elo ↑
no-preference	32.677	0.7448	62.761	12.099	1427
DreamBooth	31.756	0.7265	63.751	12.720	1452
Qwen-Image-Edit (1-Ref)	31.048	0.7209	65.802	18.459	1521
Qwen-Image-Edit (2-Ref)	30.559	0.7511	65.023	16.821	1354
GPT-5 conditioning fusion	30.550	0.6867	69.574	22.160	1765
Gemini2.5-Pro conditioning fusion	30.403	0.6910	69.174	20.090	1615
QwenVL2.5-70B conditioning fusion	29.797	0.7092	66.835	20.282	1531

GPT-5 conditioning fusion gains 338 Elo over no-preference, but CLS-T decreases from 32.677 to 30.550. An additional reference is not necessarily helpful: Qwen-Image-Edit drops from 1521 to 1354 Elo, indicating unreliable aggregation of multiple references in the tested model.

Ablation Study

Real-user results in the original Table 3 examine training and freezing strategies. The selection below keeps representative configurations without treating different parameter scales as the same ablation.

Config	CLS-T ↑	LPIPS-R ↓	CLS-R ↑	DIS-R ↑	Elo ↑
QwenVL2.5-7B, not fine-tuned	30.510	0.7320	65.460	17.275	1492
Qwen2.5VL-7B-finetuned	31.588	0.6901	66.350	17.013	1597
Learnable Compressor	31.994	0.7270	65.264	14.528	1534
Frozen Compressor	29.254	0.7248	63.612	11.883	1318

Fine-tuning the 7B model improves Elo and CLS-R, but DIS-R declines slightly from 17.275 to 17.013. Unfreezing the compression route improves several scores, while LPIPS-R changes from 0.7248 to 0.7270. Overall improvement therefore does not mean that every metric improves.

The original Table 4 controls generated-data scale and complexity, trains the same Qwen2.5-VL-7B on two data sources, and evaluates both on real users.

Training data construction	CLS-T ↑	LPIPS-R ↓	CLS-R ↑	DIS-R ↑	Win rate ↑
profile-free, explicit aesthetic tags only	32.251	0.7192	64.596	13.574	47.6%
profile-inclusive, holistic profiles	31.588	0.6901	66.350	17.013	52.4%

Profile-based training improves DIS-R by 3.439 and achieves a 52.4% win rate, but CLS-T remains lower. This supports the usefulness of profile-conditioned data, not the claim that it can fully replace real-user data.

Key Findings

• Table 2 fixes the GPT-5 rewriter and reports real-user Elo of 1603, 1503, and 1394 for Qwen-Image, Flux, and SDXL. These should not be subtracted directly from Table 1's 1765 because the competing model set changes.
• Section 4.5 evaluates preference-set quality on a 1–5 human-rating scale: 3.61 for profiles versus 3.28 for tags alone. These are data-quality ratings, not generation-model Elo scores.
• The observation that real users are easier to align with than synthetic agents concerns this sample distribution; it does not establish that human preferences are universally simpler than synthetic ones.

Highlights & Insights

• Profiles make a person's preference for diverse imagery a controllable data property rather than noise to remove. This creates a testable way to retain within-user variety and between-user distinctions simultaneously.
• Different information is supplied to the generator and the judge. The generator infers taste from historical images, while the judge uses the profile to assess whether that inference is plausible.
• Retaining CLS-T as a fidelity diagnostic is more informative than optimizing only a single preference score. Future methods should distinguish stable aesthetic cues from incidental reference-image subjects.

Limitations & Future Work

• The authors acknowledge a skewed real-user distribution and costly follow-up participation. Section 3.2 reports that 68 of 134 participants declined subsequent selection, whereas Section 3.3 reports 76 final users. The main text does not reconcile these stages, so no additional recruitment or return-participation explanation should be invented.
• Reference images are primarily generated with Qwen-Image, favoring that backbone on automatic similarity metrics, as acknowledged in Section 4.3. Profile-based Elo complements these metrics but does not establish that source-model bias has been eliminated.
• Synthetic profiles depend on psychological association rules, and automatic selection depends on an aesthetic scorer. This note's assessment is that both may encode stereotypes; cross-group human feedback and uncertainty evaluation are needed before equating synthetic coverage with real representativeness.
• Figure 4 shows near-copying by single-reference editing and an unwanted robot introduced by GPT-5 fusion from one reference, contradicting the prompt. Preference abstraction and content preservation remain unresolved.
• The cache lacks supplementary training details, and approximately 91% judge agreement does not establish reliability for every profile type. Reproduction and subgroup analysis require the supplementary material and stratified human validation.

Related Work & Insights

• vs DreamBooth: DreamBooth binds reference concepts through per-user fine-tuning, whereas PIPBench tests implicit taste across references. Subject fidelity does not automatically imply preference understanding.
• vs Viper / Tailored Visions: The former studies individual visual preferences and the latter rewrites historical prompts. PIPBench adds structured profiles to data construction and evaluation rather than establishing personalized generation from scratch.
• vs DreamBench++ / Pick-a-Pic: Object-level personalization and general image preference evaluation differ from identifying a particular person's taste. A transferable approach is to infer latent preferences from multiple references while separately checking instruction fidelity.

Rating

• Novelty: 4/5, profile-inclusive data protocols are richer than isolated style tags; the main contribution lies in evaluation and data construction.
• Experimental Thoroughness: 4/5, four conditioning paradigms, trainable approaches, and profile ablations are covered, but real-user scale and source bias limit the conclusions.
• Writing Quality: 4/5, the task and comparisons are clearly organized, while participant counts and cross-table Elo require careful interpretation.
• Value: 4/5, a diagnostic framework that jointly considers personal preferences, instruction fidelity, and data provenance.