Skip to content

Zero-Shot Image Personalization from Personas

Conference: ECCV 2026
Paper: ECCV 2026 poster page
Area: Image Generation / User preference personalization
Keywords: zero-shot personalization, natural-language personas, graph attention networks, prompt rewriting, pluralistic preference alignment

TL;DR

ZIPP uses natural-language user personas as interpretable preference conditions: an LLM roleplays the persona to rewrite image prompts, while a vision-aligned social-graph encoder mines the personas. It improves personalized generation without per-user fine-tuning, but "zero-shot" refers to the generation interface once a persona is given — it does not mean persona construction is entirely free of user data.

Background & Motivation

Generic text-to-image models draw images that follow the prompt and look broadly tasteful, yet what they produce does not necessarily match what a specific user hopes for. Given the same street scene, one person wants saturated colors and a bustling composition, another prefers film grain, natural light, and an analog feel. The personalization studied here also differs from DreamBooth-style subject customization that makes "a particular person or object look like itself": the paper cares about aesthetic tendencies that hold across topics and scenes, not the reproduction of a fixed identity or visual concept.

Existing user-level methods learn preferences from prompt histories, liked and disliked images, or detailed feedback. Tailored Visions (TV) retrieves a small set of historical prompts, DrUM trains a per-user adapter, and ViPer extracts visual preference attributes; all of them require accumulated feedback first, and all tend to treat "a style this user once liked" as a rule that transfers to every scene. In reality, a person who likes low-saturation portraits does not necessarily like low-saturation street photography. Cold start — having no history at all — and the fact that the same user's preferences shift with context are the two problems this paper tackles together.

Natural-language personas offer an intermediate representation: instead of compressing a user into a single style vector, a persona describes the user's interests, cultural background, visual leanings, and emotional temperament, and then lets a language model select the parts relevant to the current scene. A persona can come from a ready-made description or a questionnaire, or from this paper's social-graph mining pipeline. Core idea: first mine representative user personas from behavioral and visual signals, then have the language model "stand in this user's shoes" and rewrite the prompt — rather than pushing the persona's identity attributes directly into the picture.

Method

Overall Architecture

The paper calls the overall paradigm ZIPP and the prompt-rewriting implementation used in the experiments ZIPPY; the two are not separate algorithms. The full pipeline runs vision-aligned graph encoding, attention-guided persona mining, and roleplay rewriting in sequence: the first two stages build a natural-language persona from user–community interactions, and the last stage takes that persona plus the original prompt and outputs a personalized prompt for a downstream text-to-image model.

Two usage boundaries are worth separating. The large-scale mining stage trains a shared graph encoder and needs users' posts, comments, and some of their images; the online generation stage needs only an existing persona and the current prompt, with no per-user weight updates. If the user supplies a persona directly, the first two stages can be skipped. "Zero-shot generation" is therefore a constraint on how many preference examples and how much per-user training the generation stage requires — not a claim that the whole data pipeline runs on zero history and zero training.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Community interactions<br/>posts, comments, shared images"] --> B["Vision-aligned graph encoding"]
    B --> C["Attention-guided persona mining"]
    C -->|Natural-language persona| D["Roleplay rewriting"]
    P["Base prompt<br/>persona may be supplied directly"] --> D
    D --> E["Rewritten prompt<br/>text-to-image model generates"]

Key Designs

1. Vision-aligned graph encoding: learn at once which communities a user joins and what they visually share

The volume of social activity is not the same as the amount of preference information it carries: giant communities and heavy users can dominate the interaction counts without being the best explanation of anyone's aesthetics. The paper therefore builds a bipartite graph of Reddit users and subreddits, uses posting and commenting counts as edge features, log-compresses the activity and normalizes it by user and community degree to damp popularity effects, and filters out tail communities that contribute a negligible amount of activity. Community nodes are initialized with text-embedding-3-large vectors of their descriptions and user nodes with small Gaussian noise; nodes and edges are projected into a hidden space and passed through two GATv2 layers with four attention heads each, aggregating neighborhood community information into the user representation. Because the community text gives the representation transferable interest semantics rather than tying it to user IDs alone, the authors use it as an inductive encoder for new users instead of retraining a set of parameters per user.

Learning only the user–community edges could yield a representation of "what the user discusses" without capturing "what the user likes to look at". The paper therefore adds dual contrastive supervision: a user–community term pulls each user toward the communities they actually interact with, with other communities in the batch as negatives, and an ImageAlign term further aligns each user vector with the CLIP representations of the images they posted, computed only for users who have such visual data. LayerNorm precedes each attention block and the output vectors are L2-normalized, so the cosine geometry serves the contrastive objectives. This visual signal is proxy supervision over historical behavior, not a direct observation of how satisfied the user is with the final generations; it strengthens aesthetic associations but cannot guarantee that every posted image reflects a stable preference.

2. Attention-guided persona mining: spend the limited text budget on representative behavioral evidence

The graph encoder does not hand the diffusion model a latent vector. It exposes the attention weights over the communities in a user's neighborhood, which the paper aggregates and normalizes into a distribution of community importance. Persona construction works under a fixed 4,096-token content budget, allocating shares to high-weight communities in proportion to the normalized weights; within each community, the user's posts and comments are selected by score, deduplicated, and filtered for deleted content. The persona's evidence is thus constrained both by the relevance learned across the whole graph and grounded in what the user actually wrote, rather than simply picking the communities with the most interactions. This budget constrains the behavioral text sent to the persona generator, not the length the final persona must reach.

The sampled content is handed to a multimodal LLM with a structured prompt that asks for a concise, coherent natural-language description of the user's interests, aesthetics, and communication habits. The graph's job is mainly to decide "which evidence to read and how much context each community gets", while the language model turns scattered evidence into a human-readable preference prior. For example, a user active in film-photography communities who repeatedly praises grain and natural light can be described as a film photography enthusiast who prefers black-and-white film, natural lighting, and vintage tonality. The attention itself is not a causal explanation of the user's preferences, and the inferred identity attributes should not be treated as verified facts; the value of the natural-language form is that the user can inspect, correct, or delete it.

3. Roleplay rewriting: interpret the persona as an aesthetic point of view, not an entity in the scene

Once a persona exists, the paper does not splice "you are a historian somewhere who loves classical art" straight into the diffusion scene prompt; it first hands the persona to a language model as a role setting. Taking GPT-4o as the example, the model receives a condition of the form "you are someone who prefers ...", rewrites the base prompt from that person's perspective, and only then lets the text-to-image model execute the rewritten result. The distinction matters because the user's profession or cultural interests should influence materials, lighting, composition, and semantic details, without necessarily requiring a figure of that profession to appear in the frame. The paper reports that direct concatenation underperforms the no-persona baseline on all four benchmarks, but the corresponding appendix table is not part of our source, so no concrete deltas are quoted here.

Roleplay also lets the same persona activate different preference facets depending on the current prompt, instead of applying one set of tone words to every scene. This is the authors' mechanism for the improvement in intra-user pluralism: the natural-language persona stores multi-faceted priors, and the rewriter selects among them by context. The output is still an ordinary text prompt, so no downstream generative model parameters change; but the final quality also depends on whether the language model understands the persona and preserves the original subject, and on whether the text-to-image model faithfully realizes the added details. The paper observed that some models over-personalize, which shows that adding more preference elements is not automatically better.

A Worked Example

The paper illustrates the generation side with the persona of a historian based in Florence. The base prompt is "a cozy wooden table with an open book", and the persona emphasizes classical art, historical atmosphere, and a sense of narrative.

The roleplay rewrite does not mechanically add "a historian must sit beside the table". It converts those interests into visual details — weathered materials, soft low-saturation tones, a setting with a sense of age — and passes the rewritten text to the text-to-image model. The goal is to change the presentation the user would care about, not to replace the subject of "a table and a book".

If the persona comes from social data, it first passes through community-weighted sampling and linguistic summarization; if the user has supplied the same description directly, the generation side can start at the rewriting step. This example shows how the pipeline chains together; it is not an additional quantitative experiment, and it cannot prove that an inferred historian persona actually matches the real user.

Loss & Training

The shared graph encoder is trained with two contrastive objectives, user–community and user–image, the latter weighted at 1.0:

\[ \mathcal{L}=\mathcal{L}_{u,s}+\lambda\mathcal{L}_{u,\mathrm{img}},\qquad \lambda=1.0. \]

The user–community term is an InfoNCE-style in-batch cross-entropy that treats the matched community as the positive and the other communities in the batch as negatives; the user–image term follows the same paired-alignment idea. Together the two terms constrain both the social structure and the visual behavior, so that the attention later used for sampling does not only explain textual interests. The paper also mentions sampling negatives from close neighbors, but the finer sampling procedure sits in an appendix not included in our source, so it cannot be fully reconstructed from the main text.

On the training side, the paper states that community initialization vectors are 3,072-dimensional and that LeakyReLU uses a slope of 0.2; the number of epochs, learning rates, and the full generation sampling settings could not be verified from our source. The formula typesetting in the source is corrupted, so we keep only the total loss that the main text clearly confirms and do not guess the GAT expansion or the exact placement of the stable term in the edge weights.

The few-shot variant adds in-context examples at the rewriting stage rather than per-user gradient updates. The main table compares 3-shot and 5-shot; how the examples are organized and which decoding templates are used is deferred to an appendix that our source does not include. Persona provenance also differs across the external datasets: PIP personas are retrieved with CLIP text embeddings of each user's prompt history at a cosine threshold of 0.7, covering 89% of PIP users, while RapidData and MovieLens condition directly on demographic attributes and profession. PIP's "zero-shot" row therefore cannot be read as never accessing user history.

Key Experimental Results

Main Results

ZIP-Bench ties roughly 1,500 Civitai users to their Reddit behavior and includes about 40,000 historical generated images with prompts; the behavioral material used to build these users' personas is about 15,000 posts and 183,000 comments. The paper also evaluates PIP, RapidData, and MovieLens; MovieLens has no images directly tied to text-to-image outputs and is evaluated on prompts converted from movie metadata, so it should not be treated as an identical supervision source to real image-preference data.

CLIPScore (CS) measures the CLIP cosine alignment between generated images and the user's target text or preferred references; PIGReward (PIG) is a learnable, preference-oriented reward that compensates for CLIP's weakness on fine-grained style and composition preferences. The table keeps the original score scales — both metrics are higher-is-better, and their differences are not "accuracy percentage points". CMMD is the squared MMD with a Gaussian RBF kernel between the CLIP embedding distributions of reference and generated images; lower means the distributions are closer, and it measures reference-distribution matching rather than "the more diverse the better".

The table below is excerpted from Table 1 of the paper; every dataset cell reads "CS / PIG". The no-persona GPT-4o row uses the same rewriting pipeline and is the direct control that isolates the persona's effect. Few-shot and fine-tuned methods consume different data budgets, so a horizontal ranking does not imply strict equal cost.

Method & setting ZIP-Bench PIP MovieLens RapidData CMMD ↓
GPT-4o, no persona, 0-shot 59.1 / 65.3 57.7 / 61.4 73.7 / 58.5 67.3 / 56.0 0.49
ZIPPY, 0-shot 61.8 / 73.4 60.2 / 67.3 76.4 / 64.2 73.9 / 61.5 0.42
TV, 3-shot 62.4 / 72.0 63.1 / 66.5 80.1 / 68.7 77.4 / 65.3 0.25
ZIPPY, 3-shot 66.7 / 77.5 65.2 / 69.9 80.4 / 71.4 77.9 / 68.2 0.19
ZIPPY, 5-shot 68.5 / 82.8 66.3 / 72.1 80.9 / 77.9 78.3 / 74.0 0.16
DrUM, per-user fine-tuning 65.6 / 74.4 66.9 / 68.0 72.8 / 66.2 70.3 / 63.5 0.31
ViPer, fine-tuned baseline — — 76.1 / 70.5 74.1 / 63.1 0.55

On ZIP-Bench, adding the persona while staying 0-shot moves CS from 59.1 to 61.8 (+2.7) and PIG from 65.3 to 73.4 (+8.1). At the same 3-shot budget, ZIPPY leads TV by 4.3/5.5 on CS/PIG, so the advantage of combining a persona with a few examples does not come from having more examples alone.

Two of the paper's prose summaries need calibration against Table 1: the claims that zero-shot beats DrUM on a certain number of benchmarks, and that 5-shot is highest on both metrics across all benchmarks, do not fully match the table. Going by the table values, 0-shot ZIPPY's CS exceeds DrUM only on MovieLens and RapidData, and its PIG is below DrUM on all four benchmarks; 5-shot ZIPPY's PIP CS of 66.3 is still below DrUM's 66.9. The best CMMD of 0.16 also belongs to the 5-shot setting, not to the zero-shot version's 0.42.

Ablation Study

Table 2 compares persona miners, with all rewrites done by GPT-4o. Posting Accuracy measures whether the persona correctly predicts a held-out image the user posted; ZIP Lift is defined as the relative CLIPScore improvement over the unpersonalized baseline — "personalized CS minus baseline CS, divided by baseline CS". The paper does not provide the full candidate set of posted images or the split details, so this accuracy can only be interpreted under the paper's own setup.

Persona mining config Posting Accuracy ↑ (%) ZIP Lift ↑ (%)
TF-IDF 55 5.1
TF-IDF + NMI 57 5.3
LightGCN 59 6.2
LightGCN + ImageAlign 62 9.1
GraphSAGE 63 8.5
GraphSAGE + ImageAlign 67 10.2
GAT 62 11.8
GAT + ImageAlign 73 17.7

The most direct ablation is GAT versus GAT + ImageAlign: posting accuracy rises from 62% to 73% (+11 points), and ZIP Lift from 11.8% to 17.7% (+5.9 points). LightGCN and GraphSAGE also improve when visual alignment is added, though by different amounts, so the visual supervision is not specific to one encoder. The name "NMI" is kept as written in the paper; the main text does not explain its implementation in sufficient detail.

Do not read the ablation table's 17.7% ZIP Lift as the improvement rate behind the main table's zero-shot 61.8 versus 59.1: computed from the table values, the latter is about 4.6%, and the two tables cannot be exchanged as the same set of observations. The main text we have lacks the setting details needed to explain the discrepancy, so the note keeps both figures without forcing them together.

Key Findings

  • Persona priors improve the preference reward, which is not the same as beating fine-tuning across the board. Zero-shot personas improve both CS and PIG over the no-persona control on all four benchmarks, but the comparison with DrUM depends on the metric and dataset; the few-shot gains are more emphatic.
  • Distribution matching improves as examples are added. ZIPPY's CMMD drops from 0.42 at 0-shot to 0.19 at 3-shot and 0.16 at 5-shot; TV is 0.25, DrUM 0.31, and ViPer 0.55. The result supports that a persona plus context reduces the preference-distribution shift caused by a fixed style, but it does not by itself prove that every context of every user is covered.
  • Demographic reweighting exposes differences that aggregate scores hide. IPF iteratively adjusts user weights toward the marginal distribution of demographic attributes and recomputes the aggregate scores. On RapidData, DrUM's CS falls from 70.3 to 64.7 and PIG from 63.5 to 56.8; ZIPPY 5-shot moves from 78.3/74.0 to 77.4/73.0. The smaller drop is favorable evidence, but reweighting cannot create populations absent from the sample, nor does it license a claim that group bias has been eliminated.
  • Human evaluation supports that correct personas help. The paper reports 50 participants, 20 base prompts, and 1,000 annotations in total; the personas came from questionnaires rather than the direct graph-mining output. The win rate is 79% over the no-persona baseline, 58% over DrUM, and 65% over ViPer; the paper also reports an own- versus other-persona control, but our source does not include the full per-condition table.

Highlights & Insights

  • The graph representation influences generation through sampling, not latent injection. Attention decides which historical evidence enters the persona, and the language model then produces a readable summary and interpretation. This removes the need for a dedicated adapter between graph learning and off-the-shelf text-to-image models, and makes the intermediate results easier for users to inspect.
  • Identity information and scene content are handled separately. The role setting acts on the prompt rewriter rather than asking the diffusion model to render an identity label. This distinction explains why "adding more user text" does not automatically mean better personalization.
  • Average alignment and preference distribution are examined together. CS and PIG measure the alignment of individual outputs, CMMD measures distribution matching against a user's reference set, and the two are complementary; IPF further tests whether the user population's composition changes the conclusion, avoiding a single aggregate mean as the whole story of personalization ability.

Limitations & Future Work

  • The conditions under which cold start holds need to be spelled out. With a persona in hand, no preference examples are needed, but graph mining itself depends on historical behavior, and PIP's persona retrieval also uses prompt history. For a user with neither an available persona nor behavioral data, the paper does not demonstrate an ability to recover personal aesthetics from nothing.
  • Text personas introduce inferred bias and privacy concerns. Community interests and demographic attributes do not determine aesthetics, and the model may stereotype-fill from identity labels. The authors state that only public content is used and anonymized, but cross-platform account linking, persona accuracy, and user correction still need independent review; publicly visible does not equal consented for this use.
  • The evaluation population and proxy metrics limit extrapolation. ZIP-Bench comes from Civitai users tied to an AI-art community and does not represent everyone; MovieLens proxies visual preferences with movie text. IPF can only reweight data that exists and cannot repair missing groups or wrong personas.
  • Reproduction information in our source is incomplete. What we could read is the main text and references, without the appendices the paper cites; the concrete diffusion model settings, few-shot templates, and some evaluation splits cannot be verified. The graph-size description is also suspect: the text first gives about 40K communities, then says 32K remain after dropping the bottom 12K, and the two statistics do not reconcile by simple subtraction, so we do not derive a precise filtering count from them.
  • Some conclusions run ahead of what the main table supports. Zero-shot and few-shot results need to be kept apart, and the exception that PIP CS does not beat DrUM should stay on the record. More convincing follow-up validation would report per-user confidence intervals, behavior when the persona is wrong, and long-term preference drift, rather than only making the persona descriptions longer.
  • vs. Tailored Visions (TV): TV retrieves context from a few historical prompts, while this paper additionally brings in a summarizing persona prior. The 3-shot-versus-3-shot comparison makes the point, but the cost of mining personas and preparing retrieval should not be treated as negligible.
  • vs. DrUM and ViPer: DrUM learns the historical distribution with a per-user adapter, and ViPer extracts visual preference attributes; this paper re-interprets preferences per scene through language conditioning. The advantage is no per-user gradient updates and an editable condition; the price is dependence on persona trustworthiness and on the rewriting model's comprehension.
  • vs. DreamBooth and Textual Inversion: those methods focus on preserving a specific subject or concept, while this paper targets a user's aesthetics — the task goals differ. This paper's results cannot be used to claim stronger identity consistency or subject fidelity.
  • Reusable insight: compressing behavioral evidence first into an inspectable preference description, and then using it for context-dependent decisions, leaves more room for explanation than fixing a user as a single style label. This is a method-level insight; the evidence in this paper is not yet sufficient to turn it into a standalone new research proposal on its own.
  • Data resource: ZIP-Bench, link from the paper's main text; the dataset link is not treated as a verified code repository.

Rating

  • Novelty: 4/5. Wiring a vision-aligned social graph, natural-language personas, and roleplay rewriting into a user-aesthetic personalization pipeline is a clearly positioned problem, but none of the base components is newly proposed.
  • Experimental Thoroughness: 3/5. Covers multiple benchmarks, a graph-encoder ablation, distribution metrics, and human evaluation; the main table and the prose disagree in places, and several key settings appear only in appendices not included in our source.
  • Writing Quality: 3/5. The motivation and pipeline are easy to follow, but the zero-shot boundary, statistical framing, and method naming require the reader to calibrate on their own.
  • Value: 4/5. Provides a personalization interface that needs no per-user fine-tuning and stays editable, and reminds evaluation to account for intra-user preference diversity and group distributions.