---

title: >- [Paper Note] POET: Preference Optimization for Enhanced Text-to-Image Generation description: >- [ECCV 2026][Image Generation][Prompt Rewriting] An automated, SFT-free prompt rewriting framework driven by iterative Direct Preference Optimization (DPO) and multimodal LLM judges, consistently enhancing image quality, alignment, and aesthetics for frozen text-to-image backbones. tags: - ECCV 2026 - Image Generation - Prompt Rewriting - Direct Preference Optimization - Reinforcement Learning date: 2026-09-19 content_hash: 7bb60b37acb0d30b

POET: Preference Optimization for Enhanced Text-to-Image Generation

Conference: ECCV 2026
Paper: ECCV Official
Area: Image Generation
Keywords: Prompt Rewriting, Direct Preference Optimization, Text-to-Image, MLLM-as-a-judge, Reinforcement Learning

TL;DR

POET introduces an automated, SFT-free prompt rewriting framework trained via iterative Direct Preference Optimization with multimodal LLM judge feedback, substantially bridging the distributional gap between underspecified user inputs and frozen text-to-image backbones.

Background & Motivation

Recent breakthroughs in text-to-image (T2I) generation—spanning diffusion models, autoregressive transformers, and next-scale prediction architectures—have made it possible to render visually striking and highly realistic imagery. However, real-world deployment faces a persistent distributional gap: everyday user prompts are typically concise, ambiguous, or underspecified (such as "a cat disney cartoon style"), whereas state-of-the-art T2I models are primarily pretrained on long, descriptive, and syntactically dense synthetic captions. When fed simple user inputs, frozen T2I models frequently exhibit severe semantic misalignment, missing attributes, unnatural anatomical artifacts, and erratic aesthetic styles.

Existing strategies to resolve this disconnect fall into three dominant paradigms, each encumbered by notable trade-offs. In-Context Learning (ICL) and interactive human-in-the-loop prompt engineering depend heavily on delicate prompt crafting and suffer from context-window constraints while requiring continuous human intervention. Supervised fine-tuning (SFT) approaches, as exemplified by DALL-E 3's captioning pipeline, necessitate labor-intensive curation of paired short-and-expanded prompts; more critically, because different T2I models embody distinct inductive biases, SFT causes rewriters to overfit to narrow human demonstration styles and limits cross-model transferability. Reinforcement learning methods based on PPO, on the other hand, suffer from extreme training instability, high policy variance, and the cumbersome requirement of joint model-rewriter optimization.

This paper tackles the challenge from a purely input-side perspective, treating target T2I models as frozen black-box environments. If a T2I backbone possesses implicit structural preferences for certain descriptive cues, an LLM rewriter ought to discover these preferences directly from the perceptual feedback of synthesized images without needing intermediate human demonstrations. Core idea: train an automated prompt rewriter entirely through iterative Direct Preference Optimization (DPO) guided by multimodal LLM judges, using round-robin pairwise image comparisons across decoupled alignment and aesthetic reward dimensions to optimize input prompts for frozen T2I backbones without supervised fine-tuning.

Method

Overall Architecture

POET operates as an end-to-end, black-box optimization framework comprising three decoupled stages: candidate prompt expansion and image synthesis, fine-grained pairwise scoring via an MLLM judge, and iterative DPO policy updating. The parameters of the downstream T2I generators remain completely frozen throughout the entire training lifecycle.

Given an initial concise user prompt \(x\), the rewriter policy \(f_\theta\) generates \(n\) distinct candidate expanded prompts \(\{y_1, y_2, \dots, y_n\}\). A frozen T2I backbone then synthesizes corresponding images for each candidate prompt. Next, a multimodal judge model (Qwen2.5-VL-72B-Instruct) evaluates all \(n(n-1)\) ordered image pairs across decoupled quality, semantic, physical, and aesthetic dimensions. The aggregated pairwise tournament points yield a winning prompt \(y_w\) and a losing prompt \(y_l\). The triplet \((x, y_w, y_l)\) is then optimized via DPO loss, while the reference policy \(\pi_\text{ref}\) is updated iteratively across rounds to steadily advance prompt quality.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["User Input Prompt x"] --> B["Candidate Prompt Generation<br/>Rewriter samples n candidate variants"]
    B --> C["Frozen Image Synthesis<br/>T2I backbone renders n candidate images"]
    C --> D["Multimodal Decoupled Evaluation<br/>Quality / General Alignment / Physical / Aesthetics"]
    D --> E["Tournament Preference Selection<br/>Aggregate pairwise votes to choose yw and yl"]
    E --> F["Iterative DPO Policy Update<br/>Minimize DPO loss with rolling reference model"]
    F -->|Update weights| B

Key Designs

1. SFT-Free Iterative Exploration: Unlocking Unbiased Exploration Dynamics

Conventional prompt engineering frameworks typically perform supervised fine-tuning on human-curated or machine-generated expanded prompt pairs before applying reinforcement learning. The authors discover that SFT fundamentally impairs the rewriter's exploratory capacity, causing it to collapse into the linguistic templates of the annotation dataset and overfit to a single model backbone. POET bypasses SFT entirely: an instruction-tuned base LLM (such as Llama-3-70B-Instruct or Qwen-3-32B) begins directly with DPO optimization. In each iterative training round, the reference policy is synchronized with the latest model weights (\(\pi_\text{ref} \leftarrow \pi_\theta\)). Under this soft KL-divergence constraint, the rewriter naturally discovers how to inject rich spatial, lighting, texture, and contextual descriptors, evidenced by a steady, organic increase in average token length and reward scores across successive rounds.

2. Four-Dimensional Decoupled Reward System: Multi-Aspect Perceptual Supervision

To provide precise and fine-grained guidance, POET introduces Qwen2.5-VL-72B-Instruct as an MLLM-as-a-judge across four orthogonal pairwise dimensions: - Image Quality (\(r_\text{Quality}\)): Evaluates anatomical plausibility of human/animal figures, geometric regularity, adherence to basic lighting and reflection physics, overall scene semantic consistency, and absence of high-frequency noise or blur artifacts. - General Image–Text Alignment (\(r_\text{General-Alignment}\)): To prevent complex prompts from confusing the visual judge, the MLLM executes an automated two-step procedure: first decomposing the prompt into atomic binary questions (e.g., transforming "four apples on a table" into "Are there four apples?", "Is there a table?", "Are the apples on rather than under the table?"), and then performing pairwise comparative scoring against these structured queries. - Physical Image–Text Alignment (\(r_\text{Physical-Alignment}\)): Specifically inspects spatial relationships (left/right, above/below, front/behind), precise entity counting (numeracy), and attribute binding (ensuring colors, textures, and scales bind accurately to their designated objects). - Image Aesthetics (\(r_\text{Aesthetics}\)): Rates artistic composition, lighting ambiance, depth of field, and stylistic visual appeal.

Using these criteria, POET trains two distinct policy variants to navigate the inherent trade-off between semantic fidelity and decorative visual flair: a General Rewriter targeting semantic precision (\(r^\text{General} = r_\text{Quality} + r_\text{General-Alignment} + r_\text{Physical-Alignment}\)), and an Aesthetics Rewriter tailored toward maximum artistic impact by incorporating \(r_\text{Aesthetics}\).

3. Pairwise Tournament Selection: Robust Reward Aggregation Under High Variance

Directly assigning continuous scalar scores to individual synthesized images (such as in PPO or GRPO) incurs extreme variance due to subjectivity and inter-batch score drift in multimodal evaluators. POET instead employs a pairwise tournament strategy over candidate rewrites. For each prompt \(x\) and its \(n\) rendered images, all \(n(n-1)\) ordered pairs \((i, j)\) are evaluated by the reward models. A win awards \(+1\) point to the winner and \(-1\) to the loser, while a tie assigns \(0\). Aggregating points across all comparisons yields cumulative scores \(R_i = \sum_{k=1}^K r_i^k\). The candidate with the highest total score is selected as \(y_w\), and the lowest as \(y_l\). This round-robin voting mechanism filters out transient evaluator noise, producing high-confidence chosen-rejected pairs for stable preference optimization.

Loss & Training

The parameters \(\theta\) of the rewriter are updated by minimizing the standard Direct Preference Optimization objective over the curated preference triplets \((x, y_w, y_l)\):

\[\mathcal{L}_\text{DPO}(\pi_\theta; \pi_\text{ref}) = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_\text{ref}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_\text{ref}(y_l \mid x)} \right) \right]\]

Training is carried out using Low-Rank Adaptation (LoRA), which matches full parameter fine-tuning in alignment and quality metrics while dramatically reducing compute and memory footprints. The iterative loop resamples candidate prompts and updates \(\pi_\text{ref}\) each round, ensuring sustained policy improvement without policy collapse.

Key Experimental Results

Main Results

On the GenEval benchmark, POET achieves state-of-the-art text-to-image alignment across various T2I architectures. As shown in Table 1, POET delivers the highest overall score among existing prompt refinement methods on FLUX.1-dev. Table 2 details consistent improvements across diverse backbones on GenEval, T2I-CompBench++, TIFA, and MS-COCO 30K FID.

Table 1: GenEval Benchmark Comparisons (from Table 1 of the paper, higher is better)

Method	T2I Backbone	Single Object	Two Objects	Counting	Colors	Position	Color Attribution	Overall Score
Baseline	Show-o	0.95	0.52	0.49	0.82	0.11	0.28	0.53
PARM++	Show-o	0.99	0.71	0.69	0.95	0.36	0.49	0.70
Baseline	FLUX.1-dev	1.00	0.87	0.76	0.84	0.22	0.49	0.70
Zero-Shot	FLUX.1-dev	0.98	0.89	0.71	0.82	0.47	0.49	0.73
ICL	FLUX.1-dev	1.00	0.89	0.80	0.83	0.54	0.41	0.74
RePrompt	FLUX.1-dev	0.98	0.87	0.77	0.85	0.62	0.49	0.76
POET (Ours)	FLUX.1-dev	1.00	0.95	0.78	0.88	0.58	0.56	0.79

Table 2: Alignment and Image Quality Across Backbones (from Table 3 of the paper)

Model Configuration	Color↑	Spatial↑	Numeracy↑	TIFA Score↑	MS-COCO FID↓
FLUX.1-schnell Baseline	0.7492	0.2754	0.6062	0.8803	20.57
+ POET (Ours)	0.7614	0.3216	0.6161	0.8868	17.76
FLUX.1-dev Baseline	0.7647	0.2763	0.6130	0.8572	24.38
+ POET (Ours)	0.7978	0.3206	0.6343	0.8809	19.57
SD-3.5-medium Baseline	0.7988	0.2889	0.6033	0.8782	17.81
+ POET (Ours)	0.8040	0.3322	0.6320	0.8878	17.13
JanusPro Baseline	0.5294	0.1579	0.4380	0.8457	19.28
+ POET (Ours)	0.7861	0.2773	0.5983	0.8845	16.71

Ablation Study

On the Pick-a-Pic v2 benchmark, GPT-4o pairwise evaluations against DALL-E 3 validate the distinct characteristics of the General and Aesthetics rewriters, as well as the necessity of each reward component.

Table 3: Pick-a-Pic v2 Pairwise Win Rates vs. DALL-E 3 (from Table 4 of the paper)

Model & Configuration	Image Quality	Image Aesthetics	Text-Image Alignment	Average Win Rate
DALL-E 3 (Reference)	0.500	0.500	0.500	0.500
FLUX.1-schnell Original Prompt	0.469	0.314	0.419	0.401
+ ICL Expansion	0.475	0.307	0.422	0.401
+ POET (General Rewriter)	0.494	0.476	0.561	0.510
+ POET (Aesthetics Rewriter)	0.495	0.818	0.424	0.579
FLUX.1-dev Original Prompt	0.513	0.350	0.360	0.408
+ ICL Expansion	0.531	0.363	0.457	0.450
+ POET (General Rewriter)	0.536	0.491	0.575	0.534
+ POET (Aesthetics Rewriter)	0.576	0.800	0.391	0.589

Key Findings

• The Alignment vs. Aesthetics Tension: Integrating the aesthetic reward drives the aesthetic win rate up to 0.818, but drops text-image alignment from 0.561 to 0.424. Qualitative inspection confirms that aesthetic optimization often introduces ornate decorative props and complex lighting, which inadvertently obscures or dilutes core user-specified objects. Decoupling into General and Aesthetics policies provides practical flexibility.
• Robust Cross-Model Transferability: A POET rewriter trained on one generator (e.g., FLUX.1-schnell) transfers seamlessly to an unseen generator (e.g., SD-3.5-medium), achieving gains comparable to in-domain training. This proves that preferences for clear attribute binding, structured spatial relations, and balanced lighting reflect universal linguistic properties favored across modern generative vision architectures.
• LoRA vs. Full Fine-Tuning: LoRA achieves identical or superior performance compared to full parameter tuning while preventing catastrophic forgetting in the language backbone and saving significant training overhead.

Highlights & Insights

• Pure RL Without Human Demonstration: Disproving the assumption that prompt engineering necessitates extensive supervised training data, POET demonstrates that an LLM can learn optimal multimodal prompting strategies purely from frozen image evaluation feedback.
• Decomposition-Based MLLM Evaluation: By breaking complex user prompts into atomic yes/no queries prior to visual comparison, POET effectively suppresses evaluator hallucinations and anchors judgement on verified visual facts.
• Model-Agnostic, Plug-and-Play Input Side Optimization: Enhancing T2I quality without gradient backpropagation through multi-billion parameter diffusion or autoregressive models provides a cost-effective, readily deployable pipeline for production systems.

Limitations & Future Work

• Inference Latency: Generating an expanded prompt requires an initial autoregressive forward pass through an LLM, introducing non-negligible latency for real-time interactive applications.
• Evaluator Bias and Resolution Limits: The reward signal remains bounded by the perceptual capabilities of Qwen2.5-VL-72B. Hallucinations on dense micro-objects or unusual artistic styles can inject noisy preference pairs.
• Context Length in Legacy T2I Models: Longer rewritten prompts risk exceeding token limits in older T2I models (such as 77-token CLIP encoders), leading to truncated trailing descriptions. Future work could integrate adaptive length penalties into the reward objective.

Related Work & Insights

• vs. RePrompt / PARM: RePrompt integrates chain-of-thought verification into prompt expansion, which introduces intricate reasoning steps and higher inference complexity. In contrast, POET optimizes prompts end-to-end via DPO, achieving higher overall GenEval scores (0.79 vs 0.76 on FLUX.1-dev) while keeping the generator completely black-box.
• vs. DALL-E 3 Recaptioning: DALL-E 3 requires massive, carefully annotated image-caption datasets to train an auxiliary captioner for SFT. POET bypasses supervised data collection by leveraging relative preference tournaments over synthesized candidates.

Rating

• Novelty: ⭐⭐⭐⭐ [Establishes an SFT-free, iterative DPO framework for automated T2I prompt optimization driven by decoupled MLLM feedback]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across FLUX, SD-3.5, JanusPro, and Show-o with rigorous human correlation and transferability studies]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear problem formulation, elegant trade-off analysis between alignment and aesthetics, and structured experimental tables]
• Value: ⭐⭐⭐⭐⭐ [Highly practical, model-agnostic, and training-free for generative backbones, offering an immediate enhancement layer for T2I systems]

Related Papers

• [CVPR 2026] OSPO: Object-Centric Self-Improving Preference Optimization for Text-to-Image Generation
• [CVPR 2026] Compositional Text-to-Image Generation Via Region-aware Bimodal Direct Preference Optimization
• [ICLR 2026] PCPO: Proportionate Credit Policy Optimization for Preference Alignment of Image Generation Models
• [ECCV 2026] Diffusion-SDPO: Safeguarded Direct Preference Optimization for Diffusion Models
• [ICLR 2026] ViPO: Visual Preference Optimization at Scale