Scaling Multi-Reference Image Generation with Dynamic Reward Optimization¶
Conference: ECCV 2026
Paper: ECCV
Code: https://github.com/Weistrass/DyRef
Area: Image Generation
Keywords: multi-reference image generation, reward optimization, Flow-GRPO, hard-sample reweighting, personalized image generation
TL;DR¶
The paper first introduces OmniRef-Bench, covering five reference types and 2-7 reference images per instance, exposing how sharply open-source models degrade as the number of mixed-type references grows; it then proposes DyRef, a two-stage framework that pairs SFT cold-start with two additions on top of Flow-GRPO — DRS, which widens intra-group reward gaps, and DAR, which inversely reweights groups by their mean reward — lifting open-source backbones to a level comparable with the closed-source Nano Banana Pro on complex multi-reference generation.
Background & Motivation¶
Personalized image generation has matured considerably with diffusion models and DiT architectures, yet multi-reference image generation (MRIG) remains hard: a single prompt may simultaneously reference "the person in image 1", "the pose in image 2", "the lighting in image 3", "the background in image 4" and "the visual style in image 5", and the model must inject each of these heterogeneous references into the right place. These types constrain generation at completely different granularities — subject identity demands pixel-level local fidelity, pose demands geometric alignment, lighting demands globally consistent tone, style demands transfer of statistical characteristics — and they interfere with one another. This capability maps directly onto professional visual design and advertising, where several assets must be composed into one coherent image, so it is far from a toy task.
The real bottleneck lies in evaluation. Existing multi-reference benchmarks either handle only single- or multi-subject settings (XVerse, PSRBench), or introduce reference types beyond subject but with only simple pairwise combinations and very few reference images (MultiBanana with 3 combinations, DreamOmni2 with 2), or they assign a single coarse-grained overall score instead of evaluating each reference type separately. None of them combines complex reference-type combinations, a large number of reference images, expert annotation, and fine-grained multi-dimensional evaluation. Without adequate evaluation, gaps between methods cannot be measured reliably, and there is no way to tell which reference type or which reference count a model actually collapses on. Filling this gap with OmniRef-Bench immediately reveals a clear pattern: mainstream open-source models produce acceptable images with few references, but once the reference count rises to 4-6 with mixed types, the output shows obvious artifacts and semantic loss, and CLIP-I falls rapidly as references increase (Fig. 3b of the paper), whereas closed-source models decline far more gently. The gap between open-source and closed-source models is, to a large extent, precisely this difference in the rate of degradation with reference complexity.
The problem can therefore be reformulated: rather than piling on more data or redesigning the architecture, the imbalance in the training signal itself should be fixed first. During RL, Flow-GRPO treats the samples drawn from each prompt as a group and computes intra-group advantages, but it is completely insensitive to how difficult that group is: samples with many mixed-type references naturally earn lower rewards, so their advantages are also small in magnitude, and the gradient ends up dominated by easy, well-rendered, high-reward samples with few references. Worse, reward values from pretrained scorers such as CLIP and SigLIPv2 are extremely concentrated — even when two generated images differ substantially in visual quality, their rewards differ by only a few thousandths — so intra-group advantages lose nearly all numerical discriminability. Core idea: without touching the model architecture, apply two complementary corrections on the reward side after SFT — DRS, which re-spreads the over-concentrated reward distribution to restore gradient discriminability, and DAR, which inversely scales sample weights by each group's mean reward so that weak groups with many mixed-type references automatically receive a larger optimization weight.
Method¶
Overall Architecture¶
DyRef is a two-stage training framework in which the training data is constructed once and shared by both stages (the RL stage introduces no new dataset). Stage I builds roughly 14,000 samples with a self-designed multi-reference data construction pipeline and performs supervised fine-tuning with LoRA and a Flow Matching loss on a dual-stream MMDiT backbone such as Qwen-Image-Edit-2511, giving the model the basic ability to read several heterogeneous reference images at once and place each of them correctly. Stage II runs online RL (Flow-GRPO) on this cold-started model: each prompt yields a group of samples, every sample receives a scalar reward computed with CSD and CLIP / SigLIPv2, DRS first amplifies the reward gaps within the group, DAR then adjusts the group's contribution to the total loss according to its mean reward, and the policy is finally updated with the weighted GRPO objective. In short, Stage I pulls the model into the "it can do this" region, and Stage II corrects its under-learning on hard samples.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
A["Multi-reference data construction + SFT cold-start<br/>T2I target image → derive each reference type → LoRA"]
A --> B
subgraph S2["Stage II: reward optimization on top of Flow-GRPO"]
direction TB
B["Reward design<br/>CSD style reward + CLIP/SigLIPv2 semantic reward"]
B --> C["Discriminative Reward Scaling (DRS)<br/>widen intra-group reward gaps"]
C --> D["Difficulty-aware Advantage Reweighting (DAR)<br/>inverse group weighting by mean reward"]
end
D --> E["Weighted GRPO policy update"]
E -->|resample the same prompts| B
Key Designs¶
1. Multi-reference data construction + SFT cold-start: make "reading several heterogeneous reference images" something the model has seen
The first reason open-source backbones collapse on complex MRIG is that such samples simply do not exist in their training distribution — conventional editing data is mostly one image with one instruction. The data construction here works backwards, from target image to references: subject concepts are sampled from Objects365, an LLM sequentially generates subject instances and T2I prompts, and a T2I model produces the target image; anchored on that image, a segmentation model such as Grounded SAM2 cuts out the subject and a series of image editing models (e.g., Qwen-Image-Edit-2511) derive the subject's background reference, lighting reference and pose reference; style references come from the external OmniConsistency dataset, together with the corresponding stylized target image. The benefit is that each reference image stands in a verifiable correspondence to the target (the pose reference's geometry genuinely comes from the target, for instance), instead of requiring humans to assemble which images share an identity or a style. The resulting ~14,000 samples feed both SFT and the subsequent RL stage; the number of reference images varies naturally across samples (2-7 in the benchmark), and the model accepts a variable number of references by feeding them together with the instruction into the editing path of the dual-stream MMDiT — "scaling" the reference count relies not on architectural changes but on the training data covering the whole range from 2 to 7. SFT uses low-rank adaptation with a Flow Matching loss, a small and pluggable change, which is also what makes the recipe transferable to FLUX.2 [klein] 9B later on.
2. Reward design: CSD for style, CLIP/SigLIPv2 for semantics, each covering the other's blind spot
After SFT the model is already usable in most dimensions but clearly lags on style-related tasks (in Tab. 8, SFT reaches only 0.38 on the objective style metric and 7.57 on the MLLM style score, while background and subject stay above 8). The reason is that the pixel-level reconstruction loss of Flow Matching is nearly insensitive to whether the image is stylistically consistent in the statistical sense, and style is precisely the most abstract reference type in MRIG. Inspired by the USO framework, the paper adds CSD (style similarity, from Somepalli et al.'s style similarity measure) as a reward model for style alignment, directly steering the generated image toward the style reference; the other line uses CLIP or SigLIPv2 to measure semantic consistency between the generated image and its target image, acting as a safety net that keeps the output from drifting, and narrowing the distribution gap between generated outputs and the target data. Both rewards come from frozen pretrained scorers that are never trained, so the whole RL stage is model-free and driven purely by black-box scores. In the ablation, removing CSD drops the objective style score from 0.46 to 0.36 and the MLLM style score from 8.67 to 6.33, the largest single drop in the entire table, confirming that this style reward is irreplaceable.
3. Discriminative Reward Scaling (DRS): re-spread a reward distribution that has collapsed together
Similarity-based rewards such as CLIP and SigLIPv2 suffer from a congenital defect: extremely concentrated values. Fig. 5(a) gives an intuitive example — two generated samples differ noticeably in visual quality, yet their raw rewards are 0.790 and 0.794, a gap of only 0.004. GRPO's intra-group advantages are built exactly on such reward differences, so when rewards crowd together the advantage degrades into near-noise: samples with negative advantage are not penalized enough, and those with positive advantage are not rewarded enough. DRS addresses this at every optimization step by passing the raw reward \(p=\cos(F_{\text{gen}}, F_{\text{target}})\), the cosine similarity between the CLIP/SigLIPv2 features of the generated and target images, through a controllable transformation
where the hyper-parameter \(t\) controls the shape of the transformed reward distribution. After this transformation the pair above becomes 0.424 and 0.563, widening the gap by more than an order of magnitude and giving intra-group advantages usable numerical contrast again. Note that DRS does not change the relative ordering between samples (better stays better); it only releases discriminability that was previously buried in the biases of the pretrained scorer — which is also why it mainly improves high-level semantic metrics (MLLM style 8.67 vs. 8.12 without it, aesthetics 8.14 vs. 8.03) while leaving low-level objective metrics essentially intact. ⚠️ The concrete form of the transformation \(F\) is left to Appendix C.4 of the paper; only its role and constraints are kept here.
4. Difficulty-aware Advantage Reweighting (DAR): let weak groups with many references automatically earn a larger weight
Even with DRS widening the rewards, standard Flow-GRPO retains a structural blind spot: it treats every prompt group alike, while groups differ enormously in difficulty. The authors observe two things — first, the visual information in an MRIG sample grows with the number of mixed-type reference images, making these samples harder and more deserving of focus; second, the actual reward scores of these samples do decline systematically as the reference count grows (detailed analysis in Appendix C.3 of the paper). Together these mean hard groups produce small advantages, optimization is dominated by easy groups, and hard groups simply fail to learn. DAR borrows the idea behind Focal Loss in dense object detection, which rewrites each sample's contribution to the total loss to combat sample imbalance, and applies the weighting at the group level: for a training set \(\mathcal{S}\) in one optimization step partitioned by prompt into groups \(\mathcal{G}\), with index set \(I_g\) for group \(g\) and scalar reward \(p_i\in[0,1]\) for sample \(i\), it first computes the group mean reward \(\bar p_g = \frac{1}{|I_g|}\sum_{i\in I_g} p_i\) and then builds an inverse raw weight from it:
Here \(\epsilon\) is a small constant for numerical stability and \(\gamma\) controls the strength of reweighting. The inverse form means a group with a lower mean reward gets a larger weight, directing optimization pressure exactly at the weak groups with many mixed-type references, while the clip prevents any single extreme group from blowing the weight up. To avoid shifting the overall loss scale, group weights are normalized by the current batch mean \(\mu\) back to a mean of roughly one, and clipped once more:
Each sample inherits its group's weight, \(w_i = \tilde w_{g(i)}\), and enters the GRPO objective weighted by it. Compared with standard GRPO, DAR changes only the advantage/loss weighting and never touches the policy's sampling procedure, so it is extremely cheap to implement while directly altering the fundamental question of which samples drive the gradient. ⚠️ Eq. (3) is typeset incorrectly in the cached text; the inverse-power + clip + normalization structure given here follows the paper, and the concrete value of \(\gamma\) is not stated in the main text.
A Worked Example¶
Take two groups from the same optimization step in Fig. 5. Group 1 contains easy samples with few reference images; its two sampled images score 0.790 and 0.794, making it a high-quality group with almost no intra-group discriminability. Group 2 contains hard samples with many mixed-type references and scores noticeably lower overall.
DRS acts first: the raw gap of 0.004 in group 1 is transformed into 0.424 vs. 0.563, so an intra-group advantage becomes computable again. Then DAR acts: group 1 has a high mean reward, so its raw weight from \(w_g^{\text{raw}}=(\bar p_g+\epsilon)^{-\gamma}\) is small, and after clipping and normalization it lands at 0.80 — the group is maintained at a slightly below-average contribution instead of hogging the gradient. Group 2 has a low mean reward, and its normalized weight rises to 1.25, so its samples are amplified and optimized with priority. Within a batch the mean weight is normalized back to roughly one ((0.80+1.25)/2 ≈ 1.03), so the overall loss scale stays comparable to the run without DAR — DAR changes the relative mix across difficulty levels, not the global learning rate. ⚠️ The values 0.790/0.794/0.424/0.563 and 0.80/1.25 are taken from Fig. 5 of the paper and are illustrative.
Loss & Training¶
Stage I follows the standard Flow Matching objective: given the time \(t\), a \(z_0\) drawn from a standard Gaussian and a \(z_1\) drawn from the target data distribution, the network fits the velocity field formed by their difference,
(⚠️ Eq. (1) is typeset incorrectly in the cached text; the standard Flow Matching form is given here.) Fine-tuning is performed with LoRA.
Stage II adds sample weights to the Flow-GRPO objective. Let \(r_i(\theta)\) be the probability ratio between the current and behavior policies on sample \(i\), \(\hat A_i\) its estimated advantage, and \(\epsilon\) the PPO-style clipping range; the weighted objective is
where \(w_i\) is the sample weight from DAR under the DRS-transformed rewards: DRS changes the numerical discriminability of \(\hat A_i\), DAR changes the inter-group mix of \(w_i\), and the two act on different multipliers, so they compose. Key hyper-parameters include the reweighting strength \(\gamma\), the weight range \([w_{\min}, w_{\max}]\) and the DRS shape parameter \(t\); the sensitivity analysis is in Appendix C.2 of the paper.
Key Experimental Results¶
Main Results¶
On the self-built OmniRef-Bench, DyRef is implemented on two backbones — Qwen-Image-Edit-2511 (20B) and FLUX.2 [klein] 9B Base — and compared against two closed-source and three open-source models. Objective metrics are reported per reference type; MLLM evaluation uses Gemini 3 Flash, dropping the pose dimension (which MLLMs judge inaccurately) and adding aesthetics and instruction following.
| Method | Subject | Style | Bg | Light | Pose | Obj. Avg | MLLM Avg |
|---|---|---|---|---|---|---|---|
| Seedream4.5 (closed) | 0.65 | 0.34 | 0.89 | 0.74 | 0.74 | 0.67 | 8.13 |
| Nano Banana Pro (closed) | 0.64 | 0.46 | 0.89 | 0.77 | 0.72 | 0.70 | 8.50 |
| OmniGen2 | 0.59 | 0.11 | 0.85 | 0.65 | 0.65 | 0.57 | 3.79 |
| DreamOmni2 | 0.57 | 0.28 | 0.85 | 0.71 | 0.67 | 0.62 | 5.18 |
| BAGEL | 0.58 | 0.15 | 0.85 | 0.67 | 0.69 | 0.59 | 4.64 |
| Qwen-2511 | 0.55 | 0.15 | 0.84 | 0.71 | 0.71 | 0.59 | 4.97 |
| Qwen-2511 + Ours | 0.62 | 0.46 | 0.87 | 0.76 | 0.83 | 0.71 | 8.38 |
| FLUX.2 [klein] | 0.63 | 0.21 | 0.87 | 0.69 | 0.74 | 0.63 | 6.49 |
| FLUX.2 [klein] + Ours | 0.63 | 0.48 | 0.90 | 0.78 | 0.76 | 0.71 | 8.03 |
The same models on single-image editing and on easier MRIG benchmarks, verifying that the multi-reference gains do not cost basic editing ability:
| Benchmark | Metric | Qwen-2511 | + Ours | FLUX.2 [klein] | + Ours | Note |
|---|---|---|---|---|---|---|
| OmniRef-Bench | Obj. Avg | 0.59 | 0.71 | 0.63 | 0.71 | Complex multi-reference, 12% relative gain (+0.08 vs. FLUX.2) |
| OmniRef-Bench | MLLM Avg | 4.97 | 8.38 | 6.49 | 8.03 | 29% relative gain (+1.89 vs. FLUX.2) |
| ImgEdit | Overall | 4.07 | 4.32 | 4.07 | 4.11 | Beats closed-source GPT Image 1 [High] 4.20 and Seedream4.0 4.18 |
| DreamBench++ | CP×PF | 0.53 | 0.62 | 0.56 | 0.61 | Best open-source result on both backbones |
| OmniContext | Average | 8.14 | 8.24 | — | — | Easier MRIG benchmark, +1.2% |
| MultiBanana | Average | 3.78 | 4.34 | — | — | Easier MRIG benchmark, +14.8% |
Ablation Study¶
On Qwen-Image-Edit-2511, using the Stage I SFT result as the baseline and removing the three training components one at a time (Tab. 8):
| Config | Obj. Avg | MLLM Avg | Style (obj. / MLLM) | Note |
|---|---|---|---|---|
| SFT (Stage I) | 0.68 | 7.90 | 0.38 / 7.57 | Cold-start only; style is the biggest weakness |
| Ours (full) | 0.71 | 8.38 | 0.46 / 8.67 | All three components |
| w/o CSD | 0.69 | 8.00 | 0.36 / 6.33 | Removing the style reward costs 0.10 on objective style and 2.34 on MLLM style, the largest drop in the table |
| w/o DAR | 0.69 | 8.03 | 0.39 / 6.98 | Degenerates to standard GRPO; both objective and MLLM metrics fall (0.71→0.69, 8.38→8.03) |
| w/o DRS | 0.71 | 8.28 | 0.45 / 8.12 | Objective metrics barely move, but MLLM style drops 8.67→8.12 and aesthetics 8.14→8.03 |
In the user study, 30 experts ranked 20 randomly sampled examples on text fidelity, reference similarity, composition quality, visual appeal and task completion: DyRef and Nano Banana Pro take the top two places (MLLM scores 8.38 / 8.50; human scores on a 1-4 scale 3.42 / 3.09), with Seedream4.5 and Qwen-2511 third and fourth. MLLM scores correlate with human preference at Pearson 0.902, Spearman 0.800 and Kendall 0.667, indicating that the "objective metrics + MLLM" protocol points in the same direction as human judgement.
Key Findings¶
- Among the three components, the CSD style reward contributes the most at a single point: removing it drops the MLLM style score from 8.67 to 6.33, far beyond any other drop. The most "abstract" reference type in MRIG is precisely the one a pixel-level reconstruction loss cannot control, and it must be covered by a dedicated style reward.
- DAR and DRS act on different surfaces and cannot replace each other: removing DAR lowers both objective and MLLM metrics (it degenerates to standard GRPO, and hard groups lose their extra weight), whereas removing DRS leaves objective metrics essentially unchanged and only hurts high-level semantic metrics — DRS repairs the gradient-level issue of intra-group advantage discriminability, and its gains show up in semantic dimensions that low-level similarity metrics cannot capture.
- Degradation grows monotonically with reference count, and that is exactly what gets repaired: Fig. 3(b) shows open-source CLIP-I falling from roughly 0.76 at three references to clearly lower values at six, while DyRef's curve stays flatter across the whole range of reference counts — directly matching the design intent of giving hard groups a larger weight.
- The method also holds on a small backbone: the 9B FLUX.2 [klein] rises from 6.49 to 8.03 in MLLM Avg after adding DyRef, approaching the 20B backbone's 8.38, while its single-image editing ability even improves slightly (ImgEdit 4.07→4.11, DreamBench++ 0.56→0.61), showing the gains are not bought with model capacity.
- Generalization to reference types outside the training distribution: the authors also evaluate on reference types not included in training (Appendix C.6) and still see improvements, suggesting DAR/DRS learn how to allocate optimization attention rather than overfitting to specific types.
Highlights & Insights¶
- Treating "performance degrades with difficulty" as an optimization problem rather than a modeling problem. Most work responds to multi-reference degradation by changing attention or adding adapters; this paper points out that much of the degradation comes from the training signal being dominated by easy samples, and corrects it directly at the loss-weighting level with DAR. The view transfers to any RL task where sample difficulty correlates with reward magnitude (long-chain reasoning, few-shot tool use, long-tail retrieval), all of which may suffer the same systematic under-learning.
- DRS exposes a general trap: when a pretrained scorer serves as the reward, concentrated values make intra-group advantages useless. Rewards of 0.790 and 0.794 correspond to a visible quality difference, but GRPO only sees a gap of 0.004. Any RLHF/RLVR pipeline using CLIP-like models as reward models should check the intra-group variance of its rewards first instead of assuming they are usable.
- The two corrections are deliberately designed to act on different multipliers: DRS changes the discriminability of advantages, DAR changes the weighting mix of samples, so they compose naturally and can be ablated separately. This orthogonal split is what makes the ablation conclusions so clean.
- The "target image first, derive references afterwards" construction recipe is reusable. When building multi-condition control data, collecting every conditioning asset in the forward direction is expensive; going backwards from an image that already satisfies all conditions and stripping out one reference per condition with editing models is a cheap path with a built-in correspondence.
Limitations & Future Work¶
- Data construction depends on a long chain of external models, and errors compound. The target image comes from a T2I model, the subject mask from Grounded SAM2, each reference type from an editing model, and the style image from the OmniConsistency dataset — a failure at any link (a dirty mask, an editing model changing what it should not) becomes a noisy label in the training data, and such noise is further amplified during reward optimization.
- Introducing no new data in the RL stage also means the ceiling is set by SFT data coverage. The authors stress that RL uses data from the same source as SFT; if some reference combination is under-represented in the training distribution, DAR can only redistribute attention and cannot conjure a learning signal for it out of nothing. Reference counts are evaluated up to 7, and whether the method holds beyond that is untested.
- The reward models themselves are a fixed ceiling. CSD, CLIP and SigLIPv2 are all frozen scorers, and DRS can only amplify differences they already detect — if a scorer is completely insensitive to a failure mode (say, a locally broken structure whose global CLIP similarity stays high), DRS cannot manufacture discriminability, and the remedy must come from a stronger reward model.
- The key DAR hyper-parameters \(\gamma\) and the weight range \([w_{\min}, w_{\max}]\) are not given in the main text, only analyzed for sensitivity in the appendix, leaving the tuning burden to anyone reproducing the work.
- Directions for improvement: make the DRS transformation \(F\) adaptive to training progress (mild early, aggressive later), or let DAR's difficulty measure use not only mean reward but also prior difficulty signals such as reference count and the number of reference types — both are worth trying.
Related Work & Insights¶
- vs DreamOmni2 / OmniGen2: these are unified multimodal instruction-based editing and generation models where multi-reference ability is a by-product; they support at most 2 reference-type combinations and evaluate with a coarse overall score. The key difference here is that reference count and type complexity are treated as first-class citizens, with a dedicated benchmark and a training objective designed for complexity imbalance. The trade-off is that this method does not broaden the base model's overall capability — the gains concentrate on the multi-reference dimension.
- vs USO (Unified Style and Subject-driven Generation via Disentangled and Reward Learning): USO likewise uses reward learning for disentangled style-and-subject generation, and this paper borrows its CSD style reward directly. The difference is that USO targets "one style + one subject" settings, whereas this paper must handle groups of up to 7 reference images and up to 4 concurrent types, which forces it to also solve the inter-group difficulty imbalance — precisely why DAR/DRS exist.
- vs standard Flow-GRPO: Flow-GRPO provides the skeleton for online RL on flow matching models, but its intra-group advantages assume samples from the same prompt batch are comparable in difficulty. DAR breaks exactly that assumption by treating groups as units with differing difficulty, and can be seen as the group-level counterpart of Focal Loss's approach to sample imbalance in RL. That analogy is itself a transferable insight — the toolbox used against foreground/background imbalance in object detection may carry over wholesale to group-level weighting in RL.
- vs multi-subject benchmarks such as PSRBench / XVerse: their evaluation dimensions center on subject identity consistency and they do not include expert-annotated complex combinations. OmniRef-Bench widens reference types to five, combinations to ten (up to four concurrently) and reference images to 2-7, and provides both objective metrics and multi-dimensional MLLM scores; the evaluation protocol itself (objective + MLLM, validated for human correlation) is another reusable output of this work.
Rating¶
- Novelty: ⭐⭐⭐⭐ [Both the problem definition (complexity-driven performance degradation) and the solution (group-level inverse difficulty weighting plus reward discriminability scaling) are well aimed, though each component individually — Focal-style weighting, reward transformation — has precedents elsewhere; the novelty lies in the combination and the problem framing]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [A self-built benchmark plus two backbones plus four external benchmarks (two easier MRIG and two single-image editing) plus per-component ablations plus a 30-person user study with correlation analysis; very complete coverage]
- Writing Quality: ⭐⭐⭐⭐ [The motivation chain advances clearly from "benchmark gap → observed degradation → located training-signal imbalance → two orthogonal corrections", and the figures (Fig. 3, Fig. 5) directly support the argument; points deducted because key hyper-parameter values and the concrete DRS transformation are deferred to the appendix]
- Value: ⭐⭐⭐⭐ [Delivers both a usable evaluation benchmark and a training recipe that is extremely cheap to implement (only loss weighting changes, no architecture change) and transfers across backbones, directly reusable for teams working on multi-reference generation and multi-condition control]