DivRL: Disentangled Self-Similarity Rewards for Diverse Subject-Driven Generation¶

Conference: ECCV2026
arXiv: 2606.23950
Code: https://github.com/QianWangX/DivRL
Area: Image Generation / Diffusion Models
Keywords: Subject-driven generation, identity-diversity paradox, reinforcement learning, self-similarity metric, visual-semantic matching

TL;DR¶

DivRL proposes a Flow-GRPO-based post-training framework that measures structural diversity and identity consistency using disentangled visual features, employing a two-stage "explore-and-suppress" optimization strategy to enable the model to simultaneously generate images with high identity preservation and high structural diversity.

Background & Motivation¶

Subject-driven image generation aims to reconstruct the visual identity of a subject in various new scenes, poses, and styles given a reference image. Native multi-modal diffusion models, such as Flux-Kontext and OmniGen, have demonstrated strong fidelity. However, they face a fundamental conflict between identity consistency and structural diversity: the more identity preservation is emphasized, the more the generated images tend to "look identical" to the reference image—sharing the same perspective, pose, and spatial layout. Conversely, encouraging more diversity often leads to deviations from the subject's identity features. This work refers to this tension as the "identity-diversity paradox."

The deep root of this paradox lies in the tight entanglement of identity features and spatial structure within the reference image. When the model captures what the subject looks like, it inevitably learns how the subject is positioned in the reference image. Conversely, when attempting to generate images with different structures, it might alter critical identity-defining attributes. Most existing methods alleviate this conflict using pairwise preference ranking—allowing the model to compare which of two generated images is more consistent and which is more diverse. However, such relative optimization lacks explicit disentanglement, making it difficult to simultaneously advance both objectives.

The core observation of this work stems from a prior study, MTG (Mind the Glitch), which discovered that disentangled pure visual features (separated from semantic features) extracted from the diffusion model backbone are highly robust to spatial transformations. Even if the subject changes its pose or perspective, these visual features can still reliably reflect identity consistency. However, using MTG features as the sole reward is insufficient: the model can "cheat" by replicating the spatial layout of the reference image to obtain high identity rewards. Visually, a mechanism that explicitly measures whether the structure is diverse is still lacking. Core Idea: Leverage the self-similarity matrix of disentangled visual features to measure structural diversity (rather than global feature differences), and design a gated optimization strategy that converts identity consistency from a competing objective into a feasibility constraint, enabling synergistic optimization of both reward types under the GRPO framework.

Method¶

Overall Architecture¶

DivRL is an RL post-training framework based on Flow-GRPO. It takes a reference image and a text prompt as input and outputs images that preserve identity while exhibiting diverse structures. The framework consists of three core components: two reward models (VSM measuring identity consistency, and nSSM measuring structural diversity) and a two-stage "explore-and-suppress" optimization strategy. In the first stage, only nSSM is used as the reward to let the model freely explore diverse spatial configurations. In the second stage, VSM is introduced as a gated constraint—applying quadratic penalty only when the identity consistency of the generated sample falls below a threshold; otherwise, it continues to maximize nSSM. This design demotes identity preservation from a competing objective to a feasibility constraint, preventing consistency and diversity from undermining each other.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Reference Image + Text Prompt"] --> B["MTG Feature Extractor<br/>Disentangled Visual/Semantic Features"]
    B --> C1["VSM: Identity Consistency<br/>Semantic+Visual Collaborative Matching"]
    B --> C2["nSSM: Structural Diversity<br/>Pearson of Self-Similarity Matrix"]

    C1 --> D{"Stage 1: Exploration<br/>R = nSSM<br/>Freely explore diverse structures"}
    D --> E["Generate Diverse Candidate Sample Set"]
    E --> F{"Gating Decision<br/>VSM >= s?"}
    F -->|Yes| G["Reward = nSSM<br/>Continue encouraging diversity"]
    F -->|No| H["Reward = nSSM - λ(s-VSM)²<br/>Quadratic Penalty"]
    G --> I["Flow-GRPO Policy Update<br/>Stage 2: Suppression"]
    H --> I
    I --> J["Output: High Identity Consistency<br/>+ High Structural Diversity Images"]

Key Designs¶

1. nSSM: Negative Self-Similarity Measure—Measuring Structural Diversity via Internal Feature Organization

Traditional methods use global feature differences (such as CLIP image similarity) to measure diversity, but global features encode both identity and structure concurrently, making them deeply entangled. The insight of this paper is: if disentangled visual feature grids (\(48 \times 48\) dense feature maps) are extracted from the reference and generated images respectively, each grid point represents local visual information in that position. Thus, the "self-similarity matrices" within each of these feature grids encode the spatial organizational structure of the object. High correlation implies a similar spatial layout, whereas low correlation corresponds to different poses or perspectives.

Specifically, the pure visual feature maps \(\mathcal{F}^v_{ref}\) and \(\mathcal{F}^v_{gen}\) of the reference and generated images are first extracted via the MTG network, normalized along the channel dimension, and then their respective self-similarity matrices are calculated as \(\mathcal{M}^v = \hat{\mathcal{F}}^v (\hat{\mathcal{F}}^v)^\top\). The \((i,j)\)-th element of this matrix measures the visual pattern similarity between the \(i\)-th and \(j\)-th spatial locations, serving essentially as a "spatial skeleton" description of the object. Subsequently, a ground-truth object mask is applied to filter out the background, and the Pearson correlation coefficient between the two self-similarity matrices is computed:

\[\text{nSSM} = 1 - \rho(\hat{\mathcal{M}}^v_{ref}, \hat{\mathcal{M}}^v_{gen})\]

When the two matrices are highly correlated, nSSM is low (structurally similar, poor diversity); when the correlation is low, nSSM is high (structurally diverse). Since nSSM operates on dense latent feature grids, it captures fine-grained geometric organization information, which is more sensitive to structural variations than global features.

2. VSM: Visual-Semantic Matching—Providing a Reliable Metric for Identity Consistency

Identity consistency requires a metric that is both precise and robust. Existing methods often directly compute the cosine similarity of global features from CLIP or DINO, but these are not sufficiently robust to pose changes—the feature similarity significantly drops when 'the same person changes pose'. This paper introduces a two-level matching using disentangled MTG features: first, the semantic feature maps of the reference and generated images are extracted to compute coordinate-wise semantic similarity. Areas with semantic correspondence (ensuring that only 'the same part' is compared) are selected using a semantic threshold \(\tau_s\). Within these semantically matched regions, the consistency of visual features is computed. VSM measures 'the proportion of pixels with semantic correspondences that exhibit visual similarity exceeding a threshold \(\tau_v\)'. A higher value indicates better identity preservation. This metric is naturally insensitive to perspective and pose variations because it aligns semantically prior to comparison, avoiding reliance on global spatial alignment.

3. Two-Stage "Explore-and-Suppress" Optimization—Turning Identity Preservation from an Objective into a Constraint

Optimizing two objectives (consistency and diversity) simultaneously in RL often leads to gradient conflicts and reward hacking—e.g., the model learns that generating random noisy images yields a high nSSM score. The proposed two-stage strategy elegantly disentangles these two objectives. In the first stage, the identity reward is disabled, allowing the model to freely explore various structural configurations guided solely by nSSM. During this stage, the model might generate diverse samples that occasionally deviate from the identity. In the second stage, a gating mechanism is introduced, and the reward becomes a piecewise function:

\[ R_2 = \begin{cases} \text{nSSM} & \text{VSM} \geq s \\ \text{nSSM} - \lambda (s - \text{VSM})^2 & \text{VSM} < s \end{cases} \]

When the sample's identity consistency meets or exceeds the threshold \(s\), only diversity (nSSM) is rewarded, allowing the model to explore different structures with confidence. Once a sample deviates from the target identity past the threshold, a quadratic hinge penalty is applied, which is proportional to the square of the deviation level. The key aspect of this design is that identity preservation is no longer an objective to be continually 'maximized', but rather a constraint to be 'satisfied'. As long as the model does not cross the boundary, it focuses entirely on structural innovation. This 'objective-constraint separation' successfully avoids the optimization dilemma inherent in linearly weighting two competing objectives.

4. Downsampling for Stable Training—Mitigating High-Frequency Artifacts

The spatial resolution of MTG visual features is \(48 \times 48\). Directly computing nSSM at this resolution implicitly encourages the model to preserve dense local correlation patterns, leading to high-frequency texture artifacts during RL training—where the model climbs the nSSM score by creating fine textures instead of actually altering the structure. The solution is straightforward: apply \(2 \times 2\) average pooling to the normalized visual features before computing nSSM, downsampling them to \(24 \times 24\). This acts as a spectral low-pass filter, forcing the model to enhance diversity at a coarser, structural level. Ablation studies confirm that the downsampled version significantly outperforms the original-resolution counterpart in terms of visual quality.

Loss & Training¶

The overall training utilizes the Flow-GRPO framework with a group size of 21, 6 denoising steps (28 steps during inference), and a KL regularization coefficient of \(\beta=0.1\). Each of the two stages undergoes 3200 optimization steps, taking approximately 24 hours per stage across 8 A100 80GB GPUs. LoRA weights add only 1.8% parameters. Regarding hyperparameters, the semantic and visual thresholds are set to \(\tau_s=\tau_v=0.7\), the gating threshold \(s=0.5\), and the penalty weight \(\lambda=5\).

Key Experimental Results¶

Main Results¶

Evaluated on the DreamBench++ benchmark (150 subjects, covering animals, humans, objects, and style transfer) using the Flux-Kontext backbone:

Model	CLIP-T↑	CLIP-I↑	DINO↑	VSM↑	DINO-nSSM↑	MTG-nSSM↑	sIoU↓
Flux-Kontext	0.283	0.781	0.594	0.605	0.433	0.659	0.694
PaCo-RL	0.287	0.766	0.570	0.576	0.445	0.668	0.683
DivRL (VSM only)	0.278	0.806	0.670	0.688	0.383	0.608	0.757
DivRL (nSSM only)	0.287	0.756	0.520	0.562	0.489	0.704	0.634
DivRL (full)	0.279	0.786	0.600	0.614	0.453	0.689	0.697

Two extreme variants peak at their respective single objectives: "VSM only" dominates all identity consistency metrics (CLIP-I, DINO, VSM) but suffers a significant drop in diversity (DINO-nSSM, MTG-nSSM); "nSSM only" achieves the best diversity but the worst identity preservation. The full model maintains identity consistency comparable to the baseline (VSM 0.614 vs 0.605) while substantially boosting diversity (MTG-nSSM 0.689 vs 0.659), successfully occupying the "high-value zone" in the solution space where both goals coexist.

Ablation Study¶

Configuration	Key Metrics	Description
Linear weighting of VSM+nSSM	Fails to improve consistency and diversity simultaneously	Gradients directly compete; no Pareto improvement on the trade-off frontier
Two-stage gating (Ours)	Simultaneous improvement in consistency and diversity	Explore-and-suppress decoupling; VSM and nSSM cooperate
nSSM original 48×48 resolution	High-frequency artifacts	Model hacks rewards via fine textures
nSSM downsampled 24×24	Clean visuals, comparable diversity	Low-pass filtering effectively suppresses artifacts

Key Findings¶

Structural diversity yields a prompt alignment dividend: Although the "nSSM only" variant does not explicitly optimize text alignment, its CLIP-T score is the highest (0.287). This is because identity retention training inherently introduces geometric rigidity; encouraging structural diversity relaxes this rigidity, allowing the model to better adapt the subject to the contextual requirements of the prompt.
Intrinsic difference between gating and linear weighting: Under linear weighting, the gradient directions of the two rewards oppose each other, preventing simultaneous improvement of both objectives at any ratio. Conversely, the gating strategy explores first and then suppresses, avoiding concurrent gradient conflicts.
Backbone-agnostic nature of the method: Replacing MTG features with DINOv2 features and replacing VSM with cosine similarity yielded performance comparable to the default settings when re-running the entire pipeline, proving that the explore-and-suppress framework is a general paradigm.

Highlights & Insights¶

The ingenuity of nSSM lies in replacing "inter-feature differences" with "internal feature structural relationships": Traditional contrastive metrics (such as computing feature cosine distance) are inherently plagued by the coupling of identity and structural features. In contrast, the self-similarity matrix characterizes the "internal spatial organization of the object" rather than "what the object looks like," naturally decoupling identity and structure.
"Turning objectives into constraints" is an elegant paradigm for handling conflicting multi-objective RL: It does not demand maximization of identity consistency (which inevitably sacrifices diversity), but only requires it to remain above a threshold. This "good enough" design is often much more stable in practice than "the larger, the better."
The 2×2 pooling fix for high-frequency artifacts is a classic "small change, big payoff" trick: Without introducing complex regularization, simply downsampling a feature map dramatically improves visual quality. This is highly reusable in similar dense feature-driven RL scenarios.
nSSM is sensitive to structural changes and insensitive to semantic shifts: Even if the overall semantics of the image change (e.g., background substitution), nSSM remains relatively low as long as the spatial organization of the subject remains unchanged. Conversely, pose changes in the subject significantly boost nSSM. This ensures that the diversity reward truly drives structural variations rather than semantic drift.
The unexpected discovery that structural diversity automatically improves prompt alignment suggests that the "side effects" of identity-preservation training might be more severe than previously assumed. Future work could potentially replace complex prompt alignment training with relaxed structural constraints.

Limitations & Future Work¶

nSSM is currently primarily sensitive to structural changes in pose and viewpoint, and does not yet cover higher-level semantic diversity (such as composition changes, interaction modes) or combinatorial diversity variations.
RL post-training incurs additional computational overhead (48 hours on 8 A100s), which is still costly compared to purely feed-forward methods (e.g., adapters).
The base model, Flux-Kontext, only accepts a single reference image. When the reference image contains ambiguous identity cues or complex occlusions, the performance may become unstable. Extending this to multi-reference image conditioning is a natural direction.
In certain extreme cases, the model still exhibits high-frequency artifacts (inherited from the conditioning mechanism of Flux-Kontext). Although \(24 \times 24\) pooling mitigates this, resolving the root issue still requires architecture-level improvements.

vs PaCo-RL / Identity-GRPO: These methods utilize pairwise preference ranking for relative optimization, whereas this paper performs absolute optimization through explicit decoupled metrics and gated constraints. The former can only compare "which is better," while the latter can "directly drive diversity."
vs Linearly Weighted Multi-Reward RL: Gradient conflict is inevitable under linear weighting. The proposed two-stage gating provides an alternative solution—instead of searching for points on the Pareto frontier, it defines one objective as a feasibility region and freely optimizes the other objective within this region.
vs MTG (Mind the Glitch): MTG provides a powerful tool in disentangled visual features, but using them directly as an RL reward causes the model to duplicate the reference image's spatial layout. The proposed nSSM is the key to enabling explicit measurement of "diversity."

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Measuring structural diversity with the negative Pearson correlation coefficient of self-similarity matrices, along with the two-stage "explore-and-suppress" gating strategy, both designs are highly original and inspiring.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation on DreamBench++, including three-way comparisons among VSM-only/nSSM-only/full, comprehensive comparison with linear weighting, threshold ablation, backbone generalization verification, qualitative examples with multiple prompts/seeds, and failure case analysis.
Writing Quality: ⭐⭐⭐⭐ The terminology is refined and the chain of motivation is clear. However, the formulation in the methodology section features slightly too many equations and complex notations, affecting readability.
Value: ⭐⭐⭐⭐⭐ Directly addresses the most central conflict in subject-driven generation (the identity-diversity paradox). The nSSM paradigm is naturally transferable to other visual generation tasks requiring the decoupling of consistency and diversity.