DRPO: Disentangling Demographic Bias from Rewards for Fair Diffusion Alignment¶

Conference: ECCV 2026
Paper: ECCV Official
Area: Image Generation
Keywords: Diffusion Models, Preference Optimization, Reward Model Bias, Algorithmic Fairness, Feature Orthogonal Projection

TL;DR¶

Addressing the issue where preference optimization (PO) amplifies demographic biases encoded in reward models—causing representation collapse in text-to-image diffusion—DRPO extracts demographic directions via linear probing and projects them out via orthogonal complement before reward calculation, achieving high-fidelity generation with restored demographic balance.

Background & Motivation¶

Text-to-image diffusion models are increasingly aligned with human preferences via preference optimization methods such as Diffusion-DPO and DanceGRPO, relying on learned reward models like HPS-v2, PickScore, and ImageReward to guide policy updates toward superior aesthetic quality and prompt adherence. However, these popular reward models are built upon CLIP visual backbones whose demographic biases along race, gender, and age axes are well documented.

When diffusion models undergo preference fine-tuning to maximize these scalar reward signals, they inadvertently discover and exploit demographic shortcuts. For neutral prompts such as "a photo of a person" or occupation descriptions, minority-group representation drops by roughly 40% after GRPO alignment, and race distribution entropy degrades monotonically throughout training. Critically, standard diversity-preserving methods such as DRIFT and DiverseGRPO fail to remedy this collapse: while they preserve visual variation across poses, backgrounds, and artistic styles, they leave demographic distributions skewed. Probing analysis reveals that reward features encode demographic labels with high linear predictability, explaining 14% to 22% of reward variance entirely independent of visual quality.

To break this shortcut exploitation without degrading alignment fidelity, debiasing must intervene at the origin of the optimization pressure. The core idea is to identify the low-dimensional demographic subspace within reward feature representations via linear probing and singular value decomposition (SVD), and project out these directions using an orthogonal projection operator prior to reward computation, providing a plug-and-play debiased reward for arbitrary preference optimization algorithms.

Method¶

Overall Architecture¶

DRPO decouples demographic attributes from quality signals at the reward feature representation layer. The overall system operates across two phases: Stage 1 is an offline one-time preparation phase (Demographic Subspace Identification) that samples synthetic images from the unaligned base diffusion model on neutral prompts, extracts reward features, assigns demographic labels via FairFace, and trains a linear probe whose SVD yields an orthonormal basis \(\mathbf{V}_k\) of the demographic subspace. Stage 2 is online preference optimization (PO Training with Debiased Rewards), where during policy gradient or preference loss computation, every batch of candidate images has its reward features projected onto the orthogonal complement of \(\mathbf{V}_k\) before scoring by the MLP reward head.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
    subgraph S1["Stage 1: Demographic Subspace Identification"]
        direction TB
        A["Generate images with neutral prompts"] --> B["FairFace auto-labeling + CLIP feature extraction"]
        B --> C["Demographic subspace extraction<br/>SVD of linear probe weights"]
    end
    subgraph S2["Stage 2: Debiased Preference Optimization"]
        direction TB
        D["Diffusion policy generates image candidates"] --> E["Extract reward features via CLIP vision encoder"]
        C -.->|"Provide basis Vk"| F
        E --> F["Feature-level orthogonal projection<br/>P⊥ = I - VkVk^T"]
        F --> G["Compute debiased scalar rewards via MLP head"]
        G --> H["Update policy weights<br/>Compatible with GRPO / DPO"]
    end

Key Designs¶

1. Demographic Subspace Extraction: Isolating linear demographic directions

To surgically remove demographic bias from high-dimensional representations, the method first determines the geometry of bias encoding. Linear probing experiments reveal that 2-layer MLP probes outperform simple linear probes by only 0.6%–1.8% in predicting race and gender from CLIP reward features, demonstrating that demographic attributes are encoded along predominantly linear directions. DRPO samples 10,000 images from neutral prompts and trains a linear classification probe \(\mathbf{W} \in \mathbb{R}^{C \times d}\), where \(C\) is the number of demographic classes and \(d\) is the feature dimension (\(d=1024\)). Singular value decomposition is performed on the weight matrix:

\[\mathbf{W} = \mathbf{U} \mathbf{\Sigma} \mathbf{V}^\top\]

The top-\(k\) right singular vectors corresponding to the largest singular values are collected to construct the orthonormal basis \(\mathbf{V}_k = [\mathbf{v}_1, \dots, \mathbf{v}_k] \in \mathbb{R}^{d \times k}\). This basis spans the principal demographic subspace. Geometric analysis shows that the minimal principal angle between this demographic subspace and the quality-predictive subspace is \(\theta_{\min} = 72.4^\circ\), establishing that demographic indicators and aesthetic quality signals reside in near-orthogonal directions.

2. Feature-Level Orthogonal Projection: Plug-and-play debiased rewards

Rather than adjusting final scalar reward outputs by subtracting group means—which fails to untangle high-dimensional shortcuts—DRPO performs an algebraic projection in the intermediate feature space. Defining the orthogonal complement projection operator as \(\mathbf{P}_\perp = \mathbf{I} - \mathbf{V}_k \mathbf{V}_k^\top\), the debiased feature representation \(\tilde{\phi}(\mathbf{x})\) for an image \(\mathbf{x}\) is computed as:

\[\tilde{\phi}(\mathbf{x}) = \mathbf{P}_\perp \phi(\mathbf{x}) = \phi(\mathbf{x}) - \mathbf{V}_k \mathbf{V}_k^\top \phi(\mathbf{x})\]

The projected feature \(\tilde{\phi}(\mathbf{x})\) is subsequently forwarded through the frozen MLP scoring head \(h\) to yield the debiased reward \(\tilde{r}(\mathbf{x}) = h(\tilde{\phi}(\mathbf{x}))\). Because only \(k=5\) dimensions out of \(d=1024\) are eliminated, the feature norm perturbation \(\|\phi - \tilde{\phi}\| / \|\phi\|\) is under 3%. This projection reduces race and gender probe accuracy to near-chance levels (\(R^2\) drops from 0.19 to 0.01) while preserving the head's fine-grained assessment of composition and aesthetic fidelity, adding less than 0.1% wall-clock computational overhead per training step.

Loss & Training¶

The debiased reward \(\tilde{r}(\mathbf{x})\) directly replaces the standard reward score across standard alignment loss functions. For DanceGRPO, advantages across \(K\) generated samples per prompt are calculated as:

\[A_i = \tilde{r}(\mathbf{x}_i) - \frac{1}{K}\sum_{j=1}^K \tilde{r}(\mathbf{x}_j)\]

and optimized via on-policy policy gradient. For Diffusion-DPO, image pairs are re-ranked using \(\tilde{r}\) to construct direct preference pairs. No demographic metadata is required during the online optimization phase, leaving the base diffusion network architecture and conditioning mechanisms completely untouched.

Key Experimental Results¶

Main Results¶

The primary evaluation conducted on SDXL across 500 demographic-neutral prompts (generating 10,000 images per method) is summarized below (from Table 2 of the original paper). Baselines include unaligned Base SDXL, standard PO (Diff-DPO, DanceGRPO), diversity-regularized PO (DRIFT, DiverseGRPO), and inference-time guidance (ITI-GEN+GRPO).

Method	HPS-v2↑	PickScore↑	ImageReward↑	FID↓	Vendi↑	Race Entropy (Race-H)↑	Gender Entropy (Gen-H)↑	DPD↓	Worst-Group Rate (WGR)↑
Base SDXL	25.81	20.14	0.42	44.7	0.44	1.74	0.97	0.18	0.08
Diff-DPO	27.08	21.27	0.81	40.3	0.37	1.08	0.78	0.41	0.02
DanceGRPO	27.42	21.63	0.93	38.9	0.34	0.89	0.69	0.49	0.01
DRIFT	27.18	21.41	0.87	33.1	0.73	0.93	0.72	0.46	0.02
DiverseGRPO	27.29	21.52	0.90	33.8	0.71	0.96	0.73	0.44	0.02
ITI-GEN+GRPO	26.84	21.08	0.79	39.2	0.40	1.43	0.88	0.27	0.05
DRPO (w/ DPO)	26.93	21.19	0.78	37.4	0.62	1.61	0.94	0.21	0.07
DRPO (w/ GRPO)	27.31	21.54	0.91	35.9	0.67	1.67	0.96	0.19	0.07

Ablation Study¶

The impact of the projection dimension \(k\), the debiasing level, and demographic classifier robustness evaluated on SDXL with GRPO alignment (from Table 3 of the original paper):

Ablation Setting	Variant	Quality (HPS-v2)↑	Race Entropy (Race-H)↑	Principal Angle \(\theta_{\min}\)	Note
Baseline (no projection)	DanceGRPO	27.42	0.89	—	Severe demographic narrowing
Projection dimension \(k\)	\(k=1\)	27.39	1.34	—	Partial fairness recovery
Projection dimension \(k\)	\(k=3\)	27.35	1.58	—	Moderate improvement
Projection dimension \(k\)	\(k=5\) (default)	27.31	1.67	72.4°	Optimal trade-off (-0.11 HPS, near-base fairness)
Projection dimension \(k\)	\(k=10\)	27.14	1.69	—	Marginal fairness gain, accelerated quality loss
Debiasing level	Output-level (group mean)	27.38	1.23	—	Fails to eliminate high-dimensional shortcuts
Debiasing level	Feature-level (ours)	27.31	1.67	72.4°	Orthogonal geometric disentanglement
Classifier robustness	CLIP zero-shot	27.33	1.59	—	Effective without FairFace
Classifier robustness	FairFace + 20% label noise	27.32	1.56	—	Robust to imperfect annotations

Key Findings¶

Progressive and rapid bias amplification: In standard GRPO, race entropy falls monotonically from 1.74 to 0.89 over 2,000 steps (-49%), and the worst-group generation rate drops from 8% to 1%. The model seizes upon demographic shortcuts in early training iterations before substantial aesthetic gains emerge.
Independence of visual diversity and demographic fairness: DRIFT and DiverseGRPO boost visual diversity (Vendi score from 0.34 to 0.73/0.71) but fail to restore demographic fairness (Race-H remaining at 0.93/0.96), establishing that stylistic dispersion and demographic parity occupy distinct feature dimensions.
Preserved human visual preference: In a double-blind 2AFC human evaluation spanning 900 pairs, DRPO achieved a 47.3% preference rate against DanceGRPO (95% CI [41.6, 53.0]), demonstrating perceptual equivalence to unconstrained alignment, while outperforming Base SDXL with a 72.1% win rate.

Highlights & Insights¶

Diagnosing the demographic failure mode of preference alignment: Identifies that reward models based on CLIP entangle demographic information with aesthetic quality, showing that preference optimization systematically penalizes demographic diversity unless actively decoupled.
Surgical low-rank feature projection: Eliminating merely \(k=5\) dominant singular directions via \(\mathbf{P}_\perp\) in a 1024-dimensional feature space removes 95%+ of predictable demographic signal without perturbing visual evaluation capabilities.
Seamless integration across PO paradigms: Functions upstream of the objective formulation, introducing minimal overhead (<0.1% runtime) and operating identically with on-policy RL (GRPO) and offline margin ranking (DPO).

Limitations & Future Work¶

Reliance on discrete facial demographic taxonomies: Uses FairFace's predefined 7 race categories and binary gender classes for offline probe fitting, which oversimplifies intersectional, continuous, or fluid demographic identities.
Linear subspace assumptions: While validated by close parity between linear and MLP probing accuracies, non-linear demographic representations may persist in more complex vision-language foundation models.
Non-remedial regarding base model imbalances: DRPO prevents preference alignment from worsening demographic skews relative to the starting model, but does not autonomously synthesize under-represented demographics absent in the base pretrained distribution.

vs DRIFT / DiverseGRPO: Previous works incentivize diverse generations by penalizing mode collapse in image or latent space, failing to target the demographic bias in the reward function. DRPO cleanses the reward signal directly, making it complementary to visual diversity mechanisms.
vs ITI-GEN / FairDiffusion: ITI-GEN modifies text embeddings during inference, inducing trade-offs that lower aesthetic scores (HPS down by 2.1%), whereas FairDiffusion intervenes during sampling trajectories. DRPO operates during alignment training at the reward level, conferring intrinsic fairness to the fine-tuned generator.

Rating¶

Novelty: ⭐⭐⭐⭐☆ Pinpoints the mechanism of demographic bias amplification in diffusion preference optimization and resolves it via linear feature projection.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous validation across multiple diffusion backbones, reward models, alignment paradigms, metric suites, and double-blind human evaluations.
Writing Quality: ⭐⭐⭐⭐⭐ Clear progression from motivational probing to geometric formalization, ablation studies, and qualitative demonstrations.
Value: ⭐⭐⭐⭐⭐ Offers a practical, low-overhead blueprint for deploying safe and ethically fair diffusion alignment in production generative AI systems.