Diffusion-SDPO: Safeguarded Direct Preference Optimization for Diffusion Models

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/AIDC-AI/Diffusion-SDPO
Area: Image Generation
Keywords: diffusion models, direct preference optimization, gradient geometry, winner preservation, output-space approximation

TL;DR

Diffusion-SDPO adaptively scales the loser's backward gradient using its relationship with the winner gradient, addressing the failure of diffusion DPO in which the preference margin improves while both branches deteriorate; in the paper's FLUX.1-dev setting, GenEval increases from the base model's 0.66 to 0.71, whereas ordinary DPO at the same learning rate drops to 0.23.

Background & Motivation

Preference fine-tuning for text-to-image models typically uses two images associated with the same prompt: the preferred image is the winner, and the other is the loser. Diffusion-DPO translates this ranking into a comparison of denoising errors relative to a frozen reference model, encouraging better reconstruction of the winner and relative movement away from the loser. It requires neither an additional online reward model nor a full reinforcement learning procedure, but optimizes the relative gap rather than directly constraining either branch's absolute error.

The difficulty is that both branches share model parameters. An update that increases the loser's error can also increase the winner's error; as long as the loser deteriorates faster, the DPO ranking objective can still improve. Figures 1 and 3 illustrate this failure: the preference margin moves in the intended direction while both denoising losses increase, and generated images gradually develop oversaturation and texture artifacts. Reducing the learning rate can prevent some collapses but may also prevent meaningful preference learning. The central issue is therefore identifying the harmful update component, not uniformly slowing training.

The paper addresses this through the direction of each update: retain the gradient that reduces winner error and weaken the loser contribution only when its opposing effect could cancel that improvement. Core Idea: attach a winner-preserving condition to relative preference optimization, then use inexpensive output-space gradients to approximate how much loser update each step can safely apply. The protection originates in a local first-order analysis, and the practical implementation includes an approximation; it does not imply that generation quality must improve monotonically after every finite training step.

Method

Overall Architecture

The inputs remain a prompt and a winner/loser image pair. The trainable and frozen reference models process the noised images, and both branch residual objectives are evaluated at the same diffusion timestep. SDPO obtains a safety coefficient from the output-gradient inner product and the squared norm of the winner gradient, then applies it only to the loser branch's backward pass.

The output is a preference-tuned generator. At inference time, neither the reference model, image pairs, nor extra safety-coefficient calculations are required. This is not an additional generation network but an update rule that can augment Diffusion-DPO, DSPO, or DMPO. The explanation therefore focuses on gradient mechanics rather than depicting a loss modification as a multi-module system.

Two kinds of loss must be distinguished. Each branch residual objective contains the current model's reconstruction error minus the frozen reference model's reconstruction error; the outer DPO loss compares these branch objectives. For a fixed sample, noise realization, and timestep, the reference term does not change with the parameter update, so reducing the winner residual objective also reduces its current reconstruction error. This does not imply that raw losses evaluated on different random minibatches must decrease at every step.

Key Designs

1. Winner preservation: check whether an update harms the winner before enlarging the preference gap

The local direction of ordinary DPO combines decreasing the winner loss with increasing the loser loss. The outer logistic function multiplies both components by a shared positive scale. The paper first analyzes a linearized objective with a weight on the loser component: the parameter-space gradients of the two branch losses enter the update with opposing signs. The relevant question is whether the combined direction has a nonpositive directional derivative on the winner loss, not merely whether the scalar DPO loss decreases.

The first-order expansion shows that the squared norm of the winner gradient supplies a descent budget, while the loser weight times the two gradients' inner product consumes that budget. When the inner product is positive, the loser update cannot be strengthened arbitrarily: its weight is bounded by the winner gradient's squared norm divided by that inner product. A nonpositive inner product does not create this conflict because the component that increases loser error does not increase winner error to first order. Apparently aligned loss gradients are dangerous here precisely because preference training increases the loser loss while decreasing the winner loss.

This criterion explains why margin-only monitoring is insufficient and why uniformly shrinking all gradients is not equivalent. SDPO retains the winner's learning signal while selectively limiting a harmful loser component. The result concerns a linearized, sufficiently small gradient step, not unconditional convergence of the complete nonlinear training process.

2. Output-space safety coefficient: approximate expensive parameter geometry with inexpensive residual signals

Applying the criterion directly requires separate full parameter gradients for both branches and their inner product, introducing substantial memory and computation costs for large models. The authors instead use gradients with respect to model output tensors: \(g^w\) and \(g^l\) denote winner and loser loss gradients with respect to outputs such as predicted noise. For squared residual losses, these signals are directly related to prediction residuals, avoiding storage of two full parameter-gradient vectors solely to construct the coefficient.

Each branch's Jacobian separates output gradients from parameter gradients, so an output-space inner product is not exactly the corresponding parameter-space inner product. Equation 16 represents this difference through a geometry factor \(\rho\). The implementation does not explicitly estimate it; instead, a safety slack \(\mu\) contracts the output-space estimate. The following combines Algorithm 1, step 4, with the clipping and branch rules explicitly described in Section 4.2; it does not reconstruct the damaged Jacobian equation in the text extraction:

\[ \lambda_{\mathrm{safe}}= \begin{cases} 1, & (g^w)^\top g^l\leq 0,\\ \operatorname{clip}_{[0,1]}\!\left(\dfrac{(1-\mu)\lVert g^w\rVert_2^2}{(g^w)^\top g^l}\right), & (g^w)^\top g^l>0. \end{cases} \]

A nonpositive inner product retains the original loser weight. For a positive inner product, a smaller winner descent budget or stronger cross-branch effect permits less loser influence. Larger \(\mu\) means more conservative contraction, and clipping to \([0,1]\) prevents amplification beyond the original loser weight. This neither adds a reward nor discards lower-quality images; it adjusts their gradient influence step by step.

Efficiency comes with a clear qualification: \(\mu\) mitigates unknown local geometry but is not a measured \(\rho\), and it does not automatically establish the condition for arbitrary Jacobians. The output-space and parameter-space coefficient trajectories in Figure 2 are close, but the authors explicitly state that they use different values of \(\mu\). This supports approximation after adjustment, not exact equivalence without tuning.

3. Backward-only scaling: preserve DPO's forward value and preference-discrimination scale

Multiplying the loser loss by the safety coefficient before feeding it into the logistic function would change the preference margin, the outer weight, and the gradient direction simultaneously. Algorithm 1 instead retains the loser loss's forward value and scales only its backward contribution. With \(\operatorname{sg}\) denoting stop-gradient, the following is an equivalent notation for step 5, treating the safety coefficient as a constant for the current update:

\[ \widetilde{\mathcal L}^{l} =\operatorname{sg}(\mathcal L^{l}) +\lambda_{\mathrm{safe}}\bigl(\mathcal L^{l}-\operatorname{sg}(\mathcal L^{l})\bigr). \]

The parenthesized term is zero in the forward pass, so the value remains the original loser loss. In the backward pass, stopped terms contribute no derivative, leaving the loser gradient scaled by the safety coefficient. This preserves DPO's assessment of the current preference gap while applying geometric protection to the update direction. An implementation should preserve this backward-scaling behavior rather than merely change the forward preference score.

This also enables integration with different preference objectives: there is no need to replace every method with one shared loss, only to scale the appropriate loser branch. DSPO already progressively increases winner weighting, so its additional gains are smaller; Diffusion-DPO and DMPO show more pronounced corrections to their training trajectories. The mechanism targets coupling through shared parameters rather than any particular UNet or DiT layer structure.

Loss & Training

Algorithm 1 samples an image pair, timestep, and noise, runs the trainable and frozen reference models, computes both residual objectives, the safety coefficient, and the stop-gradient loser term, then updates the model using the original logistic DPO form. The symbol-misaligned expanded DPO equation in the cached text is not reproduced here; the mechanism rests on the clearly described branch-error difference and backward-scaling rule.

SD 1.5 and SDXL use Pick-a-Pic V2 preference pairs. The main text specifies 2000 steps and a global batch size of 2048, with learning rates of \(1\times10^{-8}\) and \(1\times10^{-9}\), respectively. On SD 1.5, Diffusion-DPO and DMPO use \(\mu=0.9\), while DSPO uses \(\mu=0.2\); all three SDXL variants use \(\mu=0.6\). Other hyperparameters follow their corresponding base methods.

The larger FLUX.1-dev is a 12B rectified-flow DiT trained with an in-house dataset of 186K preference examples, \(1024\times1024\) resolution, global batch size 512, and learning rate \(1\times10^{-5}\). It uses \(\mu=0.99\), and the paper reports an effective coefficient of approximately \(\lambda_{\mathrm{safe}}\approx0.1\) at typical training steps. The value \(1-\mu=0.01\) is not the final scaling coefficient because the gradient-norm-to-inner-product ratio also contributes.

For tuning, the authors recommend jointly monitoring winner loss and margin: increase \(\mu\) if winner loss drifts upward or oscillates, and decrease it slightly if the winner is stable but preference improvement stalls. This is an empirical heuristic, not an additional automatic controller or a substitute for monitoring training stability.

Key Experimental Results

Main Results

The SD experiments use test prompts from Pick V2, HPS V2, and PartiPrompts. Automatic metrics include PickScore, HPS V2, LAION aesthetic score, CLIP, and ImageReward (IR). The evaluation description lists a guidance scale of 7.5 and 50 denoising steps. Ovis-U1 and FLUX.1-dev additionally use GenEval and DPG-Bench, while Ovis-U1 also undergoes image-editing evaluation.

The following excerpts main-text Table 1: SD 1.5 backbone, HPS V2 prompts, and higher is better for every metric. SFT is retained to avoid confusing improvement over a corresponding DPO baseline with superiority over every training approach.

Method	PickScore	HPS	Aesthetics	CLIP	IR
SFT	0.2168	0.2838	5.7851	0.3591	0.6619
Diffusion-DPO	0.2131	0.2743	5.6639	0.3552	0.1705
Diffusion-DPO + SDPO	0.2174	0.2827	5.8744	0.3600	0.6211
DSPO	0.2168	0.2837	5.8346	0.3598	0.6483
DSPO + SDPO	0.2172	0.2847	5.8474	0.3586	0.6578
DMPO	0.2131	0.2766	5.6538	0.3551	0.3171
DMPO + SDPO	0.2182	0.2848	5.8574	0.3612	0.7061

DMPO is an author-implemented baseline. Adding SDPO to Diffusion-DPO raises IR by 0.4506, but the result remains below SFT's 0.6619. DSPO's CLIP score decreases from 0.3598 to 0.3586. These are overall gains, not improvements in every individual metric.

The following excerpts main-text Table 4's FLUX.1-dev learning-rate comparison, retaining three representative metrics; the original table reports seven metrics.

Method	Learning rate	IR	GenEval	DPG-Bench
Base FLUX.1-dev	Not applicable	0.9916	0.66	85.75
Diffusion-DPO	\(1\times10^{-5}\)	-0.7788	0.23	58.97
Diffusion-DPO, low learning rate	\(1\times10^{-7}\)	0.9938	0.66	85.78
Diffusion-DPO + SDPO	\(1\times10^{-5}\)	1.1108	0.71	86.36

The comparison distinguishes preventing collapse from effective learning: the reduced learning rate restores approximately base-model performance, whereas SDPO improves performance at the higher rate. However, only these two ordinary-DPO learning rates are shown, so the experiment does not establish that other tuning strategies cannot achieve comparable results.

Ablation Study

Main-text Table 5 uses SD 1.5 and HPS V2 prompts. Efficiency is measured on one NVIDIA A100 with batch size 16, 128 gradient accumulation steps, and BF16. Time refers to a training step under that accumulation setting, not a single small-batch forward pass.

Config	PickScore	HPS	Step time / seconds	Peak GPU memory / GB
Diffusion-DPO	0.2131	0.2743	183	57.0
SDPO with fixed safety coefficient	0.2158	0.2803	183	57.0
SDPO with dynamic parameter-space coefficient	0.2176	0.2828	314	63.3
SDPO with dynamic output-space coefficient	0.2174	0.2827	184	57.1

The fixed coefficient already improves the baseline, indicating that reducing loser influence itself helps. The dynamic output-space coefficient further improves PickScore and HPS, supporting adaptation to current gradient geometry rather than permanently choosing a smaller weight. The output-space method adds 1 second and 0.1 GB over ordinary DPO; the parameter-space method adds approximately 72% time and 11% memory. The main text does not specify the fixed coefficient's value, so none is inferred here.

Key Findings

• In main-text Table 2, mean automatic win rates against each method's own baseline are 67.90% for Diffusion-DPO + SDPO, 68.90% for DMPO + SDPO, and 51.95% for DSPO + SDPO. These average multiple scoring-model win rates and are not new human blind-test results.
• In main-text Table 3, GenEval scores for Ovis-U1, its DPO variant, and its SDPO variant are 0.89, 0.88, and 0.91; GEdit-EN scores are 6.42, 6.43, and 6.60, respectively. The observed benefits are therefore not limited to SD-family preference scores.
• In Figure 3's SDPO trajectories, loser loss can also decrease while a larger winner improvement still widens the gap. Successful preference optimization does not require loser absolute error to rise continuously.
• Figure 4 supports sustained gains with longer training. The cache does not include the appendix mentioned in the main text; exact curve values, full SDXL results, and unseen sensitivity data are not reconstructed from the extracted figure text.

Highlights & Insights

• Monitor ranking progress separately from absolute capability. Relative improvement can conceal deterioration in both branches; tracking winner loss, loser loss, and margin reveals more than DPO loss alone.
• Preserve the forward objective while controlling backward influence. The stop-gradient construction retains the preference score and adjusts only the harmful contribution, making localized integration into existing DPO systems possible.
• Use an expensive implementation to validate a cheap approximation. The parameter-space variant provides a comparison, while the output-space method achieves nearby scores in Table 5. This establishes practical value more clearly than an overhead claim alone.

Limitations & Future Work

• Author-acknowledged theoretical boundary. First-order approximations weaken in high-curvature regions, and differences between output-space and parameter-space geometry can lead to inaccurate scaling. The authors propose second-order or trust-region safeguards as future directions.
• Note assessment: protecting a proxy does not protect every aspect of generation quality. Denoising error on current training samples is not a sufficient statistic for perceptual quality, prompt adherence, or diversity. Output-space protection is not a strict verification of each actual parameter update.
• Note assessment: evaluation and reproducibility gaps remain. Main-text results primarily use automatic scorers, FLUX relies on 186K in-house preference examples, and the fixed coefficient and some complete results require consulting the appendix. Current evidence does not replace independent human evaluation, more random seeds, or a complete hyperparameter search.
• Testable extensions. Infrequent parameter-space inner products could calibrate \(\mu\), or unusual increases in winner error could trigger a more conservative step-size check. These are suggestions, not features implemented in the paper.

Related Work & Insights

• vs Diffusion-DPO: The base method learns rankings from reference-calibrated relative errors. SDPO adds direction-dependent loser-gradient limits that directly target collateral damage to the winner.
• vs DSPO / DMPO: These are underlying preference objectives augmented by SDPO, not mutually exclusive alternatives. DSPO already increases winner weighting progressively, making its smaller additional gains consistent with the proposed mechanism.
• vs MaPO / DPOP: In this paper's analysis, MaPO uses fixed winner weighting without a reference model, while DPOP limits deterioration through a threshold-based protection rule. SDPO instead uses per-step gradient geometry to determine loser influence.
• Transferable insight: In other shared-parameter pairwise preference tasks, first examine whether repelling negative examples harms positive examples, then consider one-sided protection. Whether an equally inexpensive and reliable output-space proxy exists must be established for the new task.

Rating

• Novelty: 4/5. Makes winner preservation concrete through geometrically adaptive, one-sided gradient scaling, with a compact and targeted modification.
• Experimental Thoroughness: 4/5. Covers multiple objectives, UNet/DiT backbones, editing, and efficiency comparisons, but human evaluation and reproducibility of internal data remain limited.
• Writing Quality: 4/5. Connects failure modes, mechanisms, and ablations clearly; the scope of first-order conditions must be distinguished from the output-space approximation.
• Value: 4/5. Fits existing diffusion preference-training workflows, and the reported low overhead makes it practically worth testing.