Finite Difference Flow Optimization for RL Post-Training of Text-to-Image Models

Conference: ECCV 2026
Paper: ECCV paper page
Code: https://github.com/NVlabs/finite-difference-flow-optimization
Area: Image Generation
Keywords: finite differences, flow matching, RL post-training, paired trajectories, reward optimization

TL;DR

FDFO constructs a reward-weighted endpoint difference from two stochastic generation trajectories sharing initial noise and uses it to guide updates along both trajectories, raising the best OneIG-Bench prompt alignment score from Flow-GRPO's 79.56 to 84.25 on Stable Diffusion 3.5 Medium while substantially accelerating progress toward a specified reward level.

Background & Motivation

Text-to-image pretraining primarily learns a data distribution, whereas post-training seeks better prompt alignment, composition, or text rendering. These goals are usually expressed through proxy rewards such as PickScore, vision-language model assessments, or human preferences. Unlike pretraining, such rewards do not constrain every aspect of an image: clearer text need not improve overall appearance, and higher alignment scores need not preserve diversity. Irrelevant changes introduced by the optimization algorithm therefore become an additional liability, not merely noise in a training curve.

Flow-GRPO and DDPO formulate denoising as a multistep Markov decision process, treating each stochastic sampling step as an action and reinforcing the trajectory according to its final reward. However, a high-scoring final image does not imply that every random perturbation was useful; some steps may have moved in an unfavorable direction that other steps compensated for. The authors argue that this long-horizon credit assignment mixes substantial reward-neutral components into updates, slowing convergence and potentially causing style drift and grid-like artifacts during extended training.

The paper exploits a property specific to image generation: differences between images produced by nearby trajectories describe a concrete visual direction that can be compared. Instead of reinforcing all incidental random steps, it feeds this endpoint change back to intermediate states. Core Idea: treat the entire generation process as one action, multiply the difference between nearby images by their reward difference to obtain a local improvement direction, and make updates along both trajectories serve that direction rather than follow each step's random noise.

Method

Overall Architecture

The inputs are a pretrained flow-matching model, text prompts, and a scalar reward function; the output is an RL-post-trained generator. Each epoch generates paired trajectories with the current model and stores intermediate states, velocity predictions, and final images. Their endpoint rewards define a normalized difference signal, which guides clipped optimization of the LoRA parameters.

Treating the whole generation process as one action does not mean updating only the last step. All sampled timesteps on both trajectories receive supervision, but their reinforcement directions are no longer defined by their individual random displacements. Stochastic sampling discovers nearby candidate images, while finite differences determine the updates; these roles are separated.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Prompts and current model"] --> Pair["Paired Stochastic Sampling"]
    Pair --> Images["Two trajectories and endpoint images"]
    Images --> Difference["Normalized Endpoint Differences"]
    Reward["Scalar reward evaluation"] --> Difference
    Images --> Reward
    Difference --> Update["Controlled Direction-Aligned Updates"]
    Update --> Model["Update LoRA parameters"]
    Model -.->|Next training rollout| Pair

The loop depicts training. Inference requires neither the reward model nor paired generation for preference comparisons; deterministic sampling from noise remains available. Training-time stochasticity provides exploration and should not be counted as a mandatory deployment cost for every generated image.

Key Designs

1. Paired Stochastic Sampling: preserve the shared scene while exploring detail changes

Each prompt receives two trajectories starting from the same Gaussian noise, with different small random perturbations injected along the way. The resulting images usually retain similar overall layouts and content while differing in parts, positions, or details. Sharing the starting point does not make the trajectories identical; it concentrates the reward comparison on nearby changes. Requiring only two rollouts per prompt also allows more distinct prompts to be covered within a fixed generation budget.

The sampler does not simply reuse Flow-GRPO's Euler-Maruyama implementation. The authors identify mismatches between time conditioning and actual noise levels in conventional substeps, as well as relatively excessive noise injection at certain steps. Inspired by EDM, their sampler first follows the ODE direction to a lower noise level than the target, then adds fresh noise and corrects the scale to return to the target noise level. The default stochasticity strength is \(\gamma_i=0.0025\) at every step; \(\gamma_i=0\) gives deterministic sampling. This primarily re-randomizes unresolved detail instead of deliberately replacing the established coarse layout. The complete scale-correction expression in cached Algorithm 1 is corrupted, so this note retains the verifiable procedure without reconstructing an implementation formula.

2. Normalized Endpoint Differences: turn visual changes into a usable reward direction

Let the endpoints be \(x_T\) and \(\hat{x}_T\). The image difference \(\Delta x=\hat{x}_T-x_T\) does not identify which image is better; the reward difference \(\Delta R=R(\hat{x}_T)-R(x_T)\) supplies that orientation. When the second image is better, its direction is retained; when the first is better, the direction reverses. Equal rewards produce no preference update from that pair. Unlike methods requiring differentiable rewards, this operation reads only scalar assessments and can therefore also accept human pairwise preferences.

Scale requires care: a pair with a larger image difference often also has a larger reward difference, so directly multiplying them overweights that pair. The paper does not merely normalize the difference to unit length; it divides by its squared root-mean-square norm. Combining the normalization in Section 4.3 with the reward weighting gives the guiding direction:

\[ d=\frac{\Delta R\,\Delta x}{\lVert\Delta x\rVert_{\mathrm{RMS}}^2+10^{-6}}. \]

The squared RMS norm measures the mean squared coordinate difference, while \(10^{-6}\) prevents numerical instability for nearly identical pairs. This operation follows the local scaling intuition that reward differences approximately grow with image differences. It does not establish an unbiased gradient estimator for arbitrary reward functions. The no-normalization ablation in Figure 8 shows that this apparently simple adjustment matters for final performance.

3. Controlled Direction-Aligned Updates: propagate endpoint improvements along the generation trajectory

FDFO bends updates at every sampled timestep on both trajectories toward the improvement suggested by the endpoint difference, rather than toward the noise drawn at that timestep. Its intuition is that denoising reveals signal from coarse to fine: adding a meaningful visual change to an intermediate state will often preserve its broad direction while subsequent denoising supplies detail. This makes it possible to convey final preferences to intermediate steps without back-propagating through the entire sampling trajectory.

However, this tendency is a condition, not a universal guarantee. The paper expresses it using the Jacobian \(J_i\) of the remaining flow map from step \(i\) to the endpoint: after propagation through that map, a change along the difference direction should still align with the endpoint reward gradient. The analytical condition is:

\[ \nabla R(x_T)^\top J_i(x_i)[\Delta R\Delta x]\geq 0. \]

The authors explicitly acknowledge counterexamples and support their intuition with numerical similarities between diffusion flows and optimal transport maps. This should not be interpreted as saying that every diffusion flow is exactly an optimal transport map. Normalization applies a positive scale and therefore does not alter the directional condition. Because sampling time decreases from 1 to 0, gradient signs for velocity parameters must respect the integration convention; bending toward a higher-reward image cannot be mechanically translated into unconditionally adding \(d\) to the velocity.

To reuse expensive rollouts, several parameter updates follow each collection phase. Current velocities can then move away from those saved during sampling, so the data are no longer strictly drawn from the current policy. Simple Policy Optimization (SPO) clipping downweights updates whose velocities have drifted too far, limiting instability from stale data. A PPO replacement yields similar ablation results, indicating that the gains cannot be attributed simply to a different clipping rule.

A Worked Example

Consider the paper's example prompt describing a girl with pigtails holding a giant sunflower. An epoch generates two similarly composed images from shared noise and evaluates their rewards separately. Suppose the second image matches the prompt better: the positive reward difference orients the normalized endpoint difference from the first image toward the second. Reversing the preference automatically reverses the direction. This is an explanation of the procedure, not a claim about the actual scores of a particular pair.

That difference supervises all 40 saved sampling timesteps on both trajectories, rather than only states near the endpoint, and does not require identifying the particular step that drew the flower incorrectly. The 432 image pairs provide \(432\times2\times40=34560\) trajectory states, split into 4 training batches of 8640 states each. The next epoch generates fresh candidates with the updated model, repeatedly improving outputs in the neighborhood of its current behavior.

Loss & Training

Experiments use Stable Diffusion 3.5 Medium, train only added LoRA layers, generate images at \(512\times512\), and optimize with AdamW. The default run lasts up to 1000 epochs, with 432 trajectory pairs generated from randomly drawn prompts per epoch. Both training and inference use 40 sampling steps; a 10-step configuration is also evaluated for wall-clock efficiency. Gradients through the velocity network are accumulated within each training batch before a parameter update, without differentiating through the reward network.

The main comparison disables KL regularization and classifier-free guidance (CFG), and turns off weight exponential moving averaging because it harmed Flow-GRPO, to reduce confounding factors. Separate experiments assess CFG, while the appendix, absent from the supplied cache, discusses KL. Model and reward evaluation counts are matched per epoch across the two algorithms, but optimizer-step counts need not match: equal epoch budgets do not imply equal update budgets.

Rewards include PickScore, prompt alignment assessed by Qwen2.5-VL-7B-Instruct, and their combination. The VLM sees the image and is asked whether it matches the prompt; its next-token Yes/No logits define the alignment score. The combined reward is:

\[ R_{\mathrm{VLM}}=100\,\operatorname{sigmoid}(\ell_{\mathrm{Yes}}-\ell_{\mathrm{No}}),\qquad R_{\mathrm{combined}}=R_{\mathrm{PickScore}}+R_{\mathrm{VLM}}/10. \]

Pick-a-Pic provides training prompts, while OneIG-Bench and HPSv2 provide external checks not directly included in these reward definitions. The cache contains the complete main paper but not the full training pseudocode referenced in Appendix B, so this note does not invent an exact SPO loss or unreported hyperparameters.

Key Experimental Results

Main Results

The following values come from the table embedded in Figure 8, not a separately numbered Table in the paper. The metric is OneIG-Bench prompt alignment under combined-reward training; higher is better. Best means the highest score reached during training, not necessarily the score at the final epoch. B through E progressively replace or extend the configuration, with stochasticity strength tuned separately for each variant.

Configuration	Alignment at 200 epochs	Best alignment	Change
A: Flow-GRPO	71.06	79.56	Baseline
B: A + proposed stochastic sampler	76.74	82.94	Correct stochastic sampling
C: B + finite difference updates	82.14	83.07	Keep unique prompts per epoch unchanged
D: C + shared initial noise	81.11	83.95	Improve stability and best score
E: D + more prompts, full FDFO	82.13	84.25	Only two trajectories per prompt

The full method improves on the baseline by 11.07 points at 200 epochs and by 4.69 points in best score. The 200-epoch score decreases from C to D, so the results do not support claiming that every component improves every training stage monotonically.

Ablation Study

This table also comes from Figure 8 and lists alternatives to the full method E. The PPO configuration uses an interval stochasticity schedule; it is not a strictly single-variable comparison that changes only the clipping formula.

Configuration	Alignment at 200 epochs	Best alignment	Interpretation
E: Full FDFO	82.13	84.25	Reference
F: Make one trajectory deterministic	79.76	84.02	Slower early progress, similar best score
G: Replace SPO with PPO	81.99	83.91	Uses interval stochasticity
H: Do not normalize the difference vector	79.74	80.78	Best score drops by 3.47
I: Use the true reward gradient	81.47	83.73	Does not outperform finite differences
J: Further back-propagate through SDE steps	66.69	68.78	Best score drops by 15.47

Key Findings

• Figure 9 and Section 5.5 report approximately 19-fold measured acceleration with 40 steps and 5-fold with 10 steps to reach the selected combined-reward level. These factors concern a particular reward threshold and NVIDIA H200 GPU hours, not a universal speedup throughout training. The authors also provide ideal timing curves excluding implementation overhead.
• Figure 5 shows that the combined reward improves prompt alignment more reliably than PickScore alone and better accommodates human preference metrics than the VLM reward alone. However, every reward reduces diversity; at matched reward levels, diversity loss is approximately similar between the two algorithms.
• Section 5.6 uses 3200 online preference pairs supplied by one author over approximately 4 hours for 50 training epochs. This demonstrates a working loop without a reward model or offline preference dataset, not a multi-annotator or large-scale human evaluation.

Highlights & Insights

• Separate the exploration path from the update direction. Randomness discovers candidates, but parameter updates use meaningful differences between final images. For generative tasks with continuous outputs, this is more targeted than treating every random step as an action worth reinforcing.
• Finite differences can suit this training process better than direct differentiation. Neither I nor J surpasses FDFO, so more explicit differentiation through rewards or sampling does not automatically yield better model updates. This is evidence for the evaluated setting, not a universal argument against differentiable-reward methods.
• Efficiency also comes from prompt coverage. Two trajectories per prompt allow more prompts under a fixed rollout budget. The full method's gains therefore include its update direction, sampler, and data organization, rather than a single formula alone.

Limitations & Future Work

• Directional transfer depends on local flow geometry. An endpoint difference injected into an intermediate state may be rotated or distorted by the remaining flow. The authors acknowledge counterexamples, and the full Appendix C derivation is absent from the cache. Other backbones, modalities, and stronger perturbations require further validation.
• Experimental coverage is concentrated. Main experiments use one backbone, LoRA, and \(512\times512\) resolution, primarily against Flow-GRPO. They do not establish superiority across all models and post-training methods.
• Diversity and reward mismatch remain unresolved. Faster training does not imply better diversity at matched rewards, and the absence of observed grid artifacts does not eliminate reward hacking. The authors identify explicit diversity rewards and regularization as future directions.
• Human feedback is still a small prototype. A single annotator introduces personal aesthetic preferences, without support from large-scale blinded assessment. Future work could study inter-annotator agreement, feedback costs, and preferential allocation of costly feedback to informative image pairs.

Related Work & Insights

• vs Flow-GRPO / DDPO: These methods treat each stochastic denoising step as an action. FDFO uses paired endpoint differences to supply a consistent basis for updates throughout the trajectory while retaining online sampling and constrained optimization.
• vs DRaFT / ReFL / DRS: These approaches use differentiable rewards and differ in which steps receive supervision or participate in back-propagation. FDFO does not require differentiability and directly accepts human pairwise feedback, but depends on suitable nearby samples and directional-transfer assumptions.
• vs Diffusion DPO: Diffusion DPO relies on pre-collected preference pairs. FDFO generates pairs around the current policy online and immediately uses their feedback, keeping preferences relevant to the current model at the cost of continuing annotation or reward-evaluation expense.
• vs EDM sampling: EDM inspires changing the noise level before injecting fresh noise, which the paper adapts to flow-matching exploration. Row B of Figure 8 suggests checking sampler numerical errors before attributing all performance problems to the RL objective.

Rating

• Novelty: 4/5. Endpoint finite differences provide preference updates across the entire flow trajectory, departing from conventional stepwise MDP credit assignment.
• Experimental Thoroughness: 4/5. External metrics, sampler and update ablations, timing, and a human-feedback prototype are included, but backbone and annotator coverage are limited.
• Writing Quality: 4/5. Motivation and progressive ablations are clear; key implementation and theoretical details depend on appendices absent from the supplied cache.
• Value: 4/5. The method offers practical guidance for image post-training with expensive or non-differentiable rewards, while diversity and cross-model generalization remain open.