MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE¶

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/Tencent-Hunyuan/MixGRPO
Area: Image Generation / Alignment & RLHF
Keywords: flow matching, group relative policy optimization, mixed ODE-SDE, sliding window, preference alignment

TL;DR¶

MixGRPO concentrates stochastic exploration and GRPO updates in a short window that moves from coarse to fine denoising stages, using deterministic sampling elsewhere; on FLUX.1-dev with HPDv2 evaluation, it improves preference scores with fewer optimized steps, while Table 1 reports approximately 49% and 71% lower training iteration time for the base version and the fastest Flash* version, respectively.

Background & Motivation¶

Flow-matching models such as FLUX typically transform noise into images through a deterministic ordinary differential equation (ODE), but deterministic transitions do not directly support GRPO's stochastic policy probabilities. DanceGRPO and Flow-GRPO therefore introduce stochastic differential equation (SDE) sampling during denoising. Intermediate latents become states, subsequent latents become actions, and terminal image rewards compare a group of candidates for the same prompt. The difficulty is that every image must be fully generated before scoring, while evaluating new-to-old policy ratios requires additional model computation. Long denoising trajectories make online alignment expensive.

A straightforward shortcut is to retain stochastic sampling throughout the trajectory but update only a few timesteps. The paper tests this directly: reducing DanceGRPO's optimized steps from 14 to 4 lowers ImageReward from 1.436 to 1.335 in Table 1. The authors argue that randomness throughout the trajectory still affects the final reward, whereas the updated steps cover only part of that process. Moreover, early denoising primarily determines global structure, while late denoising refines details, so combining these stages in an update may disperse the learning signal. This latter explanation motivates the method; it is not a conclusion independently established by a dedicated gradient-conflict experiment.

MixGRPO does more than remove gradient computations: it aligns where exploration occurs with where optimization occurs. Only a short window remains stochastic, deterministic trajectories complete generation outside it, and the window moves toward lower-noise stages during training. This preserves comparable within-group exploration and makes the post-window segment, which does not require policy-density evaluation, available to a fast solver. Core Idea: jointly shorten the stochastic decision interval and the policy-update interval, then schedule both in coarse-to-fine denoising order instead of arbitrarily dropping updates from a globally stochastic trajectory.

Method¶

Overall Architecture¶

The inputs are text prompts, a current flow-matching generator, and reward models; the output is a preference-aligned generator. Each iteration first fixes the SDE window position and then uses the old policy to generate a group of images for each prompt. Group members share the initial noise, with variation introduced by stochastic sampling inside the window. Reward models score the completed images, GRPO updates only the transitions inside the window, and subsequent iterations resample trajectories while advancing the window at a predefined interval.

The three key designs are the Progressive Sliding Window, Mixed Sampling and Stable Exploration, and Post-Window ODE Acceleration, in that order. The base method already reduces the new policy's update computation; only the Flash variants additionally reduce the old policy's full-generation cost. Reward models supply training supervision, rather than being required components of the deployed generator.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Prompt + shared noise"] --> Window["Progressive Sliding Window"]
    Window --> Mixed["Mixed Sampling and Stable Exploration"]
    Mixed --> Tail["Post-Window ODE Acceleration"]
    Tail --> Images["Completed candidate images"]
    Images --> Reward["Training supervision: rewards<br/>Group advantages and window-only GRPO"]
    Reward -.->|Update model and advance window next iteration| Window
    Reward -.-> Model["Aligned generator<br/>No reward model needed at deployment"]

Post-window acceleration in the diagram is enabled only in the Flash variants; the base version retains standard ODE sampling. The diagram describes training rollouts and their feedback loop. It does not imply that three neural networks are added to the generator, nor that training acceleration directly translates into faster deployment inference.

Key Designs¶

1. Progressive Sliding Window: learn global layout before local refinement

The window is controlled by its size, shift interval, and stride, denoted by \(w\), \(\tau\), and \(s\). The default window contains 4 steps and advances by 1 denoising step every 25 training iterations, stopping at the final valid position. Progression changes the optimization region across training iterations; it does not repeatedly move the window while generating a single image. The window remains fixed within a rollout, so the exploration compared through its final rewards occurs within the same denoising stage.

Early high-noise regions can alter overall composition and offer a larger exploration space; late low-noise regions are better suited to textures and details. Learning high-impact structural decisions before narrowing exploration motivates this schedule over random window placement. The authors also test a window frozen at the initial position. Its competitive performance indicates that early steps deserve priority, but does not establish that later steps are irrelevant. The progressive, constant-interval strategy in Table 4 gives a stronger overall result, although exponentially decaying intervals score higher on some individual metrics.

2. Mixed Sampling and Stable Exploration: confine randomness and trainable transitions to the same interval

Once the window is fixed, the old policy follows an ODE to its start, injects noise step by step inside it, and follows another ODE segment to the completed image. Because group members share the initial noise, the deterministic prefix introduces no additional candidate differences. Different random perturbations inside the window produce the candidates to compare. Every image still traverses all denoising stages: optimizing only 4 steps does not mean generating in only 4 steps, or evaluating image rewards directly on intermediate latents.

SDE steps have computable Gaussian transition densities, allowing GRPO to compare the probability of the same sampled transition under the new and old policies. ODE steps are deterministic transitions and are explicitly excluded from ratio and KL evaluation. The theoretical basis is that corresponding ODEs and SDEs can share time-marginal distributions in continuous time, permitting interval-wise switching under suitable regularity conditions on the drift and score fields. Finite-step discretization and model errors still limit this ideal equivalence. The claim concerns distributions under appropriate assumptions, not identical images from both samplers or necessarily identical RL rewards.

To reduce high-frequency artifacts introduced by noise inside the window, the final version adopts the existing Coefficients-Preserving Sampling (CPS) method. CPS better preserves the flow-matching interpolation structure while injecting randomness, helping keep sampling artifacts from distorting reward feedback. Section 3.4 states that the implementation retains the same per-step Gaussian covariance scale for ratio and KL evaluation, leaving that parameterization unchanged. CPS is an adopted sampling improvement, not an algorithm introduced by MixGRPO. The precise drift and CPS derivations are incomplete or damaged in the cached PDF extraction, so this note does not guess implementation-ready formulas from them.

3. Post-Window ODE Acceleration: reduce rollout computation in a trajectory suffix that needs no policy density

The base method shortens the interval optimized by the new policy, but the old policy must still complete the trajectory to produce an image for reward evaluation. Its rollout function evaluations therefore remain at 25. The Flash variants additionally exploit the deterministic suffix after the window, applying DPM-Solver++ adapted to flow matching. The default second-order Midpoint solver completes this segment with fewer function evaluations. No stochastic transition density for GRPO is needed there, which is precisely the flexibility created by the mixed formulation.

Why accelerate only the suffix rather than every ODE segment? The paper reports that numerical errors from prefix acceleration are amplified by subsequent stochastic sampling, damaging image quality and reward reliability. Suffix acceleration avoids that issue, although excessively reducing function evaluations still harms scores. Standard MixGRPO-Flash uses a progressive window, leaving a shorter suffix as the window advances. MixGRPO-Flash* freezes the window at the beginning and therefore retains a long accelerable suffix. The latter uses only 8 old-policy evaluations in Table 1 and is the version associated with approximately 71% lower iteration time, but it also forgoes the full progressive curriculum.

A Worked Example¶

Consider a prompt describing a vase to the right of a horse, with the main experiment's 25 denoising steps and a 4-step window. The following illustration uses zero-based step indices to explain scheduling; it is not a separately reported test case from the paper.

Initially, place the window at steps 0 through 3. Candidates in each group start from the same noise, while independent perturbations in these 4 steps can alter object positions and overall composition.
Complete the images with ODE sampling at steps 4 through 24. The base method computes no policy updates for this segment; Flash* can use a high-order solver to reduce suffix evaluations.
Score the completed images and compare their prompt agreement and preferences within the group. Better candidates receive positive advantages, encouraging their window transitions rather than converting the phrase "to the right" into manually supplied stepwise rewards.
For progressive variants, move the window to steps 1 through 4 after 25 training iterations and continue exploring and updating. Frozen variants retain steps 0 through 3.

The deterministic suffix turns different window decisions into images that can be scored, and the rewards then select better behavior inside the window. Updates do not require end-to-end back-propagation through the reward model or the whole suffix, but complete rollouts remain necessary to determine whether those decisions ultimately help.

Loss & Training¶

For each prompt, generate a group of candidates, subtract the group's mean terminal reward from each candidate's reward, and divide by the group standard deviation to obtain relative advantages. Multi-reward settings standardize rewards separately before combining them. This measures how much better a candidate is than others for the same prompt, rather than comparing absolute difficulty across prompts, and requires no additional value network to predict a baseline.

The clipped GRPO surrogate averages only over the window and includes a KL constraint. The following simplified notation summarizes the objective structure of Equation (9) in Section 3.1 for explanation; it is not a character-for-character reconstruction of the damaged PDF formula:

\[ J = \operatorname{mean}_{i,\,t\in W}\!\left[\min\!\left(r_{i,t}A_i,\operatorname{clip}(r_{i,t},1-\epsilon,1+\epsilon)A_i\right)\right]-\beta J_{\mathrm{KL}}. \]

Here, \(W\) is the current window, \(A_i\) is a candidate's group-relative advantage, \(r_{i,t}\) is the new-to-old policy probability ratio for the same window transition, \(\epsilon\) is the clipping threshold, and \(\beta\) controls the KL penalty. Positive advantages encourage higher relative transition probabilities, while negative advantages suppress them; clipping limits aggressive updates. Equation (10) defines this KL constraint between the current and old policies, so the old policy should not be casually interpreted as a permanently frozen pretrained model.

The main experiments use FLUX.1-dev and the HPDv2 collection of 103,700 training prompts. The authors report strong alignment after using 9,600 prompts. The test set contains 400 prompts across Animation, Concept Art, Painting, and Photo styles. The usual window configuration is \(w=4\), \(\tau=25\), and \(s=1\). The main text also states that the SD3.5-M extension uses LoRA fine-tuning, but Appendix 11 is absent from the cache; the GPU model, batch size, group size, and learning rate for the main experiments cannot be filled in from this source.

Key Experimental Results¶

Main Results¶

The following results are selected from Table 1, using FLUX.1-dev and the 400-prompt HPDv2 test set. HPS-v2.1, Pick Score, ImageReward, and Unified Reward are preference-model scores, all higher-is-better, not human win rates or percentages. They emphasize somewhat different aspects of aesthetics, image-text agreement, and semantic consistency. Old-policy NFE counts rollout function evaluations, whereas new-policy NFE counts forward evaluations required for policy ratios. Their sum should not be treated as total training FLOPs.

Method	Old-policy NFE	New-policy NFE	Iteration time / seconds	HPS-v2.1	Pick Score	ImageReward	Unified Reward
FLUX.1-dev	Not applicable	Not applicable	Not applicable	0.313	0.227	1.088	3.370
DanceGRPO, official configuration	25	14	291.284	0.356	0.233	1.436	3.397
DanceGRPO, 4 randomly optimized steps	25	4	149.978	0.334	0.225	1.335	3.374
MixGRPO	25	4	149.326	0.369	0.238	1.645	3.419
MixGRPO-Flash, progressive	16 (average)	4	112.372	0.358	0.236	1.528	3.407
MixGRPO-Flash*, frozen	8	4	83.278	0.357	0.232	1.624	3.402

Relative to official DanceGRPO, Table 1 implies approximately 49%, 61%, and 71% lower iteration time for the base, Flash, and Flash* versions. These are timings within the paper's experimental comparison, not universal speed guarantees across hardware, nor measured total time to convergence at a common target score. The cached main text does not identify the specific GPU configuration used for these timings.

The source contains differences that should remain explicit: Section 4.2 gives 150.839 seconds for the base version, whereas Table 1 gives 149.326 seconds. The introduction reports ImageReward of 1.629, compared with 1.645 in Table 1; Table 10 assigns these two scores to the SDE and CPS variants, respectively. The sampler distinction offers an explanation for the latter difference, but does not justify silently rewriting every section to use one number.

Ablation Study¶

The following rows come from Tables 4 and 10 in the FLUX experimental setting. Their sources remain explicit to avoid presenting separate experiments as identical configurations. Table 4 compares window movement, while Table 10 compares stochastic samplers under the same training setup. Hardware and omitted optimization hyperparameters still require verification from the appendix.

Source and configuration	HPS-v2.1	Pick Score	ImageReward	Unified Reward	Note
Table 4: frozen window	0.354	0.234	1.580	3.403	Fixed at early steps
Table 4: random window, constant interval	0.365	0.237	1.513	3.388	No coarse-to-fine curriculum
Table 4: progressive window, constant interval	0.367	0.237	1.629	3.418	Default schedule
Table 4: progressive window, exponentially decaying interval	0.360	0.239	1.632	3.416	Higher on some metrics, not the default
Table 10: MixGRPO-SDE	0.367	0.237	1.629	3.418	Standard stochastic sampling
Table 10: MixGRPO-CPS	0.369	0.238	1.645	3.419	More stable stochastic sampling

Key Findings¶

Table 1's matched-update comparison best isolates the mechanism: with 4 optimized steps each, MixGRPO scores 1.645 on ImageReward versus 1.335 for DanceGRPO with randomly reduced updates, at similar iteration time.
In Table 4, progressive movement at constant intervals increases ImageReward from the random window's 1.513 to 1.629, supporting an ordered curriculum. The default schedule does not dominate every metric.
Under joint HPS-v2.1 and CLIP Score training in Table 2, the held-out ImageReward score is 1.416 versus DanceGRPO's 1.314. This supports some cross-reward generalization, not the elimination of all reward hacking.
Flash* does not outperform official DanceGRPO on every metric: its Pick Score is 0.232 versus 0.233 in Table 1. Nor does MixGRPO dominate Table 3 on SD3.5-M: its ImageReward of 1.485 is below Online Flow-DPO's 1.500.

Highlights & Insights¶

Effective computation savings change exploration itself, rather than merely removing gradient updates. Aligning the stochastic and optimized intervals makes the matched-budget comparison particularly informative.
A deterministic suffix that needs no policy density can independently use a high-order solver. This suggests jointly designing samplers and training objectives for generative RL instead of fixing the sampler and modifying only the loss.
Window scheduling allocates both computation and a global-to-local curriculum. Transferring the idea requires checking that the target generation process has comparable stages, rather than assuming this empirical pattern holds for every continuous model.

Limitations & Future Work¶

The authors explicitly acknowledge that limited reward models still permit reward hacking, particularly late in training. MixGRPO aims to use existing rewards faster and more effectively, not to solve their fundamental mismatch with genuine preferences.
The authors also identify predefined window size, shift interval, and stride as a limitation. Adapting window movement to reward convergence or gradient variance remains a future direction.
Evidence boundary of this note: the cache contains the main paper and references, but not the cited appendices. The main text describes a 512-GPU experiment with 80B HunyuanImage-3.0 and an extension to HunyuanVideo-1.5, but does not provide enough evidence to verify their full results, human blind tests, or training details.
This note's assessment: consistent gains across reward models are more persuasive than a single reward increase, but cannot replace independent human evaluation. The main tables also lack sufficient repeated-run variability information to establish the significance of small differences.

vs DanceGRPO / Flow-GRPO: These methods provide the stochastic policy-optimization foundation for flow matching. MixGRPO mainly restricts and schedules the stochastic interval rather than introducing a new GRPO clipping objective.
vs Flow-DPO / DiffusionNFT: These approaches use different alignment objectives or optimization processes. Table 3 supports strong overall results with fewer optimized steps on SD3.5-M, but not the claim that MixGRPO maximizes every reward metric.
vs DPM-Solver++ / CPS: The former accelerates deterministic integration; the latter stabilizes stochastic sampling. MixGRPO assigns these existing tools to trajectory segments with different responsibilities, rather than inventing either the underlying solver or CPS.

Rating¶

Novelty: 4/5. The contribution lies in jointly designing stochastic exploration, update intervals, and curriculum scheduling.
Experimental Thoroughness: 4/5. Matched-budget comparisons, multiple ablations, and cross-reward evaluation are provided, but some details cannot be verified without the missing appendices.
Writing Quality: 3/5. The main argument is clear, but numerical conventions across the text, tables, and variants need careful distinction.
Value: 4/5. Practical for expensive online image-generation alignment, with benefits still dependent on reward quality and sampling error.