MPO: Single-Stream Policy Optimization for Efficient Text-to-Image Alignment¶

Conference: ECCV 2026
Paper: ECCV 2026 Poster
Full Cache: ../paper_cache/ECCV2026/eccv-3963.txt
Area: Image Generation
Keywords: text-to-image alignment, online reinforcement learning, Bayesian value tracking, variance reduction, single-stream optimization

TL;DR¶

MPO proposes a group-free, single-stream online reinforcement learning framework (one prompt, one trajectory, one policy update) that eliminates group synchronization latency and avoids intra-group advantage collapse via SDE exploration, Girsanov-drift-adaptive Bayesian value tracking, and global advantage normalization, achieving a 26x wall-clock speedup and a 5x sample-efficiency gain on FLUX.1 Dev.

Background & Motivation¶

Text-to-image generative models based on diffusion and flow-matching principles have demonstrated remarkable visual synthesis capabilities, yet precisely aligning them with nuanced human preferences—including strict compositional instruction following, object counting, spatial configurations, and visual text rendering—remains a formidable challenge. Because these evaluation criteria are often non-differentiable and evaluated via black-box reward models, online reinforcement learning (RL) has emerged as the prominent paradigm for generative post-training. State-of-the-art frameworks, predominantly represented by the GRPO series (such as Flow-GRPO and DanceGRPO), rely on a group-relative architecture: for each prompt, multiple image trajectories are sampled in parallel and compared against one another to estimate a local advantage baseline for variance reduction.

However, this concurrent group-sampling design incurs two severe fundamental bottlenecks. The first is synchronization overhead due to straggler effects: in distributed environments, an optimization step is blocked until the slowest generation trajectory in the group finishes. With multi-step numerical stochastic solvers on large foundation models, sampling latency fluctuations cause up to a 52% idle waiting overhead relative to the mean generation time. The second is intra-group advantage collapse: as policy optimization progresses and image outputs improve, multiple candidate images generated from the same prompt receive increasingly homogeneous rewards. The resulting collapse of intra-group reward variance flattens the advantage signal, drastically diminishing the learning utility of expensive parallel samples.

Existing alternatives either retreat to deterministic ODE velocity regression on the forward process—sacrificing the standard reverse-time stochastic exploration path—or demand differentiable reward signals and paired offline data. This motivates a direct and bold architectural simplification: rather than attempting to patch group-relative heuristics, can we decouple variance reduction from concurrent sampling altogether and transition to a streamlined pipeline of "one prompt $\to$ one trajectory $\to$ one update"? Core idea: eliminate prompt-level sample groups entirely by executing stochastic exploration along a single SDE trajectory, maintaining a persistent Bayesian value tracker modulated by a Girsanov-inspired policy-drift proxy to dynamically track historical baselines, and applying cross-batch global advantage normalization with an uncertainty-guided curriculum for stable single-stream alignment.

Method¶

Overall Architecture¶

MPO replaces group-wise generation with an end-to-end single-stream online RL workflow. At each training step, an active prompt $c$ is selected via an uncertainty-guided sampling curriculum with a fixed training latent $\mathbf{z}_0$; the model rolls out a single stochastic trajectory $\tau$ using an Euler-Maruyama SDE solver; the terminal image is decoded and scored by a black-box reward model $r$; the empirical policy drift is computed along the realized trajectory to dynamically adjust the process noise of a persistent Kalman-style Bayesian value tracker; and the raw advantage is normalized globally across recent batches and modulated by reward surprise to perform an immediate advantage-weighted velocity field update.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input prompt c with fixed initial latent z0"] --> B["SDE single-trajectory exploration<br/>Euler-Maruyama stochastic integration"]
    B --> C["Terminal decoding and black-box reward r"]
    C --> D["Trajectory policy drift estimation<br/>empirical velocity displacement proxy"]
    D --> E["Bayesian value tracker update<br/>Kalman filter with adaptive process noise Qk"]
    E --> F["Global advantage normalization and surprise weighting<br/>cross-batch EMA moment tracking"]
    F --> G["Advantage-weighted single-stream policy update<br/>weighted velocity field regression"]
    E -.->|posterior uncertainty feedback| A

Key Designs¶

1. SDE Single-Trajectory Exploration and Drift Estimation: Trajectory-level exploration and path-space drift proxy

Without concurrent latents within a group, exploratory diversity must be injected directly into the temporal generation dynamics. For continuous-time velocity field $\mathbf{v}_\theta(\mathbf{z}_t, c, t)$, MPO executes single-trajectory sampling via a stochastic differential equation (SDE) discretized using the Euler-Maruyama scheme: $$ \mathbf{z}{t+\Delta t} = \mathbf{z}_t + \mathbf{v}\theta(\mathbf{z}_t, c, t)\Delta t + g_t \sqrt{\Delta t} \boldsymbol{\varepsilon}_t, \quad \boldsymbol{\varepsilon}_t \sim \mathcal{N}(\mathbf{0}, \mathbf{I}) $$ where diffusion coefficient $g_t$ is linearly annealed from 0.1 to 0 across the 12 denoising steps. Furthermore, MPO fixes the initial latent $\mathbf{z}_0$ for each training prompt throughout post-training. This causal isolation ensures that observed reward fluctuations reflect actual policy shifts rather than noise resampling artifacts, greatly stabilizing baseline tracking.

Because the generative policy evolves continuously, historical reward baselines are inherently non-stationary. By Girsanov's theorem for diffusion processes with shared diffusion coefficients, the path-space KL divergence between successive policies $\theta_k$ and $\theta_{k-1}$ is bounded by integrated velocity differences. To avoid intractable and noisy full-path integrals, MPO computes a lightweight empirical drift proxy directly over the sampled trajectory: $$ \widehat{D}{\text{drift}}(\theta_k, \theta}) = \mathbb{E{t, \mathbf{z}_t \sim \tau} \left[ \left| \mathbf{v}}(\mathbf{zt, c, t) - \mathbf{v}_t, c, t) \right| \right] $$ This proxy quantitatively gauges the instantaneous distributional perturbation induced by parameter updates.}}(\mathbf{z

2. Drift-Adaptive Bayesian Value Tracker: Non-stationary baseline tracking without concurrent samples

To replace the missing local group baseline, MPO maintains a prompt-specific Gaussian posterior over expected reward: $\mathcal{V}(c) = \mathcal{N}(\mu_c, \sigma_c^2)$, where mean $\mu_c$ serves as the baseline and variance $\sigma_c^2$ quantifies estimation uncertainty. Upon receiving an observed reward $r$, the tracker updates its state via Kalman filtering equations: $$ K_k = \frac{\sigma_{c, k-1}^2}{\sigma_{c, k-1}^2 + \sigma_{\text{obs}}^2}, \quad \mu_{c, k} \leftarrow \mu_{c, k-1} + K_k (r - \mu_{c, k-1}) $$ $$ \sigma_{c, k}^2 \leftarrow (1 - K_k)\sigma_{c, k-1}^2 + Q_k, \quad Q_k = \alpha \widehat{D}{\text{drift}}(\theta_k, \theta) $$ where $\sigma_{\text{obs}}^2$ is the reward observation noise, and process noise $Q_k$ is adaptively scaled by the drift proxy via coefficient $\alpha$. When significant policy drift occurs, $Q_k$ injects uncertainty, increasing Kalman gain $K_k$ so the tracker quickly sheds stale historical observations; when updates are subtle, $Q_k$ remains minimal, preserving historical smoothing and shielding the baseline from stochastic noise.

3. Global Advantage Normalization and Uncertainty-Guided Curriculum: Mitigating local collapse and prioritizing informative prompts

The raw advantage for prompt $c$ is computed as $A = r - \mu_{c, k}$. Unlike group-relative schemes that normalize within local groups (which collapse when samples become homogeneous), MPO normalizes advantages globally across recent batches using exponential moving averages (EMA decay $\lambda=0.99$) for the mean $\mu_A$ and variance $\sigma_A^2$: $$ \tilde{A} = \frac{A - \mu_A}{\sqrt{\sigma_A^2} + \varepsilon_A} $$ This maintains consistent gradient scales and active learning signals across all training phases.

Simultaneously, the posterior uncertainty $\sigma_c$ governs dynamic prompt selection. Inspired by Upper Confidence Bound (UCB) principles, prompts are sampled according to: $$ p(c) \propto \sigma_c + \frac{\eta}{\sqrt{n_c + 1}} $$ where $n_c$ denotes the historical visitation count and $\eta$ controls coverage regularization. This curriculum prioritizes prompts undergoing rapid policy movement or displaying uncertain value estimates, while guaranteeing adequate coverage across the prompt repository.

4. Advantage-Weighted Policy Update with Surprise Prioritization: Efficient single-trajectory gradient step

To handle non-differentiable terminal rewards without incurring the high variance of REINFORCE estimators, MPO adopts advantage-weighted regression. It scales the update by a normalized reward surprise metric: $$ S_c = \frac{|r - \mu_{c, k}|}{\sigma_{c, k} + \varepsilon}, \quad w_c = 1 + \gamma S_c $$ The resulting MPO training objective is formulated as: $$ \mathcal{L}{\text{MPO}}(\theta) = \mathbb{E}t \sim \tau} \left[ \text{sg}(w_c \tilde{A}) \left| \mathbf{v}\theta(\mathbf{z}t, c, t) - \mathbf{u}\tau(\mathbf{z}_t, \mathbf{z}_0) \right|^2 \right] $$ where $\text{sg}(\cdot)$ represents the stop-gradient operator, and $\mathbf{u}_\tau$ denotes the empirical target vector field reconstructing the realized trajectory. Positive normalized advantages steer the velocity field toward high-reward trajectories, while negative advantages suppress subpar paths, completing the single-stream loop.

Loss & Training¶

The framework is optimized with AdamW using a constant learning rate of $5 \times 10^{-6}$ and a global batch size of 64 on NVIDIA H800 GPUs. The SDE sampler uses 12 Euler-Maruyama steps with diffusion coefficient $g_t$ linearly annealed from 0.1 to 0. Key hyper-parameters include EMA decay $\lambda = 0.99$, curriculum coefficient $\eta = 1.0$, and surprise weight $\gamma = 0.5$. Training prompts are decontaminated against evaluation benchmarks by filtering out any training prompt with a CLIP text embedding cosine similarity $\ge 0.8$ relative to test prompts.

Key Experimental Results¶

Main Results¶

On FLUX.1 Dev, MPO achieves substantial improvements across all compositional dimensions of GenEval, consistently outperforming autoregressive models, diffusion baselines, and group-relative online RL methods:

Table 1: Fine-grained GenEval Alignment Benchmarking

Model Architecture / Alignment Method	Overall	Single Obj.	Two Obj.	Counting	Colors	Position	Attr. Binding
Autoregressive Models
Emu3-Gen	0.54	0.98	0.71	0.34	0.81	0.17	0.21
Janus-Pro-7B	0.80	0.99	0.89	0.59	0.90	0.79	0.66
GPT-4o	0.84	0.99	0.92	0.85	0.92	0.75	0.61
Diffusion & Flow Matching Baselines
SD-XL	0.55	0.98	0.74	0.39	0.85	0.15	0.23
DALLE-3	0.67	0.96	0.87	0.47	0.83	0.43	0.45
SD3.5-M	0.63	0.98	0.78	0.50	0.81	0.24	0.52
FLUX.1 Dev (Base)	0.66	0.98	0.81	0.74	0.79	0.22	0.45
Flow-GRPO (SD3.5)	0.95	1.00	0.99	0.95	0.92	0.99	0.86
Online RL Alignment (FLUX.1 Dev)
FLUX.1 Dev + DanceGRPO	0.91	0.93	0.94	0.96	0.88	0.91	0.87
FLUX.1 Dev + MPO (Ours)	0.98	1.00	1.00	0.99	0.94	0.99	0.93

In terms of computational efficiency and training footprint, MPO demonstrates dramatic acceleration compared to DanceGRPO under matched hardware and model settings:

Table 4: Comprehensive Efficiency Comparison between MPO and DanceGRPO

Evaluation Metric	MPO (Ours)	DanceGRPO	Performance Factor / Note
Batch Size	64	64	Identical training setup
Wall-clock Time / Iteration (s)	0.85	14.12	16.6x faster per iteration
Peak GPU Memory (GB, per H800)	60	68	Lower memory overhead
Iterations to reach GenEval = 0.91	~940	~6,700	7.1x fewer optimization steps
Total GPU Hours to reach GenEval = 0.91	~9 hrs	~235 hrs	26x wall-clock speedup (235/9)
Sample Generation Consumption	1x (baseline)	~5x	5x sample-efficiency gain

Ablation Study¶

Ablation studies on FLUX.1 Dev confirm the essential role of each algorithmic design and demonstrate long-horizon stability:

Component Contributions and Extended Validations (from Section 4.4 and Table 5)

Experimental Configuration / Variant	Evaluation Setting	Key Metric	Empirical Finding
MPO Full Model	Standard GenEval	0.98	Optimal synergy across all modules
w/o Global Advantage Normalization	Raw single-sample advantage	0.93	Drops 0.05; unnormalized advantages destabilize gradients
w/o Drift-Adaptive Noise ($Q_k=0$)	Fixed process noise (no forgetting)	0.94	Drops 0.04; tracker lags behind policy evolution
Replace Bayesian Tracker with EMA	Simple moving average baseline	0.96	Drops 0.02; lacks uncertainty quantification
w/o Uncertainty-Guided Curriculum	Uniform random prompt sampling	0.96	Drops 0.02; slower convergence on hard prompts
Deterministic ODE Exploration	200 GPU-hour alignment run	Stagnates at 1.19	SDE achieves 1.32 ImageReward; ODE lacks diversity
Randomized Training Latent $z_0$	Resampled $z_0$ at each step	0.93 (19.3 hrs)	Doubled convergence time (9.1h $\to$ 19.3h); noise confounding
Extended Training (2k iterations)	Stress test vs DiffusionNFT	0.98 (NFT: 0.53)	DiffusionNFT collapses severely; MPO remains rock-solid
Blind Human Preference Evaluation	100 prompts, 10 human raters	58% win rate	DanceGRPO wins 15% (27% tie); significant visual preference

Key Findings¶

Crucial synergy between SDE and fixed training latents: While deterministic ODE sampling stagnates early on ImageReward (1.19 vs 1.32 for SDE), pairing SDE with a fixed initial noise $\mathbf{z}_0$ during training is vital. Randomizing $\mathbf{z}_0$ confounds value estimation with noise variance, causing training time to balloon from 9.1 to 19.3 hours and degrading GenEval performance to 0.93.
Global normalization is the primary variance anchor: Removing global advantage normalization causes the largest individual drop (0.98 $\to$ 0.93), proving that cross-batch moment estimation is indispensable when local group baselines are removed.
Hyperparameter robustness: Scanning the drift scaling parameter $\alpha$ across $[0.1, 5.0]$ yields consistently high GenEval scores, confirming that MPO does not depend on brittle hyperparameter tuning.

Highlights & Insights¶

Decoupling variance reduction from spatial sampling into temporal tracking: MPO's central insight is recognizing that group-relative baselines are merely an expensive spatial approximation of an expected value. By tracking this baseline over time with a Bayesian filter, MPO achieves $O(1)$ complexity, completely breaking the $O(G)$ computational bottleneck of GRPO.
Translating Girsanov's theorem into an efficient empirical proxy: Rather than computing intractable path-space integrals, MPO uses the expected velocity displacement on realized trajectories to calibrate process noise, bridging stochastic process theory with lightweight engineering execution.
Universal compatibility across architectures: MPO delivers consistent gains across modern flow-matching architectures (FLUX.1 Dev, SD3.5-M) and classic U-Net models (SD1.5, where GenEval improves from 0.43 to 0.55), establishing a general template for aligning complex generative priors.

Limitations & Future Work¶

Author-admitted limitations: Hard compositional failures still occur in rare, counter-intuitive spatial arrangements and hyper-fine-grained attribute binding. In addition, fixing $\mathbf{z}_0$ during training, while essential for causal isolation, might theoretically narrow the local exploration manifold for individual prompts during training.
Potential limitations: Unseen or out-of-domain prompts initially rely on default prior variance, which may lead to brief transient over-corrections during early Kalman updates. Furthermore, rapid single-stream optimization could potentially accelerate reward hacking if the proxy reward model has underlying flaws.
Future directions: Investigating dynamic noise scheduling to progressively relax the fixed initial latent constraint during later training stages, alongside multi-reward regularization to safeguard against reward hacking.

vs DanceGRPO / Flow-GRPO: GRPO relies on group sampling of size $G$ (typically $G=8$), suffering from up to 52% synchronization wait overhead and severe advantage collapse as samples converge. MPO eliminates group dependencies, cutting per-iteration time by over 16x and delivering a 26x end-to-end wall-clock speedup.
vs DiffusionNFT: DiffusionNFT avoids group sampling via forward-process velocity regression, but suffers catastrophic performance degradation during extended training (dropping from peak to 0.53 at 2k iterations); MPO preserves reverse-time SDE trajectories and maintains stable performance (0.98) throughout.
vs DPO / RWR: DPO and RWR operate primarily on offline preference datasets and static distributions; MPO operates as an online RL framework that dynamically explores improved generative trajectories alongside policy evolution.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Decouples group-relative sampling into a single-stream online RL formulation with Girsanov-drift-adaptive Bayesian tracking.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluations across FLUX, SD3.5, and SD1.5, including extensive ablations, long-horizon stability tests, and blind human evaluation.
Writing Quality: ⭐⭐⭐⭐⭐ Clear exposition, elegant theoretical grounding, and transparent reporting of empirical efficiency metrics.
Value: ⭐⭐⭐⭐⭐ Slashes the prohibitive computational cost of text-to-image RLHF by 26x, providing immense practical value for open-source generative alignment.