RewardFlow: Generate Images by Optimizing What You Reward

Conference: CVPR 2026 arXiv: 2604.08536 Code: https://huggingface.co/onkarsus13/RewardFlow Area: Image Generation / Editing Keywords: Reward-guided generation, diffusion models, Langevin dynamics, image editing, compositional generation

TL;DR

RewardFlow proposes an inversion-free inference-time framework that fuses multiple differentiable reward signals—including semantic alignment, perceptual fidelity, local grounding, object consistency, and human preference—via multi-reward Langevin dynamics, achieving state-of-the-art editing fidelity and compositional alignment on image editing and compositional generation benchmarks.

Background & Motivation

Background: Diffusion models and flow-matching models have achieved remarkable success in image generation, yet controllable editing and compositional generation remain challenging. Existing approaches typically rely on text guidance or model fine-tuning to realize specific editing effects.

Limitations of Prior Work: Current image editing methods suffer from three main issues: (1) inversion-based methods incur high computational overhead and are prone to error accumulation; (2) a single reward signal cannot simultaneously account for semantic correctness, visual fidelity, and local precision; (3) semantic leakage—where editing effects inadvertently propagate beyond the target region—remains a persistent problem.

Key Challenge: The core tension lies in coordinating multiple heterogeneous reward objectives (semantic alignment, perceptual quality, regional precision, human preference, etc.). Naïve weighted summation tends to suppress certain objectives, and different editing intents require different reward weight configurations.

Goal: To design a unified inference-time framework that integrates multiple complementary differentiable reward signals into the sampling process of diffusion/flow-matching models without any fine-tuning or inversion.

Key Insight: The authors ground the framework in Langevin dynamics, theorizing the reward-guided sampling process as an effective discretization of a Langevin SDE targeting a prompt-tilted density, thereby providing theoretical guarantees for stable convergence.

Core Idea: Unify multiple complementary differentiable rewards (CLIP semantics, LPIPS perception, SAM2 localization, VQA attribute-level signals, and human preference) through Langevin dynamics, and design a prompt-aware adaptive policy to dynamically modulate the weight of each reward.

Method

Overall Architecture

The overall pipeline of RewardFlow is as follows: given a pretrained diffusion/flow-matching model and an editing instruction, the method guides the denoising trajectory via multi-reward Langevin dynamics during sampling, with no inversion of the source image required. Specifically, at each sampling step, gradients of multiple differentiable rewards are computed, fused via an adaptive policy, and used to correct the sampling direction. A clean-latent KL regularizer anchors the sampling trajectory to the original latent, preventing excessive editing drift.

Key Designs

1. Multi-Reward Langevin Dynamics:

   • Function: Guides the sampling process of diffusion models at inference time using multiple complementary reward signals.
   • Mechanism: Fuses the gradients of multiple differentiable reward functions into a unified guidance signal. Five reward categories are included—semantic alignment (text-image matching via CLIP, etc.), perceptual fidelity (ensuring post-edit image quality), local grounding (region constraints guided by SAM2), object consistency, and human preference (e.g., ImageReward). At each sampling step, the gradients of all rewards are combined via weighted summation to correct the denoising direction.
   • Design Motivation: A single reward cannot cover all dimensions of editing. For instance, relying solely on a semantic reward may sacrifice visual quality, while a perceptual-only reward may yield semantically inaccurate results. Multi-reward fusion achieves a balanced trade-off across multiple objectives.

2. Differentiable VQA-based Reward:

   • Function: Provides fine-grained, attribute-level semantic supervision.
   • Mechanism: Decomposes the editing instruction into a set of attribute-related question-answer pairs (e.g., "Is the object red?" or "Is the background at night?") and computes the accuracy of each pair using a differentiable VQA model as the reward signal. This enables the reward to supervise specific attribute changes precisely, beyond overall semantic matching.
   • Design Motivation: Global semantic models such as CLIP have limited discriminative capacity for fine-grained attributes. VQA rewards provide instruction-level precise feedback through language-visual reasoning.

3. Prompt-Aware Adaptive Policy:

   • Function: Dynamically modulates the weight and step size of each reward.
   • Mechanism: Extracts semantic primitives from the editing instruction (e.g., editing type: color change, style transfer, object addition) to infer editing intent (local vs. global), then dynamically modulates the weight and step size of each reward throughout the sampling process. For example, local color editing increases the weight of the SAM localization reward, while global style transfer emphasizes the perceptual reward.
   • Design Motivation: Different editing tasks have varying dependencies on individual reward signals; fixed weight configurations cannot accommodate diverse editing requirements.

Loss & Training

RewardFlow is a purely inference-time framework that requires no additional training. Its core "loss" is manifested as reward-gradient guidance during sampling:

• Multi-reward fusion signal: \(\nabla_x \sum_i w_i(t) \cdot R_i(x_t)\), where \(w_i(t)\) is the adaptive weight of the \(i\)-th reward at timestep \(t\).
• Clean-latent KL regularization: Anchors the sampling trajectory to the original latent, preventing excessive drift induced by reward guidance. This acts as a soft constraint balancing reward maximization against fidelity to the original content.
• Theoretical guarantee: The authors demonstrate that the update process corresponds to an effective discretization of a Langevin SDE whose target distribution is the prompt-tilted density.

Key Experimental Results

Main Results

Benchmark	Metric	RewardFlow	Prev. SOTA	Gain
EMU-Edit	Edit Fidelity	SOTA	—	Significant improvement
T2I-CompBench	Compositional Alignment	SOTA	—	Significant improvement
MagicBrush	CLIP-I / DINO Score	Best	InstructPix2Pix et al.	Top on multiple metrics
InstructPix2Pix Bench	Edit Quality	Best	SDEdit, P2P	Surpasses all baselines

Ablation Study

Configuration	Edit Fidelity	Note
Full RewardFlow	Best	All rewards + adaptive policy
w/o VQA Reward	Notable drop	Missing fine-grained attribute supervision
w/o SAM Localization	Increased semantic leakage	Weaker region control
w/o Adaptive Policy	Performance degraded	Cannot adapt to diverse editing intents
w/o KL Regularizer	Excessive editing drift	Loss of original content anchoring

Key Findings

• The VQA reward contributes most to fine-grained editing (color and texture changes); removing it leads to a significant drop in attribute-level accuracy.
• The SAM2 localization reward effectively prevents semantic leakage, and is indispensable particularly in local editing scenarios.
• The adaptive policy automatically adjusts weight allocation based on editing intent, eliminating the need for manual tuning.
• The inversion-free design substantially reduces computational overhead while maintaining generation quality.

Highlights & Insights

• Theoretical elegance of multi-reward Langevin dynamics: Unifying multi-objective optimization as a discretization of a Langevin SDE is both theoretically grounded and practically efficient. The intuition of "optimizing what you reward during sampling" is highly general and transferable.
• VQA as a fine-grained reward—a novel design: Using a VQA model to provide attribute-level feedback is a clever design that can be transferred to any generative task requiring fine-grained semantic control.
• Training-free inference-time method: Avoids the cost of training task-specific models for each editing type; diverse editing behaviors are achieved simply by composing different rewards.

Limitations & Future Work

• Computing gradients of multiple reward functions increases inference latency, which may be a bottleneck for real-time applications.
• The quality of the reward functions determines the upper bound of editing performance—if a reward model is inaccurate in specific scenarios, overall performance will be affected.
• The adaptive policy currently relies on heuristic extraction of semantic primitives; learnable intent inference may yield better results.
• Robustness in highly complex compositional editing scenarios (e.g., simultaneously modifying different attributes of multiple objects) remains to be validated.

Related Work & Insights

• vs. SDEdit / DDIM Inversion: These methods require inverting the source image into the noise space before editing, incurring high computational cost and error accumulation. RewardFlow requires no inversion whatsoever, guiding the sampling process directly.
• vs. InstructPix2Pix: InstructPix2Pix requires training a dedicated editing model, whereas RewardFlow is a purely inference-time method that does not modify model weights.
• vs. single-reward guidance methods (e.g., DPS): Methods such as DPS typically employ only a single reward for guidance. RewardFlow's multi-reward fusion combined with an adaptive weighting strategy is considerably more flexible.

Rating

• Novelty: ⭐⭐⭐⭐ The multi-reward Langevin framework offers theoretical contributions, though reward-guided generation as a broad direction has prior precedent.
• Experimental Thoroughness: ⭐⭐⭐⭐ Validated across multiple benchmarks with complete ablation studies.
• Writing Quality: ⭐⭐⭐⭐ Theory and experiments are tightly integrated; the structure is clear.
• Value: ⭐⭐⭐⭐ The inference-time multi-reward guidance paradigm is highly generalizable and has strong practical value.

Related Papers

• [CVPR 2026] Low-Resolution Editing is All You Need for High-Resolution Editing
• [CVPR 2026] Enhancing Spatial Understanding in Image Generation via Reward Modeling
• [CVPR 2026] SimLBR: Learning to Detect Fake Images by Learning to Detect Real Images
• [CVPR 2026] Pixel Motion Diffusion Is What We Need for Robot Control
• [NeurIPS 2025] Understand Before You Generate: Self-Guided Training for Autoregressive Image Generation