SRUM: Fine-Grained Self-Rewarding for Unified Multimodal Models¶

Conference: ECCV2026
arXiv: 2510.12784
Code: To be confirmed
Area: Image Generation
Keywords: Unified Multimodal Models, Self-Rewarding, Text-to-Image Generation, Global-Local Reward, Post-Training Alignment

TL;DR¶

SRUM proposes a self-rewarding post-training framework where the comprehension module of a Unified Multimodal Model (UMM) acts as an internal evaluator, providing fine-grained corrective signals to the generation module through a dual global-local reward system, thereby significantly improving the quality of complex text-to-image generation.

Background & Motivation¶

Unified Multimodal Models (UMMs) aim to integrate visual comprehension and visual generation capabilities into a single framework. In recent years, archetypal architectures such as Janus, Show-O, Bagel, and BLIP3o have emerged. However, a paradox continues to plague these models: a severe asymmetry exists between their comprehension and generation capabilities. The same model can accurately determine whether a text description matches an image, yet often fails to generate a faithful and credible image based on that text. This gap of "being able to judge but unable to execute" is particularly pronounced in scenarios requiring fine-grained reasoning, such as complex spatial relationships, precise attribute binding, and numerical reasoning. Fundamentally, this stems from the capability gap between the UMM's internal comprehension and generation modules.

There are three main categories of existing approaches to narrowing this gap: first, introducing external reward models or human preference data for RLHF/DPO, which is costly due to high preference data collection costs and prone to bias in the external reward models; second, adopting chain-of-thought (CoT) or test-time verification to perform multi-turn corrections during the inference phase, which does not fundamentally improve the model's native generation capability; third, manually designing rule-level rewards for different tasks, which lacks generalizability. The common bottleneck of these methods is the assumption that the guidance for generation must come from outside the model, ignoring that the "stronger comprehension module" already existing within the UMM is itself a natural evaluator.

The key insight of this paper is: since the comprehension capability of a UMM already surpasses its generation capability, why not let the comprehension module directly "teach" the generation module? Core Idea: Freeze the UMM's own comprehension module to serve as an internal evaluator. It scores self-generated image candidates to produce a set of global-local dual reward signals, which are then backpropagated into the parameters of the generation module using a reward-weighted training objective. This forms a self-improvement closed loop without requiring external data or reward models.

Method¶

Overall Architecture¶

SRUM is a two-stage offline post-training framework. The first stage is Self-Rewarding Data Generation: the model utilizes its own generation capabilities (paired with a CoT thinking mode) to produce candidate images. Concurrently, it leverages an external segmentation model (such as SAM, for Bagel) or native grounding capabilities (for BLIP3o) to generate corresponding bounding boxes for objects in the image. The UMM's comprehension module then validates these boxes and performs region matching. The second stage is Reward-Weighted Training: the comprehension module is frozen to perform dual local & global evaluations on each candidate image to generate a dence reward map. Then, this reward signal is used to weight the training of the generation module (velocity prediction in flow matching), enabling the generator to learn to retain features in high-reward regions and adjust outputs in low-reward regions. Throughout the process, the comprehension module does not participate in gradient updates, and the reward data is cached once prior to training, making it an offline pipeline with high computational efficiency.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Text Prompt"] --> B["Generate Candidate Images<br/>CoT + Bounding Boxes"]
    B --> C["Local Reward<br/>Object-level Fidelity Score"]
    B --> D["Global Reward<br/>Overall Composition Alignment Score"]
    C --> E["Dense Reward Map<br/>α·R"]
    D --> E
    E --> F["Reward-Weighted Loss<br/>L_r + λ_c · L_ref"]
    F --> G["Update Generator Parameters<br/>(Comprehension Module Frozen)"]
    G --> H["Improved Generation Capability"]

Key Designs¶

1. Self-Data Generation Pipeline: Models creating training samples for themselves

The first design of SRUM is to completely internalize the production process of training data. Starting from existing text instructions like T2I-CompBench, the model first uses its "think mode" (a CoT reasoning mode) to generate semantically rich image candidates. Then, for models like Bagel that do not possess fine-grained native grounding capabilities, an external SAM segmentation model is used to propose preliminary object bounding box candidates. For models like BLIP3o with native grounding capabilities, their own localization module is directly utilized. The key step is: regardless of the source of the bounding boxes, the frozen UMM comprehension module must personally verify if these boxes are correct and if they match the relevant objects in the prompt, assigning a semantic reward score. Consequently, the external component (SAM) is merely a localization aid rather than a reward model, and all semantic judgments during the scoring process originate from the UMM itself, ensuring self-containment. This approach completely eliminates dependencies on external image datasets and human annotations.

2. Global-Local Dual Reward System: Multi-scale self-judgment

This is the core mechanism of SRUM. It designs two complementary evaluation dimensions to overcome the lack of information in a single overall score. Local Judgment independently scores the region within each object bounding box, with the range strictly restricted to \([-1.0, 1.0]\). High penalties (\(-0.9\) to \(-0.5\)) are imposed via non-linear mapping for severe distortion (such as missing objects or hallucinations), mimicking human visual sensitivity to conspicuous defects. More importantly, it forces the output of a "Reason" text block—in CoT format—explaining the scoring rationale before each score. This not only makes the scoring process more reliable but also indirectly forces the comprehension module to make more meticulous judgments. Global Judgment focuses on the overall compositional rationality of the entire image and whether the spatial layout aligns with the prompt's intent. For prompts without fine-grained composition requirements (e.g., "a photo of a tree"), the global reward automatically falls into a neutral interval (\(-0.4\) to \(0.4\)) to avoid unfair penalties.

The two evaluation results are ultimately fused into a dense reward map: the local rewards form a region-level heatmap \(R \in [-1, 1]\), and the global reward is scaled to \([0, 1]\) as a scalar multiplier \(\alpha\) (to prevent two negative numbers from multiplying to produce a false positive signal). The product \(\alpha \cdot R\) serves as the training weight for each spatial position. This design ensures that the model is neither penalized globally for local defects nor pardoned for local errors due to a good overall composition.

3. Reward-Weighted Training and Reference Constraint: Precise optimization to prevent drift

The objective of the training stage is to translate evaluation signals into parameter updates for the generation module. The loss function of SRUM consists of two terms. The first term is the reward-weighted loss \(\mathcal{L}_r\), which directly performs weighted MSE on the velocity prediction \(\mathbf{v}_\theta\) in the flow matching framework:

\[\mathcal{L}_r = \mathbb{E}\left[\alpha \cdot R \odot \left(\mathbf{v}_\theta - (\epsilon - \mathbf{x}_0^{gt})\right)^2\right]\]

where \(\odot\) denotes element-wise multiplication. When \(\alpha \cdot R > 0\) (indicating good generation quality in that region), the loss encourages the model to maintain its current prediction. When \(\alpha \cdot R < 0\) (bad quality), the loss pushes the model to change its prediction direction. This weighting mechanism ensures that optimization signals act precisely on spatial regions needing improvement, rather than globally applying corrections to the entire image.

The second term is the reference constraint \(\mathcal{L}_{ref} = \mathbb{E}\left[\|\mathbf{v}_\theta - (\epsilon - \mathbf{x}_0^{gt})\|^2\right]\), which is an unweighted standard MSE constraint, similar in function to the KL divergence constraint in DPO. It prevents the reward weighting from causing the policy to drift drastically from the original pre-trained distribution, thereby suppressing reward hacking. The total loss is \(\mathcal{L}_{Total} = \mathcal{L}_r + \lambda_c \cdot \mathcal{L}_{ref}\). Experiments show that \(\lambda_c = 0.5\) is the optimal choice. The entire training process does not update any parameters of the comprehension module, and reward signals are cached once and reused. Scoring 6K candidate images takes less than 4 H100 GPU hours.

Key Experimental Results¶

Main Results¶

SRUM is thoroughly verified on two mainstream UMM architectures, Bagel and BLIP3o. On T2I-CompBench (evaluating compositional generation across 8 fine-grained dimensions), Bagel+SRUM (CoT mode) achieves an overall score of 88.37, surpassing all competitors and yielding a 6.19-point increase over the baseline Bagel (82.18). It sets new SOTAs particularly in spatial reasoning (Spatial: 89.93), 3D spatial (3D Spatial: 88.60), and complex binding (Complex: 91.31) dimensions.

Model	3D Spatial	Color	Complex	Non-Spatial	Quantity	Shape	Spatial	Texture	Overall
Bagel	77.98	89.30	83.32	85.03	70.40	81.94	81.52	87.93	82.18
Bagel+SRUM (CoT)	88.60	92.90	91.31	90.48	80.12	84.47	89.93	89.15	88.37

On T2I-ReasonBench (cross-domain reasoning-to-image benchmark), SRUM also exhibits excellent performance, with Bagel+SRUM scoring 46.75 overall, leading the baseline Bagel (43.82) by nearly 3 points, showcasing clear advantages in complex prompts requiring entity knowledge and idiom comprehension.

Ablation Study¶

Configuration	CompBench Overall Score	Description
Full SRUM	88.37	Complete model
w/o Local Reward	-0.76 (CoT) / -1.04 (Standard)	Significant drop after removing object-level fine-grained feedback
w/o Global Reward	Significant decrease	Lacks guidance for overall composition alignment
w/o Reference Constraint	Significant decrease	Model drifts away from pre-trained distribution, causing reward hacking
0-1 Sparse Reward	Drastic degradation	Extremely poor performance when replacing continuous dense rewards

Key Findings¶

Clear and complementary division of labor between local and global rewards: Analyzing step-by-step scores during inference indicates that the "think mode" primarily improves early compositional layout, global rewards further refine layout at this stage, while local rewards only start to function in later inference stages. This suggests a two-stage optimization pathway of establishing the overall structure before refining local details, which a single reward cannot adequately cover.
Internal evaluator outperforms external LLMs: Replacing the UMM's own 7B comprehension module with QwenVL-2.5-32B as the evaluator surprisingly hurts performance. Self-rewarding shows superior calibration correlation (\(\rho = 0.58\) vs 0.43 for external) and positive/negative sample separation (0.67 vs -0.31) compared to the larger external VLM, signifying that effectiveness stems from the intrinsic multi-scale design rather than parameter scale.
Almost zero damage to comprehension capabilities: Across standard comprehension benchmarks such as MME, MMBench, MM-Vet, MMMU, and MathVista, the changes in comprehension scores before and after SRUM training are negligible (< 1%), and even show improvements on MMVP (hallucination evaluation). This phenomenon indicates that the improvement in generation capability does not come at the expense of comprehension.
Activation analysis reveals distinct paradigms: SFT training represses unrelated functional clusters (narrowing effect), whereas SRUM enhances primary task-related clusters while maintaining moderate activation of auxiliary clusters (enhanced scheduling effect). This might be the underlying reason for SRUM's superior generalization.

Highlights & Insights¶

A complete closed loop of "self-teaching": The most ingenious aspect of SRUM is that it introduces no external information sources—data, scoring, and training are all executed by the model itself. This self-contained self-improvement paradigm demonstrates that the "comprehension stronger than generation" deficiency can be harnessed as a valuable optimization signal.
Dense rewards over sparse rewards: Unlike methods like Dance-GRPO that only yield a single sparse reward score for the entire image, SRUM assigns a continuous reward value to each spatial location. Ablation studies confirm that this dense layout is critical to performance, providing richer gradient information to let the model know "what was done right and what went wrong."
Double defense line against reward hacking: On one hand, the reference constraint term \(\mathcal{L}_{ref}\) prevents the policy from drifting excessively; on the other hand, utilizing offline cached reward signals (rather than online iteratively updated reward models) eliminates adversarial evolution between the reward model and the generation policy. This dual-layered mechanism renders training stable and reproducible.
Impact of different training paradigms on internal division of labor: By defining and computing the average activation intensity of "comprehension clusters" and "generation clusters", it is observed that SFT over-specializes generation clusters and represses comprehension clusters, whereas SRUM achieves a more balanced activation pattern. This insight might hold general significance for understanding the underlying mechanisms of model alignment.

Limitations & Future Work¶

Room for improvement in scoring prompts: The current scoring prompts for the comprehension module use standardized templates, which have limited adaptability to diverse kinds of prompts. Allowing the comprehension module to generate questions and answers in a self-play fashion to build a more closed-loop training system is a natural next step.
Reliance on external localization aids: For UMMs like Bagel that lack native fine-grained grounding, external models like SAM are still needed to provide candidate bounding boxes. Although box validation is handled by the UMM itself, the overall pipeline has increased system complexity. A more ideal direction is enabling the UMM to learn end-to-end localization via self-play.
Aesthetic quality is not the focus of this work: SRUM concentrates on reasoning, knowledge, and compositional abilities. The aesthetic quality of generated images was not explicitly optimized (which is also a common bottleneck for most current UMMs).
Insufficiency in long-tail scenarios like causal multi-image generation: For long-tail tasks requiring causal chains in a single image (e.g., "comparison before and after the Industrial Revolution" demonstrated in the paper's analysis), the UMM struggles due to a lack of such samples in the training data.

vs External Reward Models (UnifiedReward / QwenVL Evaluators): These methods introduce external VLMs to score generated images. SRUM proves that the internal comprehension module, by sharing underlying representations with the generation module, provides better-calibrated reward signals than larger external models.
vs Reconstruction Alignment (ReCA): ReCA utilizes reconstruction loss for post-training, which improves semantic understanding but has limited effect on directly guiding generation. SRUM's reward-weighted mechanism converts semantic comprehension into generation quality improvements more precisely.
vs Chain-of-Thought / Test-time Verification (CoT / Got): These methods perform multi-turn optimization during the inference phase, without increasing the generator's native capacity. SRUM optimizes during the training phase, making them complementary—indeed, SRUM performs best under the CoT inference mode.
vs Self-Rewarding Language Models (CSR / SRPO): Self-rewarding has been explored in language models (on the comprehension side of MLLMs), but SRUM is the first to successfully apply this paradigm to the generation side of unified multimodal models and introduce multi-scale visual evaluation.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ First to introduce the self-rewarding paradigm to the generation side of UMMs; the dense design of global-local dual rewards is highly inspiring.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Exhaustively verified across 2 architectures and 3 benchmarks, featuring ablation studies, generalization analysis, and activation analysis.
Writing Quality: ⭐⭐⭐⭐⭐ Clear logic, well-defined motivation, progressive methodology explanation, and deep experimental analysis (especially the persuasive step-by-step inference analysis).
Value: ⭐⭐⭐⭐⭐ Provides a virtually zero-cost solution for boosting generative capabilities, and conceptually opens up the research pipeline of "UMM self-improvement."