RubricRM: Generative Reward Modeling via Dynamic Rubrics for Image Generation and Editing¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/zijiankan/RubricRM
Area: LLM Reasoning
Keywords: reward model, dynamic rubrics, image generation and editing, multimodal evaluation, GRPO
TL;DR¶
Addressing the lack of interpretability and input-adaptability in scalar reward models and static MLLM judges for visual generation, RubricRM dynamically generates an input-adaptive rubric with atomic dimensions, normalized weights, and 5-level scoring criteria to evaluate candidate images in a single forward pass, optimized through trajectory SFT and fine-grained dimension-level GRPO.
Background & Motivation¶
Visual generative models increasingly rely on reward models for post-training preference alignment via reinforcement learning or direct preference optimization. However, existing visual evaluation paradigms suffer from fundamental trade-offs between expressive granularity and task adaptability: global similarity metrics like CLIPScore lack sensitivity to compositional semantics; discriminative preference models such as ImageReward, HPS v2/v3, PickScore, and EditReward output single uninterpretable scalar scores that provide no inspectable justification for preference decisions; and recent multimodal LLM judges predominantly rely on predefined, static dimensions and unweighted evaluation heuristics that fail to adapt to varied task contexts.
This rigidity becomes acutely problematic in text-to-image (T2I) generation and instruction-guided image editing, where different inputs naturally require entirely distinct evaluation dimensions. A realistic photographic prompt demands strict anatomical coherence, lighting logic, and physical plausibility, whereas a creative fantasy prompt values conceptual novelty, stylistic adherence, and visual harmony. In image editing, the trade-off is even more complex: the evaluator must simultaneously verify faithful instruction execution, guard against unintended corruptions of non-target areas, and preserve background blending, with criteria varying drastically between object replacements, color modifications, and multi-step reasoning edits. Evaluating such diverse scenarios with static or unweighted criteria inevitably introduces evaluation drift, subjective scoring noise, and poor calibration across disparate visual scenarios.
To resolve this bottleneck, an intermediate evaluation protocol must be established between free-form visual rationales and scalar preference decisions. Automated rubrics fulfill this objective by explicitly defining what atomic aspects should be judged, how much weight each dimension carries, and what explicit qualitative standards determine each score level. By grounding preference judgments in input-adaptive rubrics, the evaluation process achieves context-aware sensitivity while keeping the entire reasoning path fully verifiable and inspectable. Core idea: formulate visual preference judgment as input-conditioned dynamic rubric generation followed by rubric-grounded dimension-level scoring within a single inference pass, trained via a two-stage pipeline combining full-trajectory SFT with fixed-rubric dimension-level GRPO.
Method¶
Overall Architecture¶
RubricRM provides a unified, adaptive framework for pairwise visual preference modeling across text-to-image generation and instruction-guided image editing. For text-to-image generation, the model receives a text prompt \(q\) and two candidate images \(I_A, I_B\). For image editing, \(q\) additionally includes the original source image \(I_{\text{src}}\) and the editing instruction. In a single forward pass, RubricRM sequentially generates the task intent analysis, dynamic rubric specifications, per-candidate dimension-level critique with discrete scores, and the final aggregated preference. Formally, the generated rubric \(\mathcal{R} = \{(d_i, w_i, L_i)\}_{i=1}^N\) specifies \(N\) atomic dimensions \(d_i\), normalized importance weights \(w_i \in [0, 1]\) satisfying \(\sum_{i=1}^N w_i = 1\), and graded scoring descriptors \(L_i\). The model then assigns discrete scores \(s_k^i \in [0, 4]\) for each candidate image \(I_k\) (\(k \in \{A, B\}\)) under dimension \(d_i\), computing the overall weighted score \(S(I_k) = \sum_{i=1}^N w_i \cdot s_k^i\) to determine the preferred image with the higher total score.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
In["Input: Prompt q (and source image) + Candidate Pair (I_A, I_B)"] --> Step1["Classification-Driven Balanced Sampling and Label-Conditioned Trajectory Synthesis"]
Step1 --> Step2["Dynamic Rubric Construction and Single-Pass End-to-End Inference"]
Step2 --> Step3["Fixed-Rubric Rollout with Dimension-Level Reward Design"]
Step3 --> Step4["Continuous GRPO Policy Optimization with Saturated Group Filtering"]
Step4 --> Out["Output: Dynamic Rubric + Per-Dimension Scores + Weighted Total Preference"]
Key Designs¶
1. Classification-Driven Balanced Sampling and Label-Conditioned Trajectory Synthesis: Resolving Long-Tail Deficits and Teacher Preference Bottlenecks Learning generalizable dynamic rubrics requires broad data distribution across diverse prompt types and compositional styles. Because raw visual prompts frequently exhibit multi-faceted characteristics, a hierarchical multi-label taxonomy is developed (6 primary and 42 secondary categories for T2I; 12 primary and 30 secondary categories for editing). Automated multi-label tagging via DeepSeek-V3.2 identifies severely underrepresented long-tail categories such as text rendering (6.45%), logical reasoning (8.98%), and world knowledge (15.83%). Supplementary prompts are synthesized and validated by human experts, yielding 31,835 T2I pairs and 30,695 editing pairs. To synthesize high-quality training trajectories without inheriting the standalone preference errors of a teacher model, label-conditioned trajectory synthesis is employed: the human ground-truth preference label \(y_j\) is explicitly fed to Gemini 3.1 Pro alongside the inputs, anchoring the teacher's task to decomposing the human preference into structured intent, relevant dimensions, normalized weights, and per-dimension scores rather than determining the winner independently.
2. Dynamic Rubric Construction and Single-Pass End-to-End Inference: Eliminating Multi-Stage Cascading Latency and Standardizing Atomic Criteria Rather than orchestrating multi-agent pipelines with multiple disjoint LLM/VLM calls to create and verify rules, RubricRM integrates intent mining, rubric generation, fine-grained visual comparison, and score aggregation into a single, cohesive autoregressive generation pass. Given the input, the model dynamically generates 3 to 5 atomic dimensions that avoid overlapping concepts, assigning normalized percentage weights summing strictly to 100%. Scoring across all dimensions follows a universal 5-level rubric scale (\(0 = \text{Failed}\), severe breakdowns or errors; \(1 = \text{Poor}\), weak satisfaction with major artifacts; \(2 = \text{Fair}\), partial satisfaction with notable flaws; \(3 = \text{Good}\), well-satisfied with minor flaws; \(4 = \text{Excellent}\), fully satisfied and essentially flaw-free). This self-contained single-pass formulation eliminates formatting fragility and parsing latency, enabling tightly coupled reasoning between criteria selection and candidate comparison.
3. Fixed-Rubric Rollout with Dimension-Level Reward Design: Overcoming Sparse RL Feedback and Dimension Misalignment Directly rewarding the policy on binary preference accuracy provides only sparse, uninformative signals that reward lucky guesses even when individual dimension judgments are completely inverted. To provide dense, dimension-level feedback, reinforcement learning must evaluate model-predicted score gaps against reference score gaps on each dimension. This requires strict one-to-one correspondence between rollout dimensions and reference dimensions. During GRPO training, the synthesized rubric \(R_j\) is fixed as input context, prompting rollouts to generate only the dimension evaluations and scores \(J_j\). For each dimension \(d_i\), the predicted gap \(\Delta_{\text{pred}}^i = s_{A,\text{pred}}^i - s_{B,\text{pred}}^i\) is compared against the reference gap \(\Delta_{\text{gt}}^i = s_{A,\text{gt}}^i - s_{B,\text{gt}}^i\). The dimension reward combines a normalized magnitude alignment factor \(b_i = 1 - \frac{|\Delta_{\text{pred}}^i - \Delta_{\text{gt}}^i|}{2(s_{\max} - s_{\min})}\) and a direction factor \(\phi_i\) (\(\phi_i = 1.0\) if signs match; \(\phi_i = 0.6\) if exactly one gap is zero; \(\phi_i = 0.1\) for full sign reversal), aggregated as \(R = \sum_{i=1}^N w_i b_i \phi_i\).
4. Continuous GRPO Policy Optimization with Saturated Group Filtering: Mitigating Vanishing Variance Gradient Exploitation In Group Relative Policy Optimization (GRPO), sampled responses within a group (\(G = 8\)) are assigned relative advantages \(\hat{A}_i = \frac{R_i - \mu_R}{\sigma_R + \epsilon}\). Under continuous dimension-level rewards, rollout groups where all completions receive almost identical rewards exhibit a near-zero standard deviation (\(\sigma_R \approx 0\)). In such saturated, low-spread groups, negligible numerical noise is amplified into massive relative advantages, causing the policy to violently update on non-informative perturbations. This leads to empirical training collapse: late-stage reward drops, runaway response length inflation, and policy entropy collapse. To ensure training stability, saturated group filtering is introduced with a variance threshold \(\sigma_{\text{th}} = 0.05\). If \(\sigma_R < \sigma_{\text{th}}\), all advantages in the group are zeroed (\(A_i = 0\)), bypassing noisy updates while preserving clean optimization steps on rollout groups with meaningful reward divergence.
Loss & Training¶
The model is optimized across two sequential stages: Stage 1 consists of Supervised Fine-Tuning (SFT) over filtered trajectory dataset \(\mathcal{D}_{\text{SFT}}\) using standard autoregressive next-token cross-entropy: $\(\mathcal{L}_{\text{SFT}} = - \sum_{j} \sum_{t=1}^{T_j} \log \pi_\theta(t_{j,t} \mid x_j, t_{j,<t})\)$ trained for 2 epochs with learning rate \(5\times 10^{-6}\) and cosine decay schedule.
Stage 2 performs reinforcement learning with dimension-level GRPO over disjoint 15K splits per task, maximizing the clipped objective: $\(\mathcal{J}_{\text{GRPO}}(\theta) = \mathbb{E}_{\tilde{x}, \{o_i\}_{i=1}^G} \left[ \frac{1}{G} \sum_{i=1}^G \min\left( \rho_i A_i, \, \text{clip}(\rho_i, 1-\epsilon, 1+\epsilon) A_i \right) - \beta D_{\text{KL}}(\pi_\theta(o_i \mid \tilde{x}) \parallel \pi_{\text{ref}}(o_i \mid \tilde{x})) \right]\)$ where \(\rho_i = \frac{\pi_\theta(o_i \mid \tilde{x})}{\pi_{\theta_{\text{old}}}(o_i \mid \tilde{x})}\), group size \(G = 8\), clipping parameter \(\epsilon = 0.2\), KL weight \(\beta = 0.05\), and learning rate \(5\times 10^{-7}\). Training is executed on 8 NVIDIA H20 GPUs across Qwen3.5-4B and Qwen3.5-9B backbones.
Key Experimental Results¶
Main Results¶
On text-to-image generation benchmarks (MMRB2, GenAI-Bench, and GenAI-Bench-Verified) and image editing benchmarks (MMRB2, EditReward-ERB, and EditScore-ERB), RubricRM was compared against leading proprietary MLLMs, general open-source MLLMs, and specialized reward models.
| Model | MMRB2 (Gen) | GenAI-Bench | GenAI-Bench-Ver. | MMRB2 (Edit) | EditReward-ERB (Avg) | EditScore-ERB (Avg) |
|---|---|---|---|---|---|---|
| Claude Sonnet 4.6 (Proprietary) | 70.80 | 65.80 | 75.30 | 71.70 | 44.10 | 79.30 |
| GPT-5.4 (Proprietary) | 67.50 | 64.20 | 74.20 | 68.50 | 42.50 | 74.60 |
| Gemini 3.1 Pro (Proprietary) | 74.40 | 73.90 | 84.80 | 74.90 | 45.00 | 81.60 |
| Qwen3-VL-8B | 61.20 | 63.30 | 72.50 | 63.40 | 40.90 | 76.90 |
| Qwen3.5-9B | 66.30 | 63.30 | 70.70 | 64.40 | 37.40 | 72.00 |
| Qwen3.5-397B-A17B | 72.70 | 66.20 | 77.00 | 73.70 | 43.90 | 81.20 |
| HPSv3 | 60.20 | 70.90 | 81.00 | - | - | - |
| EditReward-7B | - | - | - | 67.20 | 38.40 | 78.30 |
| UnifiedReward-Flex-8B | 69.20 | 73.40 | 84.20 | - | - | - |
| RubricRM-4B (Ours) | 70.50 | 73.20 | 83.11 | 73.20 | 45.50 | 85.50 |
| RubricRM-9B (Ours) | 72.00 | 74.12 | 84.45 | 75.40 | 46.40 | 85.64 |
On EditScore-ERB fine-grained sub-metrics, RubricRM-Edit-9B achieves 88.60% in Prompt Following (PF), 79.66% in Consistency (C), and 87.70% in Overall Quality (O), notably outperforming the specialized EditReward-7B baseline on Consistency by +12.46 percentage points.
Ablation Study¶
The table below details the step-by-step ablation of the two-stage training pipeline (Base Backbone → +SFT → +RL/GRPO) across text-to-image generation and image editing, alongside direct comparison with Label-only SFT.
| Configuration | T2I MMRB2 | GenAI-Bench | GenAI-Ver. | Edit MMRB2 | EditReward-ERB | EditScore-ERB |
|---|---|---|---|---|---|---|
| Qwen3.5-4B (Base) | 63.30 | 61.90 | 69.70 | 64.70 | 37.10 | 72.20 |
| + Rubric SFT | 70.10 (+6.8) | 72.00 (+10.1) | 82.90 (+13.1) | 71.50 (+6.8) | 43.60 (+6.5) | 83.30 (+11.1) |
| + Dimension GRPO (Full) | 70.50 (+0.4) | 73.20 (+1.2) | 83.10 (+0.2) | 73.20 (+1.7) | 45.50 (+1.9) | 85.50 (+2.2) |
| Qwen3.5-9B (Base) | 66.90 | 63.40 | 72.50 | 64.40 | 37.40 | 72.00 |
| + Rubric SFT | 70.30 (+3.4) | 73.00 (+9.6) | 83.20 (+10.7) | 73.80 (+9.4) | 45.00 (+7.6) | 85.10 (+13.1) |
| + Dimension GRPO (Full) | 72.00 (+1.7) | 74.10 (+1.2) | 84.50 (+1.2) | 75.40 (+1.6) | 46.40 (+1.4) | 85.60 (+0.5) |
| Qwen3.5-9B + Label-only SFT | 66.40 (-3.9) | 71.10 (-1.9) | 80.90 (-2.3) | 69.20 (-4.6) | 40.80 (-4.2) | 83.90 (-1.2) |
Key Findings¶
- SFT Instills Evaluation Protocols, GRPO Calibrates Scoring Accuracy: Trajectory SFT drives the vast majority of performance gains (+3.4 to +13.1 points across benchmarks), proving that learning to formulate atomic dimensions, assign weights, and generate graded criteria is the primary driver of preference discernment. Dimension-level GRPO delivers consistent further gains (+0.2 to +2.2 points), specifically fine-tuning relative score differences and penalizing preference direction reversals.
- Rubrics Outperform Direct Label Imitation: Across all metrics, Rubric-based SFT substantially outperforms Label-only SFT (e.g., +3.9 points on T2I MMRB2 and +4.6 points on Edit MMRB2 for 9B). This demonstrates that the improvements stem from structured intermediate reasoning rather than merely memorizing preference distributions.
- Saturated Group Filtering Prevents Entropy Collapse: Filtering rollout groups with reward variance below \(\sigma_{\text{th}} = 0.05\) effectively halts policy drift, keeping response length bounded and reward trajectories stable throughout RL training.
Highlights & Insights¶
- Inspectable and Auditable Evaluation Trace: Unlike opaque scalar models, RubricRM outputs explicit atomic criteria and numerical weights that allow human developers to pinpoint whether an alignment failure stems from poor prompt adherence, stylistic discrepancies, or background artifacts.
- Fixed-Rubric Conditioning for Dense RL: Fixing the rubric during training rollouts solves the dimension mismatch dilemma in continuous reinforcement learning, providing dense per-dimension feedback without sacrificing single-pass test-time flexibility.
- Variance-Gated Advantage Normalization: The saturated group filtering mechanism provides a lightweight, mathematically sound safeguard against gradient explosion caused by near-zero variance groups in continuous-reward GRPO.
Limitations & Future Work¶
- Dependence on Teacher Trajectory Quality: Synthesizing training trajectories via Gemini 3.1 Pro risks transferring latent teacher priors and aesthetic biases into student rubrics, even when anchored by ground-truth human labels.
- Restriction to Static Imagery: The current formulation evaluates static 2D images and does not extend to video generation or temporal editing, which require modeling dynamic continuity, motion smoothness, and long-range coherence.
- Future Directions: Exploring self-improving teacher-free rubric evolution, incorporating human-in-the-loop criterion correction, and extending dynamic rubric modeling to multimodal embodied agent trajectories and video generation.
Related Work & Insights¶
- vs ImageReward / HPS v2/v3 / EditReward: These discriminative reward models produce scalar scores that compress rich visual information into a single uninterpretable number; RubricRM generates transparent, input-adaptive rubrics with verifiable dimension-level scores while outperforming them across generation and editing benchmarks.
- vs UnifiedReward-Think / UnifiedReward-Flex: While earlier generative models introduced free-form rationales or adaptive prompts, they lacked atomic dimension decomposition, explicit normalized weighting, and graded score criteria; RubricRM delivers a fully structured, weighted rubric and score calculation in a single end-to-end forward pass.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ Introduces input-specific dynamic rubrics for visual preference modeling and formulates a fixed-rubric dimension-level GRPO framework.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 6 competitive benchmarks, fine-grained subtask breakdowns, robust ablations, and cross-teacher sensitivity audits.
- Writing Quality: ⭐⭐⭐⭐⭐ Lucid technical narrative with rigorous mathematical formulation, structured prompt specifications, and clear case studies.
- Value: ⭐⭐⭐⭐⭐ Establishes a transparent, high-performing reward modeling paradigm for post-training alignment in image generation and editing.