Skip to content

title: >- [Paper Note] GR-GRPO: Graph-Diffused Credit for Autoregressive Image RL Alignment description: >- [ECCV2026][vlm_reasoning][GRPO] Densifies token-level credit in VQ-based autoregressive image RL by diffusing positive trajectory evidence over a codebook K-NN graph, surpassing large-group GRPO (G=64) with ~4.6x less rollout time. tags: - ECCV2026 - vlm_reasoning - GRPO - Autoregressive Image Generation - Graph Regularization date: 2026-09-19 content_hash: 0426aea63b0da5d6

GR-GRPO: Graph-Diffused Credit for Autoregressive Image RL Alignment

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/vivoCameraResearch/GR-GRPO
Area: Multimodal VLM / Image Generation / Alignment & RLHF
Keywords: Autoregressive Image Generation / Reinforcement Learning Alignment / GRPO / Vector-Quantized Codebook / Graph Diffusion Credit Assignment

TL;DR

Addressing the compute-heavy rollout bottleneck and point-wise credit sparsity in small-group (\(G\)) GRPO for VQ-based autoregressive image generation, GR-GRPO leverages local substitutability in codebook embedding space to diffuse positive rollout evidence across a precomputed K-NN graph into soft targets, outperforming large-group GRPO (\(G=64\)) with \(\sim 4.6\times\) less rollout time on Janus-Pro-1B.

Background & Motivation

Autoregressive (AR) models employing decoder-only Transformers and next-token prediction paradigms have become a highly competitive approach for visual synthesis and unified multimodal understanding. To align these generative policies with complex human preferences or compositional prompt constraints, reinforcement learning techniques—especially Group Relative Policy Optimization (GRPO)—have been increasingly adopted. However, on-policy RL training for image synthesis is severely constrained by rollout costs: generating a single image requires sampling hundreds to thousands of visual tokens sequentially, making policy rollouts dominate total training compute. While scaling up the group size \(G\) in GRPO provides broader trajectory exploration and stabilizes intra-group advantage estimation, computational cost scales linearly with \(G\), rendering large-group exploration prohibitive under realistic resource budgets.

This compute bottleneck is further exacerbated by the nature of discrete vector-quantized (VQ) representations. In a visual codebook, adjacent code IDs in embedding space frequently decode into visually and semantically coherent local patterns, exhibiting high "local substitutability" at the token level. Standard GRPO, however, relies on point-wise categorical policy-gradient updates via softmax parameterization: when a rollout succeeds and yields a positive advantage, the gradient explicitly increases the logit of only the exact sampled code ID while depressing all other unobserved IDs proportional to their current probabilities. Under small-group training (\(G=8\)), this point-wise mechanism severely under-covers success-supported alternatives (quantified by SuccessSetMass), trapping the policy in a narrow set of token realizations and causing persistent performance gaps compared to large-\(G\) training.

Through token-substitution perturbation experiments, the authors demonstrate that replacing tokens in successful trajectories with their codebook K-NN neighbors preserves verifier success and CLIP semantic fidelity drastically better than random replacement. This confirms that positive credit can be safely shared within local geometric neighborhoods. The core idea is to harness the intrinsic geometric structure of the precomputed VQ codebook by diffusing positive token hits from successful rollouts over a codebook K-NN graph to build dense soft targets, regularizing the policy via a confidence-adaptive gated cross-entropy objective without requiring additional rollout sampling.

Method

Overall Architecture

The GR-GRPO framework resolves the compute–coverage bottleneck by translating sparse empirical trajectory hits into graph-diffused soft target distributions at near-zero extra computational overhead. The pipeline begins with prompt-level difficulty estimation to dynamically allocate rollout budgets \(G(p)\). The policy then generates \(G\) visual token sequences, decodes them into images, and evaluates them with a discrete verifier or continuous reward model to derive trajectory advantages. For positive rollouts, discrete token hit counts are accumulated at each decoding step and diffused across a precomputed codebook K-NN graph. A support-restricted normalization step constructs a compact soft target distribution, which is then matched via an auxiliary cross-entropy loss modulated by a logit-margin-based confidence gate, jointly optimizing the policy alongside the standard GRPO loss and KL penalty.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Text Prompt"] --> B["Difficulty-Aware Budget Allocation<br/>Assess initial difficulty to assign G(p)"]
    B --> C["Autoregressive Rollout & Scoring<br/>Sample G trajectories and compute advantages"]
    C --> D["Positive Evidence Aggregation & Graph Diffusion<br/>Diffuse positive token counts over codebook K-NN"]
    D --> E["Support-Restricted Target Normalization<br/>Normalize mass over local support Ut to form qt"]
    E --> F["Confidence-Adaptive Gating<br/>Modulate auxiliary cross-entropy via logit margin"]
    F --> G["Joint Policy Optimization<br/>L_GRPO + KL divergence + L_GR"]

Key Designs

1. Positive Evidence Aggregation and Codebook K-NN Graph Diffusion: Bridging Point-Wise Credit Gaps
Standard GRPO awards credit exclusively to sampled code IDs, disregarding the semantic redundancy embedded in the VQ codebook manifold. To transform sparse sampling into structured supervision, the framework first filters the positive rollout set \(\mathcal{P} \subseteq \{1,\dots,G\}\) within each group (defined as \(\mathcal{P} = \{j \mid R^{(j)}=1\}\) for discrete verifiers, or top-50% trajectories by reward for continuous scorers). At each position \(t\), empirical hit counts are tallied as \(c_t(i) = \sum_{j \in \mathcal{P}} \mathbb{I}[x_t^{(j)} = i]\) across vocabulary \(i \in \{1,\dots,V\}\). A fixed \(K\)-NN directed graph \(\mathcal{N}(i)\) is precomputed over normalized codebook embeddings \(\bar{e}_i = e_i / \|e_i\|_2\) using cosine similarity, with transition weights defined via softmax-normalized similarities: $\(a_{ij} = \frac{\exp(\mathrm{cos}(\bar{e}_i, \bar{e}_j))}{\sum_{\ell \in \mathcal{N}(i)} \exp(\mathrm{cos}(\bar{e}_i, \bar{e}_\ell))}, \quad j \in \mathcal{N}(i)\)$ Observed counts are diffused to immediate neighbors via \(\tilde{c}_t(j) = \sum_{i: c_t(i)>0} c_t(i) a_{ij} \mathbb{I}[j \in \mathcal{N}(i)]\). Importantly, diffusion is strictly restricted to positive evidence: failure modes are heterogeneous and spatially dispersed across the embedding space, and diffusing negative evidence corrupts valid semantic neighborhoods, causing empirical degradation.

2. Support-Restricted Normalization and Soft Target Alignment: Set-Valued Probability Pull
Diffusing credit globally over the entire vocabulary \(V\) would introduce excessive computational overhead and cause cross-prefix semantic distortion. Thus, supervision is constrained to a sparse local support \(U_t = \{i \mid c_t(i) > 0\} \cup \mathcal{N}(\{i \mid c_t(i) > 0\})\), which contains only the observed successful tokens and their immediate \(K\)-NN neighbors. Unnormalized mass is constructed on \(U_t\) as: $\(\hat{q}_t(i) = c_t(i) + \alpha_s \tilde{c}_t(i), \quad i \in U_t\)$ where \(\alpha_s \in [0, 1]\) governs neighborhood diffusion strength. Normalizing across \(U_t\) yields soft targets \(q_t(i) = \hat{q}_t(i) / \sum_{j \in U_t} \hat{q}_t(j)\). Under unclipped gradient dynamics, standard GRPO applies logit updates proportional to \(A^{(j)}(\mathbb{I}[i=x_t^{(j)}] - \pi_\theta(i))\). By replacing the one-hot target with \(q_t(i)\), the auxiliary loss pulls probability mass toward \((q_t(i) - \pi_\theta(i))\), transforming point-wise credit into a set-valued attraction over local substitutable candidates and substantially enriching token-level supervision.

3. Confidence-Adaptive Gating Mechanism: Mitigating Over-Smoothing in Decisive Contexts
Across an autoregressive image sequence, model certainty varies drastically: certain deterministic backgrounds or rigid local patterns demand sharp token selection, whereas compositional layout decisions admit multiple valid alternatives. Applying uniform graph regularization indiscriminately can over-smooth peak logits where the policy is already confident. To prevent this, the auxiliary objective is dynamically gated by the top-1/top-2 logit margin \(m_t^{(j)} = z_{t,(1)}^{(j)} - z_{t,(2)}^{(j)}\) computed on prefix \((x_{<t}^{(j)}, p)\): $\(g_t^{(j)} = \sigma\left(\frac{\beta - m_t^{(j)}}{\tau}\right)\)$ When the margin is narrow (low confidence, high ambiguity), \(g_t \to 1\), deploying strong graph-diffused supervision to guide exploration toward valid neighborhoods; when the margin is large (high confidence), \(g_t \to 0\), suppressing diffusion to protect sharp, settled predictions. A stop-gradient operator is applied to \(g_t\) so it functions purely as an adaptive reweighting scalar.

4. Difficulty-Aware Rollout Budget Allocation: Orthogonal Utility Maximization
In addition to trajectory-level credit densification, prompt-level exploration demands are non-uniform: easy prompts succeed with minimal rollouts, while difficult compositional prompts frequently yield zero successful trajectories under small \(G\), completely stalling graph regularization. The system introduces an offline difficulty scorer using the initial policy to partition prompts into difficulty bins, assigning larger group sizes \(G(p)\) to challenging prompts while trimming sampling on easy ones under a fixed total rollout budget. This orthogonal mechanism stabilizes the availability of positive seed trajectories for graph diffusion.

Loss & Training

The overall training objective combines the clipped surrogate GRPO objective, a reference policy KL penalty, and the gated graph-regularization cross-entropy loss: $\(\mathcal{L}(\theta) = \mathcal{L}_{\text{GRPO}}(\theta) + \lambda_{\text{KL}}\mathcal{L}_{\text{KL}}(\theta) + \lambda_{\text{GR}}\mathcal{L}_{\text{GR}}(\theta)\)$ where the graph regularization loss is evaluated exclusively over positive rollout prefixes: $\(\mathcal{L}_{\text{GR}}(\theta) = -\frac{1}{G}\sum_{j \in \mathcal{P}}\sum_{t=1}^{T} g_t^{(j)} \sum_{i \in U_t} q_t(i) \log \pi_\theta(i \mid x_{<t}^{(j)}, p)\)$ Training is executed on 8× NVIDIA H20 GPUs for 1600 steps on 50K GenEval prompts (discrete verifier) and 15K HPSv2 prompts (continuous HPSv2.1 reward). Default hyperparameters include \(\lambda_{\text{KL}} = 0.01\), \(\lambda_{\text{GR}} = 0.05\), \(K = 64\), \(\alpha_s = 0.5\), gating constants \((\beta, \tau) = (1.0, 0.5)\), and learning rates of \(3\times 10^{-6}\) for GenEval and \(1\times 10^{-6}\) for DrawBench.

Key Experimental Results

Main Results

Evaluated on Janus-Pro-1B and Janus-Pro-7B backbones, GR-GRPO consistently outperforms the base models and competitive RL baselines across both discrete verifier (GenEval) and continuous preference (DrawBench) benchmarks.

Table 1: GenEval Compositional Alignment Results (Official Verifier Success Rate)

Method / Model Overall↑ Sing Obj.↑ Two Obj.↑ Counting↑ Color↑ Position↑ Color Attr.↑
Janus-Pro-1B (Base) 0.73 0.98 0.82 0.51 0.89 0.65 0.56
Janus-Pro-1B + GRPO 0.84 1.00 0.95 0.59 0.84 0.88 0.77
GCPO-1B 0.85 1.00 0.96 0.63 0.88 0.91 0.73
GR-GRPO-1B (Ours) 0.87 1.00 0.98 0.67 0.89 0.94 0.76
Janus-Pro-7B (Base) 0.80 0.99 0.89 0.59 0.90 0.79 0.66
Janus-Pro-7B + GRPO 0.87 0.99 0.92 0.71 0.94 0.92 0.73
GCPO-7B 0.90 0.99 0.95 0.90 0.90 0.95 0.76
STAGE-7B 0.89 0.99 0.95 0.82 0.90 0.89 0.79
GR-GRPO-7B (Ours) 0.92 1.00 0.97 0.91 0.92 0.96 0.74

Table 2: DrawBench Preference and Visual Quality Multi-Metric Evaluation

Method / Model HPSv2.1↑ DeQA↑ PickScore↑ ImageReward↑ Aesthetic↑ UniReward↑
Janus-Pro-1B (Base) 25.63 3.58 21.55 0.54 5.89 2.76
Janus-Pro-1B + GRPO 27.22 3.72 21.71 0.73 6.04 2.80
GCPO-1B 27.18 3.71 21.72 0.71 6.05 2.82
GR-GRPO-1B (Ours) 27.55 3.76 21.73 0.74 6.06 2.85
Janus-Pro-7B (Base) 26.74 3.61 22.06 0.87 5.90 3.02
Janus-Pro-7B + GRPO 27.52 3.68 22.11 0.92 5.99 3.06
GCPO-7B 26.84 3.59 22.04 0.84 5.87 3.01
STAGE-7B 27.76 3.70 22.19 0.96 6.00 3.10
GR-GRPO-7B (Ours) 28.25 3.74 22.21 0.99 6.06 3.07

Ablation Study

All ablations are conducted on Janus-Pro-1B evaluated on GenEval.

Table 3: Component Ablations on GenEval (Janus-Pro-1B)

Method Configuration Overall↑ Sing Obj.↑ Two Obj.↑ Counting↑ Color↑ Position↑ Color Attr.↑ Note
GRPO (Baseline) 0.84 1.00 0.95 0.59 0.84 0.88 0.77 Standard baseline
GRPO + Budget Control 0.84 1.00 0.96 0.60 0.84 0.88 0.76 Negligible gain without graph credit
GR-GRPO w/o Budget Control 0.86 0.99 0.96 0.64 0.88 0.90 0.77 Graph regularization alone yields +0.02
GR-GRPO + Budget Control (Ours) 0.87 1.00 0.98 0.67 0.89 0.94 0.76 Synergy achieves top performance (+0.03)

Table 4: Hyperparameter Sensitivity and Mechanism Ablation (GenEval, Janus-Pro-1B)

Dimension Setting Overall↑ Key Findings
Neighborhood Size \(K\) 32 / 64 (Ours) / 128 0.84 / 0.87 / 0.86 \(K=32\) is too small for coverage; \(K=128\) introduces noise; 64 is optimal
Gating Strategy Constant \(g_t \equiv 1\) / Adaptive (Ours) 0.83 / 0.87 Constant gating causes severe over-smoothing; adaptive gating adds +0.04
Diffusion Scope Positive Only (Ours) / Pos + Neg 0.87 / 0.84 Negative modes are heterogeneous; diffusing negative evidence degrades performance
Diffusion Strength \(\alpha_s\) 0.3 / 0.5 (Ours) / 0.7 0.84 / 0.87 / 0.84 Low \(\alpha_s\) under-regularizes; high \(\alpha_s\) over-dilutes ground-truth evidence

Key Findings

  • Graph Credit Densification is the Core Driver: Graph regularization at \(G=8\) independently pushes Janus-Pro-1B GenEval score from 0.84 to 0.86, with standout improvements on spatial binding (Position: 0.88 → 0.94) and object enumeration (Counting: 0.59 → 0.67).
  • Substantially Superior Compute–Coverage Pareto Frontier: In standard GRPO, increasing \(G\) from 4 to 64 yields gradual gains but multiplies sampling time by nearly an order of magnitude. GR-GRPO raises SuccessSetMass structurally, outperforming GRPO at \(G=64\) with \(\sim 4.6\times\) less rollout time (0.87 vs ~0.86).
  • Adaptive Gating is Essential: Replacing adaptive gating with \(g_t \equiv 1\) causes performance to plummet to 0.83 (below vanilla GRPO's 0.84), verifying that unconstrained diffusion across confident decision steps impairs attribute fidelity.

Highlights & Insights

  • Repurposing VQ Manifold Geometry for Online RL: While most visual RL frameworks treat visual token codebooks as disconnected discrete integers, GR-GRPO recognizes that VQ embedding space possesses smooth, exploitable local topologies that naturally bridge point-wise credit to set-valued coverage.
  • Asymmetric Diffusion Properties of Positive vs. Negative Evidence: The empirical insight that positive solutions cluster coherently along codebook manifolds while failure patterns are dispersed and heterogeneous provides valuable intuition for future discrete generative RL regularizer designs.
  • Negligible Compute Overhead with Instant Portability: The graph diffusion step operates purely as a sparse table lookup and accumulation on precomputed static graphs, demanding zero additional forward passes or neural discriminators.

Limitations & Future Work

  • Reliance on Discrete Codebook Topology: The formulation relies on a fixed VQ codebook embedding space and K-NN graph, making it non-trivial to apply directly to continuous-token autoregressive models (e.g., Fluid) or pure flow/diffusion models without explicit token embeddings.
  • Static Neighborhoods Lack Context Sensitivity: The precomputed K-NN graph is static and condition-agnostic. In rare contexts, certain neighboring IDs might cross semantic boundaries (e.g., contrasting colors or fine attributes).
  • Future Directions: The authors suggest extending graph credit densification to continuous-token representations via dynamic metric learning or online neighborhood graphs, as well as exploring prompt-conditioned graph pruning.
  • vs. Standard GRPO (DeepSeekMath): Standard GRPO relies on one-hot categorical policy gradients that fail to credit unsampled yet valid alternatives under tight rollout budgets; GR-GRPO injects graph-diffused soft targets to reproduce large-\(G\) coverage under small-\(G\) compute.
  • vs. GCPO (Group Critical-token Policy Optimization): While GCPO focuses on temporal/step-wise critical token identification along trajectories, GR-GRPO operates along the vocabulary/embedding spatial dimension. Both mechanisms are complementary.
  • vs. Reasoning-Augmented AR Generation (T2I-R1, GoT-R1): CoT-based methods explicitly introduce multi-stage verbalized reasoning tokens (layout planning, semantic chains), expanding decoding length and rollout cost; GR-GRPO preserves vanilla end-to-end visual token decoding while substantially improving compositional alignment.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ [Pioneering diagnosis of the compute-coverage bottleneck in VQ-based visual RL; elegant graph diffusion credit assignment]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across 1B/7B scales, discrete GenEval and continuous DrawBench metrics, paired with meticulous ablations]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Crisp formulation, seamless logical transitions from motivation to gradient derivations and empirical analysis]
  • Value: ⭐⭐⭐⭐⭐ [Directly tackles the costliest rollout bottleneck in generative RL alignment with an open-source, highly practical drop-in solution]