Policy-Based Tuning of Autoregressive Image Models with Instance- and Distribution-Level Rewards

Conference: ECCV 2026
Paper: ECCV Poster
Code: https://github.com/bugrabaran/ar-policy-tuning
Area: Image Generation
Keywords: Autoregressive image generation, policy optimization, GRPO, distribution-level reward, leave-one-out FID

TL;DR

Formulates discrete-token autoregressive image generation as an MDP and applies GRPO policy fine-tuning using a novel EMA-based leave-one-out FID (LOO-FID) distribution-level reward paired with instance-level perceptual rewards and adaptive entropy regularization, resolving diversity collapse while matching or surpassing Classifier-Free Guidance quality to deliver a 2x inference speedup.

Background & Motivation

Autoregressive (AR) models have re-emerged as a competitive paradigm for high-fidelity visual synthesis by leveraging Transformer decoders over discrete image tokens (e.g., VQGAN, LlamaGen), effectively unifying image generation with the GPT-style language modeling paradigm. Despite recent advances matching or exceeding diffusion models in visual quality and sampling throughput, modern AR image generators are almost universally trained via standard maximum likelihood estimation (MLE). Because MLE optimizes token-level cross-entropy conditioned on teacher-forced ground-truth prefixes, it cannot directly optimize image-level perceptual preferences, semantic alignment, or sample diversity, severely constraining post-training controllability.

In the diffusion domain, reinforcement learning and preference optimization (such as DDPO and Diffusion-DPO) have become foundational for human alignment, yet they consistently suffer from severe sample diversity collapse, mode dropping, and excessive computational overhead across multi-step denoising trajectories. Concurrently, initial attempts to apply RL to AR image generators (e.g., AR-GRPO and VA-\(\pi\)) exploit AR generation's natural discrete MDP structure to bypass value networks. However, these methods rely exclusively on per-image instance-level scalar rewards (such as CLIPScore and HPSv2). Optimizing strictly for per-instance scores inevitably encourages the policy to collapse onto narrow, high-scoring modes, trading distributional coverage and visual diversity for deceptive quality metrics.

The fundamental tension lies in the fact that instance-level rewards evaluate each sample in isolation and cannot detect whether the overall generated ensemble accurately reflects the real data distribution. To overcome this limitation, the authors rethink the reward formulation: individual perceptual fidelity must be explicitly counterbalanced by global distribution alignment signals directly within policy optimization. Core idea: cast token-based AR generation as an MDP, introduce a novel distribution-level Leave-One-Out FID (LOO-FID) reward driven by an exponential moving average (EMA) of generator feature moments to assign marginal alignment credit to individual samples, and optimize jointly with instance-level rewards and adaptive entropy regularization under GRPO to eliminate mode collapse.

Method

Overall Architecture

The framework models class-conditional discrete autoregressive image generation as a Markov Decision Process (MDP). Given a target class condition \(c\) and an image token sequence \(x = (x_1, \dots, x_T)\) mapped into a discrete codebook of size \(K\), at each step \(t\) the state is \(s_t = (c, x_{<t})\), the action is the discrete codebook index \(a_t = x_t\), and the generation policy is parameterized by the AR Transformer decoder \(\pi_\theta(a_t \mid s_t)\). Once a full token trajectory \(x_{1:T}\) is sampled, the discrete tokenizer reconstructs it into a pixel-space image, which is evaluated by both instance- and distribution-level reward models.

Policy optimization is conducted using Group Relative Policy Optimization (GRPO), eliminating the need for a separate value network. For each conditioning class, the model samples a group of \(G\) candidate token trajectories. The group rewards are normalized into advantages, which update the policy parameters using clipped importance sampling ratios. The entire system comprises three tightly integrated components: composite instance-level rewards (CLIPScore and HPSv2) for semantic and aesthetic alignment, the EMA-based LOO-FID distribution reward for global manifold coverage, and an adaptive closed-loop entropy bonus that dynamically halts premature policy collapse.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Class condition c and prefix tokens"] --> B["Autoregressive policy rollout<br/>sample group of G candidate sequences"]
    B --> C["Discrete tokenizer reconstruction<br/>decode synthesized candidate images"]
    C --> D["Composite reward assignment<br/>CLIP+HPSv2 instance rewards and EMA LOO-FID distribution reward"]
    D --> E["Adaptive entropy regularization<br/>closed-loop control tracking target H"]
    E --> F["GRPO policy parameter update<br/>advantage normalization and clipped ratio gradient step"]

Key Designs

1. Leave-One-Out FID (LOO-FID) with running feature moment EMA: eliminating batch variance and preventing mode collapse

Relying solely on per-sample scalar rewards permits mode collapse because individual scores cannot penalize missing modes or redundant generations. However, directly applying the standard Fréchet Inception Distance (FID) to small minibatches is intractable: per-batch sample statistics exhibit extreme variance, depend heavily on transient batch composition, and fail to track the generator's global evolving distribution.

To solve this, the authors develop a Leave-One-Out (LOO) FID estimator powered by an Exponential Moving Average (EMA). Pre-computed reference mean and diagonal standard deviation vectors \((\mu_r, \sigma_r) \in \mathbb{R}^D \times \mathbb{R}^D_{\ge 0}\) are obtained from real images in the Inception feature space. During training, the generator maintains a running EMA of its generated first-order raw moments \(\mu^{(t)}\) and second-order raw moments \(m_2^{(t)}\) with decay factor \(\alpha \in (0, 1)\):

\[\mu^{(t+1)} = (1-\alpha)\mu^{(t)} + \alpha\hat{\mu}, \qquad m_2^{(t+1)} = (1-\alpha)m_2^{(t)} + \alpha\widehat{m_2}\]

The running standard deviation is computed element-wise as \(\sigma^{(t+1)} = \sqrt{\max(m_2^{(t+1)} - \mu^{(t+1)} \odot \mu^{(t+1)}, 0) + \varepsilon}\). To measure the exact marginal contribution of an individual sample \(j\) with Inception feature \(f_j\), a hypothetical LOO batch update is formed by recomputing moments \((\hat{\mu}_{-j}, \widehat{m_{2,-j}})\) without \(f_j\), producing hypothetical EMA moments \((\mu_{-j}^{(t+1)}, \sigma_{-j}^{(t+1)})\). The per-sample distribution reward is then defined as the change in EMA-aligned diagonal FID:

\[r_j^{\text{dist}} = \text{FID}_{\text{diag}}(\mu_r, \sigma_r;\, \mu_{-j}^{(t+1)}, \sigma_{-j}^{(t+1)}) - \text{FID}_{\text{diag}}(\mu_r, \sigma_r;\, \mu^{(t+1)}, \sigma^{(t+1)})\]

A positive reward \(r_j^{\text{dist}} > 0\) is assigned if and only if omitting sample \(j\) degrades (increases) the global FID—meaning sample \(j\) pulled the generator distribution closer to the real data distribution. Samples that crowd already overpopulated modes or present low-density outliers yield non-positive rewards, directly penalizing mode collapse and encouraging coverage of underrepresented modes.

2. Composite instance-level perceptual rewards: guaranteeing semantic accuracy and perceptual fidelity

While the distribution-level reward guides global statistical coverage, it does not guarantee that any single generated image accurately adheres to its specific conditioning prompt. The framework pairs the distribution reward with a composite instance-level reward \(r_j^{\text{inst}} = r_j^{\text{clip}} \cdot r_j^{\text{hps}}\).

Specifically, \(r_j^{\text{clip}}\) computes the cosine similarity between the CLIP image embedding and the text embedding for prompt "a photo of {class-name}", enforcing semantic correctness. Concurrently, \(r_j^{\text{hps}}\) evaluates the image with Human Preference Score v2 (HPSv2), a regression model trained on large-scale human visual preference datasets, penalizing structural visual artifacts and rewarding photorealistic texture. The complete sample reward is the product \(r_j = r_j^{\text{clip}} \cdot r_j^{\text{hps}} \cdot r_j^{\text{dist}}\). Within each sampling group of size \(G\), advantages are standardized as \(A_j = (r_j - \bar{r}) / (s_r + \varepsilon)\), steering the policy toward generations that are simultaneously realistic, aesthetic, and distributionally diverse.

3. Adaptive closed-loop entropy regularization: dynamically stabilizing exploration and preserving token diversity

During multi-objective policy updates over large vocabularies (\(K = 16384\)), autoregressive softmax distributions tend to peak prematurely at high-frequency tokens, precipitating sudden drops in entropy and irrevocable diversity loss. Standard static or monotonically decaying entropy schedules cannot react to reward-driven collapse events.

The method introduces a dynamic closed-loop entropy controller. For each sequence, average token entropy is computed as \(H_t = -\sum_{v=1}^K \pi_\theta(x_t=v \mid s_t) \log \pi_\theta(x_t=v \mid s_t)\) and normalized by the maximum possible entropy \(H_{\max} = \log K\) to obtain the fraction \(\hat{H} = \frac{1}{T}\sum_{t=1}^T H_t / H_{\max}\). Given a target entropy level \(\hat{H}_{\text{target}}\) and a deadband \(\delta\), the effective regularization coefficient adjusts dynamically around a base cosine schedule \(c_{\text{sched}}(p)\):

\[c_{\text{eff}} = \text{clip}\left(c_{\text{sched}}(p) \cdot e^{k(\hat{H}_{\text{target}} - \hat{H})},\, c_{\min},\, c_{\max}\right)\]

When measured entropy dips below the target threshold, \(c_{\text{eff}}\) increases exponentially, applying a stronger exploration bonus to counteract collapsing token distributions. When entropy is safely above target, the coefficient relaxes, permitting focused policy convergence without destabilizing sampling.

Loss & Training

During training, the sequence-level importance sampling ratio is computed over all \(T\) steps: \(\rho_j(\theta) = \prod_{t=1}^T \frac{\pi_\theta(x_{j,t} \mid c, x_{j,<t})}{\pi_{\theta_{\text{old}}}(x_{j,t} \mid c, x_{j,<t})}\). The clipped surrogate objective is given by:

\[\mathcal{L}_{\text{GRPO}}(\theta) = -\frac{1}{G}\sum_{j=1}^G \min\left(\rho_j(\theta)A_j,\; \text{clip}(\rho_j(\theta), 1-\epsilon, 1+\epsilon)A_j\right)\]

To restrict excessive drift from the initial model, an approximate KL divergence penalty \(D_{\text{KL}}(\pi_\theta \parallel \pi_{\text{ref}})\) is added. Together with the adaptive token entropy bonus, the total training loss optimized across parameters \(\theta\) is:

\[\mathcal{L}(\theta) = \mathcal{L}_{\text{GRPO}}(\theta) + \beta D_{\text{KL}}(\pi_\theta \parallel \pi_{\text{ref}}) - c_{\text{eff}}\frac{1}{T}\sum_{t=1}^T H_t\]

The entire optimization requires only 600 iterations on 4 NVIDIA A100 GPUs (approximately 10–14 wall-clock hours), providing an extraordinarily lightweight post-training phase.

Key Experimental Results

Main Results

On the ImageNet 256×256 class-conditional benchmark evaluated over all 50,000 validation images, the method was tested across LlamaGen backbones (B, L, XL) with Classifier-Free Guidance (CFG scale 1.5, Table 1) as well as under completely unguided generation (CFG scale 1.0, Table 2).

Table 1: Performance comparison across LlamaGen backbones with Classifier-Free Guidance (CFG=1.5, original paper Table 1)

Model	FID ↓	IS ↑	CLIPScore ↑	Precision ↑	Recall ↑
LlamaGen-B (baseline)	7.06	119.52	0.2262	0.74	0.61
LlamaGen-B + ours	6.31	163.37	0.2347	0.82	0.54
LlamaGen-L (baseline)	4.64	196.78	0.2350	0.78	0.63
LlamaGen-L + ours	3.83	215.90	0.2360	0.79	0.63
LlamaGen-XL (baseline)	3.96	187.75	0.2346	0.74	0.67
LlamaGen-XL + ours	3.82	195.58	0.2370	0.77	0.67

Table 2: Performance comparison without Classifier-Free Guidance (No CFG, scale=1.0, original paper Table 2)

Model	FID ↓	IS ↑	CLIPScore ↑	Precision ↑	Recall ↑
LlamaGen-B (No CFG)	20.89	47.96	0.2082	0.47	0.46
LlamaGen-B + ours (No CFG)	8.91	112.83	0.2304	0.78	0.54
LlamaGen-L (No CFG)	10.24	82.38	0.2200	0.53	0.48
LlamaGen-L + ours (No CFG)	5.12	143.32	0.2325	0.77	0.63
LlamaGen-XL (No CFG)	12.66	78.02	0.2192	0.60	0.74
LlamaGen-XL + ours (No CFG)	4.55	148.16	0.2305	0.74	0.68

Ablation Study

Table 3: Comparison with AR-GRPO and reward ablations on LlamaGen-L (CFG=1.5, original paper Table 4 & Table 8)

Configuration / Method	Reward Components	FID ↓	IS ↑	Precision ↑	Recall ↑
LlamaGen-L (pretrained)	-	4.64	196.78	0.78	0.63
Instance-only ablation	CLIP only	6.00	217.54	-	-
Instance-only ablation	HPSv2 only	7.30	176.73	-	-
Instance-only ablation	CLIP + HPSv2	6.21	220.28	-	-
AR-GRPO baseline (w/o dist. reward)	CLIP + HPSv2	6.37	223.95	0.82	0.56
Ours (Full model)	CLIP + HPSv2 + LOO-FID	3.83	215.90	0.79	0.63
AR-GRPO baseline (+MANIQA)	CLIP + HPSv2 + MANIQA	9.34	147.30	0.70	0.59
Ours (+MANIQA)	CLIP + HPSv2 + MANIQA + LOO-FID	6.43	175.34	0.74	0.64

Table 4: Diagonal vs. Full-Covariance LOO-FID runtime and fidelity (original paper Table 10)

Covariance Formulation	Feature Dimension	LOO-FID Time per Step (ms)	FID ↓	IS ↑
Diagonal LOO-FID (PCA)	128	~30 ms	4.30	224.16
Full-covariance LOO-FID (PCA)	128	~550 ms	4.13	209.45
Diagonal LOO-FID (No PCA, Default)	2048	~30 ms	3.83	215.90

Key Findings

• Distribution-level reward is essential to prevent mode collapse: When fine-tuning with instance-level rewards alone (Table 8 & Table 4), Inception Score increases, but FID degrades dramatically from 4.64 to 6.21 or 7.30, and Recall plummets to 0.56 in AR-GRPO. Adding LOO-FID drives FID down to 3.83 while preserving full Recall (0.63), DreamSim (0.53), and LPIPS (0.73) diversity matching the un-tuned model.
• Unprecedented no-CFG gains deliver a 2x inference speedup: Without CFG guidance, the method slashes LlamaGen-XL FID from 12.66 down to 4.55 and boosts IS from 78.02 to 148.16. In fact, unguided tuned models rival or outperform standard models using CFG 1.5 (FID 4.55 vs 4.64 / 3.96), allowing practitioners to bypass CFG's unconditional forward pass and cut sampling compute in half.
• Diagonal covariance provides an optimal speed-accuracy trade-off: Computing full covariance matrices, even in a reduced 128-D PCA space, inflates reward computation time by ~18x (550 ms vs 30 ms). The diagonal approximation scales linearly (\(O(D)\)), maintaining a negligible ~30 ms footprint directly in the native 2048-D feature space and delivering the best overall FID (3.83).

Highlights & Insights

• Elegant sample attribution via EMA LOO-FID: Directly optimizing distribution-level metrics in mini-batch RL has historically failed due to high variance and lack of per-sample credit assignment. Formulating the marginal gain through an EMA-smoothed Leave-One-Out delta elegantly solves sample-level credit assignment for a global distributional objective.
• Intrinsic policy rectification versus CFG crutches: Classifier-Free Guidance artificially sharpens low-probability sampling trajectories during inference, which acts as a band-aid for flawed base model distributions. By directly shaping the underlying AR token policy through RL, the model achieves clean, photorealistic samples autonomously without CFG.
• Orthogonality and broad architectural generality: The method works seamlessly across both VQGAN and multi-scale LlamaGen transformers with identical hyper-parameters, consistently outperforming standard MLE even when the latter is trained for 5000 additional iterations. Because the distributional reward is optimizer-agnostic, it can be combined with token-pruning or advantage-reweighting variants.

Limitations & Future Work

• Reliance on pretrained visual extractors and reward priors: LOO-FID is computed in Inception-V3 latent space, inheriting any representation blind spots or inductive biases of that network; similarly, CLIP and HPSv2 models carry known demographic and stylistic preferences.
• Evaluation restricted to class-conditional benchmarks: Quantitative validation was performed exclusively on ImageNet 256×256 class-conditional generation, leaving open-ended, complex composition text-to-image prompts and ultra-high-resolution generation unexplored.
• Future directions: Adapting LOO distribution rewards using richer multimodal vision backbones (e.g., DINOv2 or Vision-Language token spaces), expanding credit assignment to token-level granularities, and applying the framework to video or 3D autoregressive generation.

Related Work & Insights

• vs AR-GRPO: AR-GRPO applies GRPO directly to AR models with instance-only rewards, inevitably trading distributional coverage for inflated single-image metrics (FID worsens to 6.37, Recall drops to 0.56). This paper's LOO-FID and adaptive entropy regularization restore full coverage while reaching a superior FID of 3.83.
• vs VA-\(\pi\): VA-\(\pi\) leverages variational pixel-aware policy alignment, achieving 7.65 FID on unguided LlamaGen-XL. By contrast, this work reaches 4.55 FID without CFG and 3.82 with CFG (vs VA-\(\pi\)'s 6.62), underscoring the power of explicit distributional constraints.
• vs Diffusion RL fine-tuning (DDPO / Diffusion-DPO): Unlike diffusion models that require backpropagating through iterative, continuous stochastic trajectories and frequently collapse to homogenous modes, discrete AR generation provides a clean MDP where token-level exploration and global distribution rewards can be harmonized stably.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Pioneering introduction of EMA Leave-One-Out distribution-level FID rewards combined with adaptive entropy in autoregressive RL tuning.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive coverage of multiple architectures (LlamaGen-B/L/XL, VQGAN), extensive CFG vs no-CFG evaluations, human preference testing, and rigorous runtime/covariance ablations.
• Writing Quality: ⭐⭐⭐⭐⭐ Highly coherent mathematical formulation, clear narrative, and crisp ablation motivations.
• Value: ⭐⭐⭐⭐⭐ Solves the longstanding diversity collapse bottleneck in generative RL while providing a practical recipe for halving inference latency via CFG elimination.

Related Papers

• [ECCV 2026] DivRL: Disentangled Self-Similarity Rewards for Diverse Subject-Driven Generation
• [ECCV 2026] RubricRL: Simple Generalizable Rewards for Text-to-Image Generation
• [ECCV 2026] RealGen: Photorealistic Text-to-Image Generation via Detector-Guided Rewards
• [ECCV 2026] Distribution Matching Distillation Meets Reinforcement Learning
• [ECCV 2026] Revisiting Autoregressive Models for Generative Image Classification