Generative Refinement Network for Visual Synthesis

Conference: ECCV 2026
Paper: ECCV Paper
Code: https://github.com/bytedance/GRN
Area: Image Generation
Keywords: visual generation, discrete tokenizer, autoregressive model, global refinement, hierarchical binary quantization

TL;DR

GRN introduces theoretically near-lossless Hierarchical Binary Quantization (HBQ) to resolve the discrete tokenization bottleneck and couples it with a global refinement mechanism inspired by human painting alongside entropy-guided adaptive sampling, eliminating autoregressive error accumulation while achieving complexity-aware adaptive-step visual synthesis.

Background & Motivation

The visual generation field has advanced rapidly, driven primarily by scaling diffusion and flow transformers. By progressively integrating trajectories along a learned velocity field that transports simple noise prior to the empirical data distribution, continuous flow models exhibit strong capabilities in synthesizing high-fidelity visual content. However, because they are optimized via mean squared error without explicit likelihood estimation, these models inherently lack adaptive-step capacity and rely on rigid sampling schedules with a predetermined number of steps. This uniform computational allocation expends identical compute on samples regardless of visual complexity, resulting in considerable waste on simple scenes.

Meanwhile, inspired by token-level likelihood estimation in large language models, autoregressive (AR) models have garnered extensive attention in visual synthesis. While naturally complexity-aware via predictive categorical likelihoods, conventional AR paradigms suffer from two critical bottlenecks: first, standard discrete visual tokenizers (such as VQ-VAE and FSQ) lag substantially behind continuous VAEs in reconstruction fidelity unless latent channel dimensions are drastically expanded; second, strictly causal prediction mechanisms—whether operating token-by-token or scale-by-scale—suffer from severe error accumulation, as initial sampling errors are irrevocably compounded across subsequent steps. Furthermore, even masked parallel generative models like MaskGIT permanently fix high-confidence tokens once selected, lacking any holistic mechanism for retroactive error correction.

To bridge this fundamental divide between the rigid computation of continuous diffusion and the error compounding and lossy tokenization of discrete autoregressive models, this paper proposes an intuitive paradigm inspired by human artwork. An artist does not paint strictly in an irreversible scanline or permanently lock early strokes, but rather sketches a rough composition, globally inspects the canvas, adds details, and erases or revises earlier mistakes. Core idea: achieve near-lossless discrete representation with exponential error decay via Haar-wavelet-inspired Hierarchical Binary Quantization (HBQ), and construct a global refinement autoregressive mechanism on this latent space that supports dynamic token erasure and entropy-guided adaptive-step scheduling.

Method

Overall Architecture

The GRN pipeline consists of two principal components: a unified 3D causal visual tokenizer equipped with Hierarchical Binary Quantization (HBQ) that compresses continuous image and video signals into compact discrete binary representations, followed by the Generative Refinement Network (GRN) operating over these discrete tokens.

During synthesis, the model begins with a completely random noise token canvas. At each refinement step, the input state combines the current drawing predictions and blank random tokens through a stochastically updated selection map. After processing the full canvas through a bidirectional Transformer, the model evaluates its predictive uncertainty via global average entropy to dynamically adjust the sampling progression, permitting new token additions, preservation of stable predictions, and explicit erasure and replacement of erroneous past tokens.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Input image / video X"] --> HBQ["Hierarchical Binary Quantization Tokenizer<br/>tanh activation + multi-round bisection"]
    HBQ --> Init["Initial canvas state F_0<br/>100% random noise tokens"]
    Init --> Transformer["Global Refinement Transformer<br/>predicts canvas distribution p(Y_{t+1})"]
    Transformer --> Entropy["Entropy-Guided Adaptive Sampling<br/>evaluates mean entropy and schedules l_t"]
    Entropy --> Update["State evolution update<br/>fill new tokens / keep stable / erase early errors"]
    Update -->|below termination step| Transformer
    Update -->|reaches adaptive termination| Final["Synthesized discrete token map"]
    Final --> Decoder["HBQ 3D Decoder reconstructs visual output"]

Key Designs

1. Hierarchical Binary Quantization (HBQ): near-lossless discrete tokenization with exponential error decay

Existing discrete tokenizers fail to rival continuous VAEs because codebook capacity is bounded, while scaling latent dimensions slows training convergence and inflates generative model size. Inspired by Haar Wavelet expansion in signal processing, HBQ formulates feature quantization as a multi-round binary search problem. A \(\tanh(\cdot)\) activation caps continuous latent features \(F\) within \([-1, +1]\). Each feature element then undergoes \(M\) rounds of hierarchical binary quantization based on dynamically computed bucket centers \(c_i\):

\[c_i = \sum_{j=1}^{i-1} \frac{\delta[q_j]}{2^j}, \quad q_i = \begin{cases} 0, & \text{if } F \le c_i \\ 1, & \text{if } F > c_i \end{cases}\]

where \(\delta[q_j] = 1\) when \(q_j = 1\) and \(-1\) when \(q_j = 0\). In each round \(j\), the quantization error bound is strictly guaranteed to be less than \(2^{-j}\), decaying exponentially with round count. Reconstructed features \(\hat{F} = \sum_{j=1}^M \delta[q_j] \cdot 2^{-j}\) feed directly into the decoder. This enables the discrete tokenizer to match or surpass continuous counterparts using 4 to 8 quantization rounds without expanding latent channel counts.

2. Global refinement mechanism: retroactive token erasure and holistic error correction

Standard autoregressive and masked generative architectures suffer from irreversible commitment: once tokens are placed, subsequent steps are compelled to build upon them, magnifying early flaws into severe artifacts. GRN defines the canvas state \(F_t\) at step \(t\) as a composite of current predictions \(Y_t\), uniform random noise tokens \(Y_{\text{rand}}\), and a binary selection map \(S_t\):

\[F_t = S_t \cdot Y_t \oplus \overline{S_t} \cdot Y_{\text{rand}}\]

where the proportion of ones \(l_t\) increases monotonically towards 100%. Crucially, \(S_t\) is sampled stochastically across the canvas rather than hard-thresholded on per-token certainty. Previously filled positions can be erased and reverted back to random noise or directly overwritten by newer global predictions. As broader semantic context develops across refinement steps, this mechanism allows the model to actively revise and correct premature decisions.

3. Entropy-guided adaptive sampling: complexity-aware dynamic computational allocation

Allocating fixed sampling steps imposes redundant compute on simple visual concepts while under-refining intricate compositions. GRN leverages discrete categorical output distributions to measure visual generation difficulty through normalized information entropy. The average entropy \(H(Y_t)\) across all \(N\) tokens at step \(t\) is formulated as:

\[H(Y_t) = -\frac{1}{N \log_2 K} \sum_{i=1}^N \sum_{j=1}^K p(y_{(i,j)} \mid F_{t-1}, \text{cond}) \cdot \log p(y_{(i,j)} \mid F_{t-1}, \text{cond})\]

To prevent instability during initial random states, an empirical warm-up period of \(t_0 = 5\) steps is established. Following warm-up, the slope of \(l_t\) and the allocated total steps adapt dynamically based on \(H(Y_{t_0+1})\): low entropy indicates high structural confidence, triggering an accelerated schedule that terminates in as few as 20 steps (or 10 steps in text-to-image); high entropy prompts extended refinement up to 50 steps. This dynamic allocation achieves an average \(1.25\times\) (up to \(2.5\times\)) speedup across datasets without fidelity degradation.

Loss & Training

The visual tokenizer is trained end-to-end using a combination of L1 reconstruction loss, LPIPS perceptual loss, and PatchGAN adversarial loss:

\[\mathcal{L}_{\text{tokenizer}} = \lambda_{\text{recons}} \mathcal{L}_{\text{recons}} + \lambda_{\text{LPIPS}} \mathcal{L}_{\text{LPIPS}} + \lambda_{\text{GAN}} \mathcal{L}_{\text{GAN}}\]

with loss weights configured to 1.0, 1.0, and 0.3 for the image tokenizer, and 1.0, 0.2, and 0.005 for the joint image-video tokenizer, using the Straight-Through Estimator (STE) for backpropagation.

For generative training, a selection ratio \(l_t \in [0, 1]\) is sampled uniformly per iteration. The canvas input \(F_t\) combines \(N \cdot l_t\) ground-truth tokens \(Y_{\text{gt}}\) and \(N \cdot (1 - l_t)\) uniform random tokens \(Y_{\text{rand}}\). The Transformer optimizes all token predictions simultaneously using standard cross-entropy loss:

\[\mathcal{L}_{\text{CE}} = - \mathbb{E} \left[ \frac{1}{N} \sum_{i=1}^N \log p(y_i \mid F_t, \text{cond}) \right]\]

supporting both categorical scalar index prediction (\(\text{GRN}_{\text{ind}}\), vocabulary size \(K=2^M\)) and bitwise binary prediction (\(\text{GRN}_{\text{bit}}\), \(K=2\)).

Key Experimental Results

Main Results

On the ImageNet \(256 \times 256\) class-conditional benchmark, GRN-G (2B parameters) establishes a state-of-the-art generation record of 1.81 gFID, outperforming top diffusion models DiT-XL/2 (2.27), flow models SiT-XL/2 (2.06), and leading autoregressive baselines VAR-d30 (1.92) and LlamaGen-XXL (2.34). In visual reconstruction, 4-round HBQ achieves 0.56 rFID, markedly surpassing continuous SD-VAE and modern discrete tokenizers.

Benchmark / Task	Model	Tokenizer Type	Parameters	Core Metric (FID / rFID / GenEval)	Quality Metric (IS / PSNR / LPIPS)
ImageNet Tokenizer Recon	SD-VAE	Continuous (C)	-	rFID: 0.87	PSNR: 24.08 / SSIM: 0.68
ImageNet Tokenizer Recon	VAR	Discrete (D)	-	rFID: 0.85	PSNR: 22.47 / LPIPS: 0.15
ImageNet Tokenizer Recon	LlamaGen	Discrete (D)	-	rFID: 2.19	PSNR: 20.79 / SSIM: 0.68
ImageNet Tokenizer Recon	HBQ (M=4, Ours)	Discrete (D)	-	rFID: 0.56	PSNR: 23.01 / LPIPS: 0.13
ImageNet 256×256 Generation	DiT-XL/2	Continuous (C)	675M	gFID: 2.27	IS: 278.2
ImageNet 256×256 Generation	SiT-XL/2	Continuous (C)	675M	gFID: 2.06	IS: 277.5
ImageNet 256×256 Generation	LlamaGen-XXL	Discrete (D)	1.4B	gFID: 2.34	IS: 253.9
ImageNet 256×256 Generation	VAR-d30	Discrete (D)	2B	gFID: 1.92	IS: 323.1
ImageNet 256×256 Generation	GRN-G (Ours)	Discrete (D)	2B	gFID: 1.81	IS: 299.0
GenEval Text-to-Image (T2I)	SD3 Medium	Continuous (C)	2B	Overall: 0.62	Two Obj: 0.74 / Color: 0.67
GenEval Text-to-Image (T2I)	Infinity	Discrete (D)	2B	Overall: 0.71	Two Obj: 0.85 / Pos: 0.49
GenEval Text-to-Image (T2I)	GRN (Ours)	Discrete (D)	2B	Overall: 0.76	Two Obj: 0.90 / Pos: 0.52
VBench Text-to-Video (T2V)	CogVideoX-5B	Continuous (C)	5B	Overall: 81.61	Quality: 82.75 / Semantic: 77.04
VBench Text-to-Video (T2V)	Lumos-1	Discrete (D)	3.6B	Overall: 78.30	Quality: 79.50 / Semantic: 73.50
VBench Text-to-Video (T2V)	GRN (Ours)	Discrete (D)	2B	Overall: 82.99	Quality: 84.41 / Semantic: 77.35

Ablation Study

The authors thoroughly benchmark the global refinement paradigm against conventional token-locking masking, compare index versus bitwise targets, and analyze entropy-guided adaptive sampling.

Configuration / Ablation	Mechanism Description	CFG / Temperature \(\tau\)	FID ↓	IS ↑ / Note
GRN Global Refinement (Refine)	Supports retroactive erasure and continuous revision	2.4 / \(\tau=1.23\)	3.63	285.5 (stable high-fidelity convergence)
Conventional Masking (Mask)	Fixed tokens locked permanently once sampled	2.4 / \(\tau=1.23\)	185.62	4.3 (generation collapse under identical hyperparameters)
Conventional Masking (Mask, Tuned)	High CFG and low sampling temperature grid-searched	8.0 / \(\tau=0.50\)	18.13	220.2 (still significantly inferior to Refine)
\(\text{GRN}_{\text{ind}}\)-B (130M)	Predict categorical scalar indices (\(K=2^M\))	2.4 / \(\tau=1.33\)	3.56	280.3 (index prediction slightly favored at 130M)
\(\text{GRN}_{\text{bit}}\)-B (130M)	Predict flattened binary bits (\(K=2\))	2.4 / \(\tau=1.23\)	3.63	285.5
\(\text{GRN}_{\text{ind}}\)-L (458M)	Predict categorical scalar indices (\(K=2^M\))	2.0 / \(\tau=1.30\)	2.64	314.8
\(\text{GRN}_{\text{bit}}\)-L (458M)	Predict flattened binary bits (\(K=2\))	1.9 / \(\tau=1.20\)	2.47	287.0 (bitwise prediction clearly superior at scale)
Fixed 50-step Sampling (Baseline)	All samples uniformly evaluated for 50 steps	-	3.56	1.0× baseline relative runtime
Entropy-Guided Sampling	Dynamically allocated between 20 and 50 steps	-	3.47	1.25× average speedup (up to 2.5×)

Key Findings

• Global refinement is essential to avoid autoregressive generation failure: When evaluated under identical sampling hyperparameters, enforcing permanent token immutability (as in MaskGIT) causes FID to collapse from 3.63 to 185.62. Even after extensive grid searching to lower temperature and inflate CFG, the masked baseline only achieves an FID of 18.13, demonstrating that retroactive error correction is the foundational mechanism allowing GRN to eliminate compounding errors.
• Bitwise prediction excels on larger models and temporal synthesis: While index prediction has a slight advantage at 130M parameters, bitwise prediction achieves superior FID (2.47 vs. 2.64) at 458M. In text-to-video generation, bit prediction delivers substantially cleaner spatio-temporal coherence and mitigates token aliasing artifacts, with global refinement overcoming traditional bit-independence assumptions.
• Entropy-guided scheduling achieves simultaneous gains in speed and fidelity: Over 97.9% of generated images require fewer than the 50-step maximum, with 6.3% converging reliably in only 20 steps. Crucially, preventing over-refinement on simple concepts improves final FID from 3.56 down to 3.47 while cutting inference latency by 1.25× on average.

Highlights & Insights

• Exponentially decaying quantization via Haar Wavelet principles: HBQ structures feature discretization as an iterative bisection process, bounding residual error below \(2^{-j}\) without widening the latent dimension, thereby resolving the long-standing fidelity disadvantage of discrete tokenizers relative to continuous VAEs.
• Artist-inspired iterative canvas refinement: Rather than committing irreversibly to sequential token choices, GRN treats early predictions as malleable drafts, providing a clean and unified training paradigm where retroactive erasure and correction emerge naturally.
• Native suitability for unified multimodal foundation models: By demonstrating that a purely discrete autoregressive framework can match or surpass leading continuous diffusion and flow models in image and video synthesis, GRN paves the way toward native, unified discrete-token architectures that bridge language understanding and visual generation without hybrid patchworks.

Limitations & Future Work

• Exploration at extreme foundation scales: Due to academic computational constraints, experimental validation was capped at 2B parameters, leaving the behavior of GRN at the 10B+ scale of contemporary frontier models to future exploration.
• Texture detail and motion distortion in video generation: The authors note that while the T2V model performs robustly on human-centric actions, complex dynamic physical scenes occasionally exhibit local distortions and loss of fine high-frequency details.
• Step distillation for real-time synthesis: Combining refinement-based training with step distillation represents a promising trajectory to compress the adaptive scheduling window into 4–8 steps for ultra-fast inference.

Related Work & Insights

• vs MaskGIT / BERT: MaskGIT iteratively locks tokens based on scalar confidence thresholds, permanently fixing early errors; GRN applies stochastic global updates, actively enabling the model to erase and rectify flawed intermediate tokens.
• vs VAR (Visual Autoregressive): VAR structures autoregression across hierarchical spatial scales, but remains causal and irreversible within and across scales; GRN refines directly across a uniform latent grid, avoiding inter-scale error propagation.
• vs Continuous Diffusion / Flow Transformers (DiT / SiT / Wan 2.1): Diffusion models operate on continuous spaces with fixed numerical ODE/SDE solver steps; GRN's discrete categorical formulation naturally yields an informational entropy metric that drives native, policy-free adaptive-step sampling.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Introduces mathematically elegant HBQ discrete tokenization and transforms visual autoregression into an iterative global refinement paradigm.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorously validated across image reconstruction, class-conditional generation, 1024p text-to-image, and 480p text-to-video with informative ablation studies.
• Writing Quality: ⭐⭐⭐⭐⭐ Crystal-clear motivation, self-contained formulations, and well-structured empirical discussions.
• Value: ⭐⭐⭐⭐⭐ Provides an important milestone for discrete-token visual generation, demonstrating that autoregressive models can overcome error accumulation and rigid compute allocation simultaneously.