Logit Refiner: Improving Visual Autoregressive Models via Intra-Scale Dependency Modeling

Conference: ECCV 2026
Paper: ECCV Official
Project Page: https://compvis.github.io/logit-refiner/
Area: Image Generation
Keywords: Visual Autoregressive Models, Intra-Scale Dependency Modeling, Mean-Field Approximation, Image Generation, Logit Refiner

TL;DR

Visual Autoregressive Models (VAR) suffer from local spatial incoherence due to an implicit mean-field-style parallel decoding rule within scales; Logit Refiner remedies this by sequentially sampling tokens via a lightweight causal transformer over frozen backbone features, boosting generation fidelity with minimal training compute and modest inference overhead.

Background & Motivation

Scale-wise visual autoregressive modeling, spearheaded by Visual Autoregressive Modeling (VAR), has emerged as a compelling paradigm for generative vision. By structuring discrete image representations into a hierarchy of token maps at increasing spatial resolutions, VAR generates images coarse-to-fine while predicting all tokens within each scale in parallel. Compared to traditional token-by-token raster scanning, scale-wise autoregression collapses the sequence length from thousands of steps to dozens of steps, delivering exceptional efficiency, favorable scaling properties, and broad applicability across high-resolution text-to-image synthesis and unified multimodal architectures.

Despite accurate per-token marginal predictions, VAR generations frequently exhibit pronounced local spatial incoherence. Samples regularly display disjointed structural contours, mismatched high-frequency textures ("texture soup"), or implausible instance amalgamations—even when each individual token prediction is plausible in isolation. Curiously, expanding model capacity fails to resolve this flaw: even scaling the transformer backbone to 2 billion parameters leaves these structural discontinuities intact. This paradox reveals that the bottleneck does not stem from insufficient backbone representational capacity, but from an inherent limitation in the decoding rule itself. Within each scale, VAR decomposes the conditional joint probability into a product of independent per-token marginal distributions, which mathematically constitutes a naive mean-field-style factorization that ignores spatial dependencies between same-scale tokens.

A minimal \(2 \times 2\) checkerboard toy scenario makes this failure concrete: on a dataset containing only two valid alternating patterns, a next-scale model accurately predicts a 50/50 marginal probability for each pixel at the second scale. Yet independent sampling produces \(2^4 = 16\) joint outcomes, 14 of which are invalid checkerboards. Thus, accurate pointwise marginals do not yield coherent joint samples under independent decoding. Core idea: preserve the parallel feature extraction of the expensive backbone, and restore intra-scale spatial dependencies via a lightweight causal autoregressive Logit Refiner operating on frozen backbone representations, removing the mean-field approximation at minimal computational cost.

Method

Overall Architecture

Logit Refiner decouples scale-wise visual generation into two collaborative stages: global cross-scale context extraction and local intra-scale dependency refinement. The heavy pretrained VAR backbone is kept strictly frozen, executing a single parallel forward pass per scale using bidirectional self-attention to generate rich, contextualized hidden state representations for all token positions simultaneously. Subsequently, a lightweight causal autoregressive module (comprising only 2 transformer blocks) processes these representations in raster-scan order, sequentially sampling each token conditioned on both the backbone features and previous same-scale tokens.

Because the frozen backbone already incorporates all coarse-scale semantics and high-level structural constraints, the attached refiner does not need to build spatial context from scratch. Instead, it is tasked purely with modeling the fine-grained residual spatial dependencies discarded by independent sampling. This design achieves structural coherence while preserving the fast, parallelized backbone execution.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-Scale Discrete Token Input<br/>Generated scale sequence r_&lt;k"] --> B["Frozen VAR Backbone<br/>Single parallel pass extracting features h^(k)"]
    B --> C["Intra-Scale Autoregressive Factorization<br/>Conditioning on preceding same-scale tokens"]
    D["Residual Dependency Modeling Architecture<br/>Feature fusion with lightweight causal Transformer"]
    C --> D
    D --> E["Identity Initialization<br/>Zero-initialized projections for smooth warm start"]
    E --> F["Hierarchical Adaptive Inference Strategy<br/>Early-scale refinement with KV caching"]
    F --> G["Coherent High-Quality Output Image<br/>Eliminating texture soup and structural artifacts"]

Key Designs

1. Intra-Scale Autoregressive Factorization: Breaking the Mean-Field Assumption

Standard VAR models the token sequence \(\mathbf{r}_k = (r_1^{(k)}, \dots, r_{L_k}^{(k)})\) at scale \(k\) via a naive mean-field-style independence approximation: $\(p_\theta(\mathbf{r}_k \mid \mathbf{r}_{<k}) \approx \prod_{i=1}^{L_k} p_\theta(r_i^{(k)} \mid \mathbf{h}^{(k)})\)$ Because tokens are drawn independently, sampling decisions cannot coordinate across adjacent spatial locations, frequently yielding mismatched textures. Logit Refiner resolves this by substituting independent sampling with an autoregressive joint factorization across same-scale tokens: $\(q_\phi(\mathbf{r}_k \mid \mathbf{r}_{<k}) = \prod_{i=1}^{L_k} q_\phi\left(r_i^{(k)} \mid r_{<i}^{(k)}, \, \mathbf{h}_{\le i}^{(k)}, \, \mathbf{r}_{<k}\right)\)$ Tokens are ordered via standard raster scan (left-to-right, top-to-bottom). This formulation strictly generalizes the original VAR decoding rule: setting the refiner to ignore intra-scale history \(r_{<i}^{(k)}\) directly recovers the original mean-field decoder. By conditioning each token on previously sampled neighbors, the refiner restores spatial correlation and eliminates unnatural patch discontinuities.

2. Residual Dependency Modeling Architecture: Causal Add-On over Frozen Representations

To avoid retraining the base backbone or inducing prohibitive latency, the refiner is constructed as a minimal add-on module. For position \(i\) at scale \(k\), the input vector \(\mathbf{z}_i^{(k)}\) is assembled by concatenating the frozen backbone hidden state \(\mathbf{h}_i^{(k)} \in \mathbb{R}^w\) with an autoregressive context embedding \(\mathbf{c}_i^{(k)}\): $\(\mathbf{z}_i^{(k)} = \mathbf{W}_\text{proj} \left[ \, \mathbf{h}_i^{(k)} \, \parallel \, \mathbf{c}_i^{(k)} \, \right]\)$ where \(\mathbf{c}_i^{(k)} = \text{emb}_\phi(r_{i-1}^{(k)})\) is derived from the immediately preceding sampled token (with a learned \(\mathbf{z}_\text{sos}\) vector at the initial position). The fused features are passed through \(d_r = 2\) lightweight causal transformer blocks: $\(\tilde{\mathbf{h}}_i^{(k)} = \text{TransformerBlocks}_\phi\left(\mathbf{z}_{1:i}^{(k)}\right)\)$ The refined states are mapped to logits via an output head to sample the next token: \(\tilde{\boldsymbol{\ell}}_i^{(k)} = \text{head}_\phi(\tilde{\mathbf{h}}_i^{(k)})\), \(r_i^{(k)} \sim \text{Cat}(\text{softmax}(\tilde{\boldsymbol{\ell}}_i^{(k)}))\). Since the backbone's bidirectional attention has already contextualized \(\mathbf{h}_i^{(k)}\), \(d_r = 2\) blocks suffice to capture the residual dependencies, requiring only \(\sim 10\%\) extra parameters relative to the base model.

3. Identity Initialization: Zero-Distortion Warm Start and Residual Specialization

Naively initializing the refiner with random weights destabilizes the well-trained prior of the base VAR model and prolongs training. The authors devise an identity initialization strategy: the token embedding matrix and output classification head are copied directly from the pretrained backbone. The input projection matrix is initialized as \(\mathbf{W}_\text{proj} \leftarrow [\,\mathbf{I} \parallel \mathbf{0}\,]\), initially silencing the autoregressive context stream, while the output projections of attention and feedforward layers in each refiner block are zero-initialized.

Under this setup, the refiner's output distribution at step zero is functionally identical to the original VAR. Training loss begins from the stable base model checkpoint rather than an uncalibrated state, driving optimization to focus entirely on residual probability adjustments required for joint intra-scale coherence.

4. Hierarchical Adaptive Inference Strategy: Early-Scale Focus and KV Caching

While sequential autoregressive sampling incurs token-level recurrence, empirical inspection demonstrates that spatial structural coordination is predominantly established at coarse, low-resolution scales (\(1\times 1\) to \(8\times 8\)). At high resolutions (\(16\times 16\)), tokens primarily encode high-frequency texture details where independent sampling causes minimal perceptual damage.

Exploiting this hierarchy, the authors introduce an adaptive inference schedule: the refiner is engaged sequentially only at early scales (e.g., \(\le 8\times 8\) or \(\le 10\times 10\)), while higher scales revert to parallel sampling from the backbone head. Combined with standard KV caching for the causal transformer blocks, this selective refinement retains \(88\% \sim 99\%\) of the full FID improvements while slashing the refiner's latency overhead by \(71\% \sim 84\%\), establishing an optimal quality-throughput Pareto frontier.

Loss & Training

During refiner training, the VAR backbone parameters \(\theta\) remain strictly frozen; only the refiner parameters \(\phi\) are optimized. Training employs standard teacher forcing, substituting generated history \(r_{<i}^{(k)}\) with ground-truth tokens. Under causal attention masking, all scales and token positions are trained in parallel in a single forward pass without sequential unrolling. The optimization objective is standard cross-entropy across all scales: $\(\mathcal{L}(\phi) = - \sum_{k=1}^K \sum_{i=1}^{L_k} \log q_\phi\left(r_i^{(k)} \mid r_{<i}^{(k)}, \, \mathbf{h}_{\le i}^{(k)}, \, \mathbf{r}_{<k}\right)\)$ Refiner training requires only 30 epochs with a learning rate of \(2.5 \times 10^{-5}\) (compared to hundreds of epochs for base VAR pretraining), completing in just 66 H200 GPU hours (<5% of base pretraining compute).

Key Experimental Results

Main Results

Evaluated on class-conditional ImageNet \(256 \times 256\) using 50k generated samples across standard metrics (FID, Inception Score, Precision, Recall), Logit Refiner consistently surpasses vanilla VAR baselines and competing scale-wise architectures:

Model Architecture	Params	FID↓	IS↑	Precision↑	Recall↑	FID Gain vs VAR
VAR-d16 (Baseline)	310M	3.30	274.4	0.84	0.51	-
MVAR-d16	310M	3.09	285.5	0.85	0.51	-
M-VAR-d16	464M	3.07	294.6	0.84	0.53	-
HMAR-d16	465M	3.01	288.6	0.84	0.55	-
VAR-d16 + Refiner (Ours)	356M	2.81	267.2	0.81	0.56	▼ 0.49
VAR-d20 (Baseline)	600M	2.57	302.6	0.83	0.56	-
HART-d20	649M	2.39	316.4	-	-	-
VAR-d20 + Refiner (Ours)	671M	2.17	274.7	0.80	0.60	▼ 0.40
VAR-d24 (Baseline)	1.0B	2.09	312.9	0.83	0.57	-
HART-d24	1.0B	2.00	331.5	-	-	-
VAR-d24 + Refiner (Ours)	1.1B	1.83	288.2	0.79	0.63	▼ 0.26
VAR-d30 (Baseline)	2.0B	1.92	323.1	0.82	0.58	-
HART-d30	2.0B	1.77	330.3	-	-	-
VAR-d30 + Refiner (Ours)	2.2B	1.76	319.4	0.80	0.62	▼ 0.16
ImageNet Validation Set	-	1.78	-	-	-	-

In open-vocabulary text-to-image synthesis at \(1024 \times 1024\) resolution with Infinity-2B, the refiner trained on FLUX-6M lifts human preference scores (HPSv3) from 9.79 to 9.91 across 12 diverse evaluation categories.

Ablation Study

To isolate whether performance gains stem from intra-scale autoregressive dependency modeling rather than added parameter capacity or extra training, controlled ablations were conducted on VAR-d16:

Configuration	Joint Modeling	Extra Params	Extra Epochs	FID↓	Core Finding
VAR-d16 Baseline	✗	0	0	3.30	Baseline mean-field ceiling
+ Additional Training	✗	0	30 ep	3.12	Extended training yields marginal gain
+ Parallel Refiner (Bidirectional Attn)	✗	+46M	30 ep	3.15	Added capacity with independent decoding is ineffective
+ AR Refiner (Causal Attn, Ours)	✓	+46M	30 ep	2.81	Causal joint dependency modeling drives the breakthrough

Architectural and training component ablations further demonstrate the parameter and compute efficiency of the refiner:

Ablation Dimension	Setting	Trainable Params	FID↓	Note
Refiner Depth \(d_r\)	\(d_r = 0\) (Linear proj + head only)	+8M	3.02	1-step autoregression already improves FID
	\(d_r = 1\)	+27M	2.85	1 causal block captures large gains
	\(d_r = 2\) (Default)	+46M	2.81	Optimal balance of quality and efficiency
	\(d_r = 4\)	+84M	2.82	Deeper refiners saturate
Trainable Components	Tune Head & Embeddings only	46M (Trained)	2.86	Frozen projections retain strong quality
	Freeze Backbone, Full Refiner (Default)	46M (Trained)	2.81	Highest compute ROI (66 H200-h)
	Joint Backbone + Refiner Fine-Tuning	356M (Trained)	2.72	Minor gain at double compute (127 H200-h)

Key Findings

• Gains Stem from Dependency Modeling, Not Capacity: An architecture-matched parallel refiner with bidirectional attention and independent sampling achieves an FID of only 3.15, whereas the causal AR refiner reaches 2.81, conclusively isolating joint within-scale sampling as the operative mechanism.
• Outperforming Models Twice the Size: VAR-d24 equipped with Logit Refiner (1.1B total parameters) attains an FID of 1.83, substantially outperforming the unrefined 2.0B VAR-d30 (FID 1.92).
• Diversity and Coverage Gains: Unlike truncation or guidance tuning tricks that trade recall for fidelity, Logit Refiner consistently increases Recall by \(0.04 \sim 0.06\) across all model sizes, proving that spatial coherence recovers previously discarded modes.
• Early-Scale Dominance: Scale ablation shows that omitting the refiner at scale \(2\times 2\) degrades FID from 2.81 to 2.98, whereas omitting it at \(16\times 16\) only shifts FID to 2.83, justifying early-scale selective refinement.

Highlights & Insights

• Incisive Theoretical Diagnosis: The authors identify that spatial artifacts in visual autoregressive models are not learning capacity failures, but artifacts of an implicit mean-field sampling rule, providing an elegant analytical foundation.
• Non-Invasive Plug-and-Play Design: Requiring no changes or retraining of base models, the refiner plugs onto existing VAR checkpoints and achieves state-of-the-art results with <5% additional training compute.
• Identity Initialization Scheme: Initializing input projections to \([I \parallel 0]\) and zero-initializing transformer block output projections prevents cold-start representation drift, ensuring rapid and stable convergence.

Limitations & Future Work

• Inference Latency Overhead: Despite having only 2 layers and KV caching, sequential token generation introduces non-negligible latency at higher resolutions compared to fully parallel decoding.
• Fixed Raster-Scan Ordering: Tokens are decoded strictly left-to-right and top-to-bottom, which may not align with semantic saliency or natural image formation dependencies.
• Non-Autoregressive Intra-Scale Alternatives: Exploring parallel non-autoregressive refinement (e.g., iterative masked prediction or continuous diffusion denoising) represents a promising path to eliminate sequential decoding overhead.

Related Work & Insights

• vs Vanilla VAR (Tian et al., 2024): VAR established next-scale visual autoregression but decoded within-scale tokens independently; Logit Refiner supplies the missing intra-scale dependency chain without altering the base backbone.
• vs M-VAR / MVAR (Ren et al., 2024; Zhang et al., 2026): These approaches modify the core backbone via decoupled attention or spatial Markovian conditioning, whereas Logit Refiner acts as an orthogonal external add-on that can be combined with these modified backbones.
• vs HART / FlowAR (Tang et al., 2024; Ren et al., 2024): While those models switch to hybrid discrete-continuous or flow-matching representations, Logit Refiner proves that pure discrete scale-wise models can close the fidelity gap simply by fixing the sampling rule.

Rating

• Novelty: ⭐⭐⭐⭐☆ [Clearly identifies the mean-field decoding limitation and presents an elegant intra-scale autoregressive correction]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive ablations isolating capacity from dependency modeling, scaling from 310M to 2B parameters, and validation on T2I]
• Writing Quality: ⭐⭐⭐⭐⭐ [Exceptionally structured narrative, clean mathematical formulation, and intuitive toy checkerboard grounding]
• Value: ⭐⭐⭐⭐⭐ [Highly practical plug-in approach that delivers substantial generation quality gains with minimal training overhead]