Revisiting Autoregressive Models for Generative Image Classification

Conference: ECCV 2026
arXiv: 2603.19122
Code: https://github.com/yandex-research/ar-classifier
Area: Image Generation
Keywords: Generative classifier, autoregressive model, any-order modeling, order marginalization, diffusion model

TL;DR

This paper revisits the potential of autoregressive (AR) models for generative image classification, identifying that a fixed token order is the key bottleneck limiting AR classifier performance. It proposes using an any-order AR model (RandAR) to average the class-conditional log-likelihoods over multiple random token orders via the Jensen lower bound (order marginalization). This approach comprehensively outperforms diffusion classifiers on ImageNet and various out-of-distribution (OOD) benchmarks while being 25x faster in inference, bringing generative classifiers to a competitive level with DINOv2 for the first time.

Background & Motivation

Generative models have recently shown powerful capabilities in approximating complex visual data distributions, raising a critical question: can they directly function in discriminative tasks? Generative classifiers (GCs) derive the posterior \(p(c|x)\) from the class-conditional likelihood \(p(x|c)\) via Bayes' theorem, offering appealing properties such as avoiding shortcut learning and exhibiting a shape bias closer to human perception.

However, prior research on GCs has almost exclusively focused on Diffusion Classifiers (DCs). While DCs yield impressive accuracy, they suffer from a major drawback: estimating the ELBO of a single image requires 100 to 250 model forward passes, leading to extremely slow inference. In contrast, AR models require only a single forward pass to calculate the log-likelihood of the entire sequence, which is two orders of magnitude more efficient. Nevertheless, fixed-order (raster-scan) AR classifiers yield accuracy far inferior to DCs, causing them to be long neglected.

This paper uncovers the root cause of this performance gap: the fixed token order imposes an excessively strong inductive bias on image understanding. Under a raster-scan order (left-to-right, top-to-bottom), AR models predict tokens step-by-step by relying heavily on discriminative clues from locally observed neighboring tokens, lacking a grasp of global image structure. Analysis experiments clearly demonstrate this phenomenon: the same image is correctly classified under some token orders but misclassified under others; the discriminative log-likelihood heatmap of a single-order prediction only highlights local regions of an object, whereas the complete contour of the object is revealed only after averaging multiple orders (\(K=20\)).

The key challenge is thus clarified: single-order prediction relies on local clues and is insufficient, but enumerating all possible orders is intractable. The key insight of this work is to leverage an Any-Order AR (AO-AR) model, RandAR, which is explicitly conditioned on token positions via position prompt tokens, naturally supporting generation and likelihood evaluation under any order. Building on this, averaging the predictions of \(K\) random orders (order marginalization) achieves the benefits of multi-perspective reasoning while keeping the inference cost within \(K\) forward passes (\(K \approx 20\)), which is still an order of magnitude faster than diffusion classifiers.

Core Idea: Use an any-order AR model to evaluate class-conditional log-likelihoods over multiple random token orders, aggregating them into an order-marginalized prediction via the Jensen lower bound. This achieves accuracy superior to a diffusion classifier with 250 forward passes using only about 20 forward passes, leading to a 25x speedup in inference.

Method

Overall Architecture

The goal of this method is to estimate the order-marginalized class-conditional log-likelihood \(\log p(x|c)\) across all classes \(c\) for a given image \(x\), and then predict the class via the argmax operation using Bayes' theorem. The entire pipeline consists of the following steps: a VQ-VAE encodes a \(256 \times 256\) image into 256 discrete tokens; each token is paired with a position prompt token to generate \(K\) randomly permuted sequences; a class-conditional token is appended to the end of each sequence (one sequence per class for \(M\) classes, shared across the same set of \(K\) orders for fairness); RandAR performs a single forward pass for each (order, class) pair to output the order-conditional log-likelihood; finally, the likelihoods across \(K\) orders are aggregated via the Jensen lower bound, and the argmax is taken to predict the class.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image<br/>256×256"] --> B["VQ-VAE Tokenization<br/>Image → 256 Discrete Tokens"]
    B --> C["Any-Order AR Modeling<br/>Position Prompts + Random Permutations × K → Order-Conditional Likelihood"]
    C --> D["Order-Marginalized Classification<br/>Jensen Lower Bound Aggregation of K Orders → Class log p(x|c)"]
    D --> E["argmax Predict Class<br/>c* = argmax log p(x|c_i)"]

Key Designs

1. Any-Order Autoregressive Modeling: Liberating AR Models from Fixed Token Orders

Traditional AR image models (e.g., LlamaGen, VAR) perform next-token prediction by arranging image tokens in a fixed raster-scan order, which is equivalent to forcing the model to "read" the image in a single direction. This becomes a performance bottleneck in classification tasks—the model can only infer class information within the local context of observed tokens, lacking the ability to comprehensively understand the image from multiple perspectives. Experiments support this: the per-token discriminative log-likelihood of the raster-scan model only highlights partial regions of an object, which is far less complete than the multi-order average.

This paper adopts RandAR as the base model. Its core mechanism involves introducing a set of position prompt tokens \(P = \{p_1, \ldots, p_N\}\) alongside the standard image token sequence \(x = \{x_1, \ldots, x_N\}\), forming an interleaved sequence \([p_1, x_1, \ldots, p_N, x_N]\). By applying a random permutation \(\pi\) to the token indices, the rearranged sequence \([p_1^{\pi(1)}, x_1^{\pi(1)}, \ldots, p_N^{\pi(N)}, x_N^{\pi(N)}]\) is obtained, where the position prompt token explicitly informs the model which spatial position of the original image the current token to be predicted belongs to. The order-conditional likelihood of RandAR is defined as:

\[p(\mathbf{x}|\pi,c) = \prod_{n=1}^{N} p(\mathbf{x}_n^{\pi(n)} | \mathbf{p}_1^{\pi(1)}, \mathbf{x}_1^{\pi(1)}, \ldots, \mathbf{x}_{n-1}^{\pi(n-1)}, \mathbf{p}_n^{\pi(n)}, c)\]

The key to this design is that the position prompt tokens decouple the "generation order" from the "spatial location," enabling the model to learn to reasonably predict the next token under any sequence sequence. When \(\pi\) is the identity map, RandAR degenerates into a standard raster-scan AR model; when \(\pi\) is randomly sampled, each order provides a different observation path, laying the foundation for subsequent multi-order aggregation. Notably, the accuracy of a single random order (\(K=1\)) is \(0.670\), which is lower than the raster-scan order (\(0.701\)). This is because fitting an arbitrary order is slightly harder for the model than fitting a single order; the gain of order marginalization comes from the complementary information of multiple different orders.

2. Order-Marginalized Classification: Efficiently Aggregating Multi-Order Signals with the Jensen Lower Bound

With any-order modeling capabilities, the core problem is how to leverage predictions from multiple orders to obtain more robust classification results. The order-unconditioned likelihood can be written as the expectation over all possible permutations \(p(x|c) = \mathbb{E}_{\pi}[p(x|\pi,c)]\), but directly computing this expectation is intractable. This work identifies a key practical design choice that determines performance: using the Jensen lower bound to estimate \(\log p(x|c)\), rather than directly estimating \(p(x|c)\) and then taking the log:

\[\log p(\mathbf{x}|c) = \log \mathbb{E}_{\pi}[p(\mathbf{x}|\pi,c)] \geq \mathbb{E}_{\pi}[\log p(\mathbf{x}|\pi,c)] \approx \frac{1}{K}\sum_{k=1}^{K}\log p(\mathbf{x}|\pi_k,c)\]

This choice is not a minor theoretical detail. Experiments show that the Jensen lower bound (averaging \(\log p\)) yields significant and continuous accuracy improvements as \(K\) increases (IN-Val: from \(0.712\) at \(K=1\) to \(0.813\) at \(K=20\)), whereas the gain from directly estimating \(\log \mathbb{E}[p]\) is much smaller (only \(0.760\) at \(K=20\), a gap of 5.3 percentage points). The reason is that RandAR optimizes the log-likelihood objective during training, meaning the model's estimation of \(\log p(x|\pi,c)\) is naturally more accurate, aligning the lower bound formulation with the training objective. Similarly, in diffusion classifiers, using the training objective (uniform weighting ELBO) for inference is also far superior to the true ELBO estimate.

During classification, the conditional tokens for the \(M\) classes are appended to the end of the \(K\) permuted sequences, respectively. RandAR performs one forward pass for each (order, class) combination, obtaining \(M \times K\) order-conditional log-likelihoods, which are then aggregated by class into \(M\) values of \(\log p(x|c_i)\), followed by taking the argmax. The efficiency advantage of the AR model is fully reflected here: each (order, class) pair requires only a single forward pass, whereas the diffusion model requires a forward pass for each (timestep, class) pair. When \(M=1000, K=20\) vs. \(T=250\), the total number of forward passes is \(20K\) for AR compared to \(250K\) for diffusion, making the former approximately 12.5 times faster. Furthermore, the authors prove that the order-marginalization objective of AO-AR is equivalent in expectation to the ELBO of Masked Diffusion (MD). However, the AR evaluation method consistently outperforms the MD evaluation given the same \(K\) (since a single AR forward pass covers the entire set of token likelihoods, while MD only estimates the token at a specific prefix length in each step).

3. Latent Noise Augmentation for Tokenizers: Preventing Minor Perturbations from Altering Token Sequences

The VQ-VAE tokenizer maps continuous images into discrete token sequences, but this mapping can be unstable under minor perturbations. An almost identical image with a slight offset in the latent space may produce different discrete token sequences. This is unfavorable for AR model training, as the model faces different token sequences for essentially the same image, increasing unnecessary modeling difficulty.

This paper systematically compares the noise robustness of two tokenizers: MaskGIT (codebook size 1024) and LlamaGen (codebook size 16384). By applying a flow-matching noise process \((1-t)z + t\epsilon\) (\(\epsilon \sim N(0,I)\)) to the continuous latent variables before quantization, the relationship between the "proportion of flipped tokens" and rFID is tracked. The results show that the large-codebook LlamaGen can tolerate approximately 27% token flipping while rFID only increases from 1.68 to 1.69. In contrast, the small-codebook MaskGIT sees its rFID rise from 1.67 to 1.71 at around 10% flipping. This suggests that the larger codebook provides more redundancy to absorb latent perturbations.

Based on this observation, the authors propose applying moderate noise to the latent representations of the LlamaGen tokenizer during RandAR training as a data augmentation technique, teaching the model to ignore minor token fluctuations that do not alter the image semantics. Experiments show that this augmentation bridges the in-domain accuracy gap of LlamaGen relative to MaskGIT (IN-Val: from \(0.769\) to \(0.780\)) while maintaining its robustness advantages on all OOD benchmarks.

Loss & Training

RandAR is trained from scratch on ImageNet-1K using the standard next-token prediction cross-entropy loss, without any additional discriminative objectives or contrastive learning alignment. The optimizer is AdamW, with the learning rate warmed up over 40k steps to a peak of \(6 \times 10^{-4}\) followed by a cosine decay (500k iterations in total). RandAR-L uses a batch size of 512, and RandAR-XL uses 256. Random cropping and latent noise augmentation (only for the LlamaGen tokenizer) are applied during training. The classification evaluation phase uses \(K=20\) random orders (employing the same random order sampling strategy as training), without any fine-tuning or extra alignment steps.

Key Experimental Results

Main Results

RandAR is compared with baselines including discriminative models, diffusion, AR, and joint energy models on ImageNet-Val and five OOD benchmarks (IN-R/S/A, IN-C Gauss/JPEG). All models are pre-trained on ImageNet-1K \(256 \times 256\), with architecture size and token counts aligned. Evaluation is conducted on a 2K subset (IN-Val/IN-S/IN-C) or the full set (IN-R/IN-A, 200 classes).

Model Scale	Method	Type	IN-Val	IN-R	IN-S	IN-A
L/16	ViT	Discriminative	0.803	0.409	0.291	0.166
L/16	DINOv2	Discriminative SSL	0.819	0.476	0.358	0.363
L/16	DiT	Diffusion	0.771	0.393	0.361	0.133
L/16	LlamaGen	AR (Fixed Order)	0.640	0.298	0.232	0.143
L/16	VAR	AR (Multi-Scale)	0.656	0.255	0.177	0.083
L/16	A-VARC+	AR + Noise Avg	0.717	0.277	0.175	0.072
L/16	RandAR raster	AR (Fixed Order)	0.701	0.351	0.301	0.174
L/16	RandAR (Ours)	AR (Any Order)	0.780	0.463	0.406	0.145
XL/16	DINOv2	Discriminative SSL	0.827	0.486	0.354	0.345
XL/16	DiT	Diffusion	0.772	0.402	0.367	0.153
XL/16	SiT + REPA	Diffusion + SSL Alignment	0.733	0.296	0.262	0.169
XL/16	RandAR (Ours)	AR (Any Order)	0.813	0.530	0.459	0.233

RandAR-L outperforms DiT-L by 0.9 percentage points on IN-Val, with even larger margins on OOD benchmarks (IN-R +7.0%, IN-S +4.5%). RandAR-XL extends this lead further (outperforming DiT by 12.8% on IN-R and 9.2% on IN-S). Compared to DINOv2-XL, the in-domain gap is 1.4%, but RandAR-XL surpasses it on 3 out of 5 OOD sets (IN-R +4.4%, IN-S +10.5%, IN-C Gauss +15.2%), matches it on IN-C JPEG, and only falls behind on IN-A by 11.2%. Fixed-order AR methods (LlamaGen 0.640, VAR 0.656, RandAR raster 0.701) are comprehensively outperformed by RandAR with any-order modeling (0.780), demonstrating clearly that the performance gains stem from order marginalization rather than the model architecture itself.

Ablation Study

Likelihood Estimation Strategies (RandAR-XL, IN-Val):

Strategy \ \(K\)	1	2	5	10	20
Jensen Lower Bound \(\mathbb{E}[\log p]\)	0.712	0.762	0.795	0.804	0.813
Direct Estimation \(\log \mathbb{E}[p]\)	0.712	0.724	0.742	0.747	0.760

The Jensen lower bound yields a continuous and significant gain as \(K\) increases (an increase of 10.1 percentage points from \(K=1 \to 20\)), whereas the gain from direct estimation is substantially smaller (only +4.8 percentage points). The gap is 5.3 percentage points at \(K=20\), validating the hypothesis of "alignment with the training objective."

Tokenizer and Noise Augmentation (RandAR-L):

Tokenizer	Noise Augmentation	IN-Val	IN-R	IN-S	IN-A
MaskGIT (1024)	None	0.780	0.448	0.379	0.101
LlamaGen (16384)	None	0.769	0.469	0.409	0.146
LlamaGen (16384)	Yes	0.780	0.463	0.406	0.145

The large-codebook LlamaGen outperforms MaskGIT comprehensively in OOD robustness (leading by 4.5% on IN-A). Noise augmentation bridges its in-domain accuracy gap compared to MaskGIT, leading to the selection of the LlamaGen + noise augmentation configuration for all main experiments.

Key Findings

• Order marginalization is the primary source of gains, and random ordering itself is not free: The accuracy of a single random order (\(K=1\)) is \(0.670\), which is even lower than the raster-scan order of \(0.701\)—indicating that the model's capacity to fit arbitrary sequences is weaker than its capacity to fit a single sequence. The complementary effect of multiple orders only takes effect when \(K > 1\), surpassing fixed orders. At \(K=2\), accuracy reaches \(0.726\), already surpassing the \(0.701\) of the raster-scan order.
• Tokens in the middle positions are the most discriminative: The per-token accuracy first rises and then falls with prefix length, peaking at around 50–80 prefix tokens. At this point, the model has captured high-level semantics but is not yet "saturated" with local details, forcing it to generate class-defining details. Accuracy decreases for later tokens, as the model can rely on a large number of observed neighboring tokens instead of class-specific information.
• Overwhelming efficiency advantages: On the accuracy-runtime curve, RandAR outperforms DiT (with varying timestep numbers) at all operating points (varying \(K\) values), achieving a 25x faster inference under the optimal configuration. Even with \(K=20\) (20 forward passes), it remains more than 10x faster than a diffusion classifier with 250 forward passes, while offering higher accuracy.
• Generation quality does not equate to classification accuracy: RandAR-XL yields an FID of 2.34 vs. DiT-XL's FID of 2.27. Despite a slightly worse generation quality, RandAR-XL achieves significantly higher classification accuracy (\(0.813\) vs. \(0.772\)). A low FID primarily reflects perceptual fidelity rather than depth of class understanding.
• Strong robustness under real-world distribution shifts: On the WILDS benchmark (Camelyon17/CelebA/FMoW), RandAR overall outperforms DiT and all discriminative baselines on OOD sets. Notably, on Camelyon17 OOD, RandAR-L reaches \(0.783\), far exceeding DiT-L (\(0.586\)) and the best discriminative baseline (\(0.667\)).

Highlights & Insights

• The Jensen lower bound trick (as opposed to expectation estimation) holds significant practical value: Whether the average is taken inside or outside the logarithm is a seemingly minor implementation choice, yet it leads to an absolute accuracy difference of 5.3 percentage points at \(K=20\). This echoes the phenomenon in diffusion classifiers where "the training objective (uniform weighting) outperforms the true ELBO." Together, they reveal that generative classifiers should employ estimation formulations aligned with the training objective during inference, rather than striving for strict probabilistic inference.
• Position prompt tokens transform "order" from a limitation into a free parameter: While the token order of traditional AR models is a fixed implementation detail, RandAR converts it into a manipulable degree of freedom through explicit position modeling, enabling "multi-perspective observation." This concept can be transferred to other sequence-modeling tasks, such as changing frame order in video understanding, changing scan order in 3D point clouds, or even altering token order in language models for data augmentation.
• The OOD robustness advantages of generative classifiers are further validated and extended: RandAR not only defeats discriminative models on standard OOD benchmarks (IN-R/S/A/C) but also leads under the real distribution shifts of WILDS, showing a particularly pronounced advantage in worst-group accuracy. This supports the hypothesis that "density-based classification naturally avoids spurious correlations," carrying significant implications for safety-critical applications such as healthcare and autonomous driving.
• The formal equivalence between AO-AR and Masked Diffusion provides a bridge to unified understanding: The paper demonstrates that the order-marginalized AO-AR objective is equivalent in expectation to the MD-ELBO, though the AR evaluation mode proves far superior to MD in practice (especially at small \(K\)). This uncovers radical differences between the two evaluation paradigms under limited sample regimes, providing a theoretical foundation for designing more efficient hybrid inference strategies.

Limitations & Future Work

• Computational cost scales linearly with the number of classes: Each added class requires \(K\) extra forward passes. For ImageNet-1K (1000 classes), \(20K\) forward passes are required; while cheap individually, the total sheer volume is substantial. The authors propose distilling GCs into discriminative models in the future to balance efficiency and accuracy, presenting a pragmatic and promising direction.
• Consistently weaker than DINOv2 on IN-A: RandAR-XL obtains only \(0.233\) vs. DINOv2-XL's \(0.345\), a gap exceeding 11 percentage points. IN-A contains adversarial natural images, which may expose that AR models are highly sensitive to anomalous visual patterns; fine-grained texture information lost during discrete tokenization could act as a bottleneck in adversarial scenarios. Combining any-order AR with continuous representations (e.g., a diffusion tokenizer) can be explored to bridge this gap.
• Only one AO-AR architecture (RandAR) was verified: Although control experiments with RAR-B show that the gains of order marginalization are not limited to RandAR, currently only RandAR natively supports any-order generation, restricting exploration of the architecture space. More efficient AO-AR architectures (such as parallel decoding or adaptive order selection) could further improve the accuracy-efficiency trade-off.
• Adaptive order selection remains unexplored: Currently, \(K\) orders are sampled randomly and uniformly, but per-token accuracy analysis shows that the discriminative power of tokens at different positions varies drastically (center regions are much more discriminative than margins). Learning an image-adaptive optimal order or weighting the aggregation according to token importance could yield comparable or higher accuracy with a smaller \(K\). The success of adaptive timestep selection in diffusion classifiers [wang2025noise] provides a valuable reference for this direction.
• Computational cost hinders large-scale practical deployment: Although 25x faster than a diffusion classifier, it still requires \(20K\) forward passes for a 1000-class problem, which is far higher than the single forward pass of standard discriminative models. Distillation into a discriminative model is the outlet suggested by the authors, though whether the OOD robustness of the generative classifier can be preserved during distillation remains an open question.

Related Work & Insights

• vs. Diffusion Classifiers (DiT/SiT): Diffusion classifiers classify by estimating the ELBO over hundreds of timesteps, which is accurate but very slow (250 forward passes). This paper reveals the core methodological difference between AR and diffusion: AR uses token orders as the degree of freedom for marginalization, whereas diffusion uses noisy timesteps. While the order marginalization of AR approaches saturation at \(K=20\), the timestep marginalization of diffusion requires 250 steps to converge. This stems from the fact that a single token order already encodes rich image structural information, whereas a single-timestep noisy sample provides extremely limited information.
• vs. A-VARC+ (VAR-based GC): A-VARC+ is also based on the VAR architecture and enhances classification accuracy by averaging multiple noisy samples, but essentially remains stuck with a fixed multi-scale raster-scan order, leading to poor OOD performance (IN-A \(0.072\) vs. RandAR's \(0.145\), IN-S \(0.175\) vs. \(0.406\)). The key difference is that A-VARC+ averages "different noise samples under the same order," whereas this work averages "different orders"—the latter introduces far greater diversity of observation perspectives, which is the root cause of the performance leap.
• vs. Masked Diffusion (MaskGIT): The paper proves that the order-marginalization objective of AO-AR is equivalent in expectation to the MD-ELBO, allowing RandAR to be evaluated as an MD model. However, experiments demonstrate that the AR evaluation mode consistently outperforms MD across all \(K\) values (\(K=1\): \(0.586\) vs. \(0.348\); \(K=20\): \(0.708\) vs. \(0.662\)), because an AR forward pass covers the likelihood of all tokens simultaneously, while MD only estimates the token at a specific prefix length at each step. This suggests that the AR evaluation paradigm should be preferred in practical deployment.
• vs. Self-Supervised Discriminative Models (DINOv2): DINOv2 learns robust discriminative representations through large-scale pre-training and contrastive learning, maintaining a 1.4% advantage in in-domain accuracy. However, RandAR surpasses it in multiple OOD scenarios and comprehensively leads in worst-group accuracy, implying that generative modeling and contrastive learning may be complementary in terms of robustness. A direction worth exploring is using order-marginalized generative classification signals as auxiliary supervision for DINOv2 training or as an ensemble strategy during inference.

Rating

• Novelty: ⭐⭐⭐⭐ Systematically uncovers the vital role of token ordering for AR generative classifiers, pushing AR-GC to the state-of-the-art via order marginalization; while the core component RandAR was not proposed in this paper and the Jensen lower bound trick has precedent in diffusion classifiers, their integration and the depth of analysis are novel.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Covers 6 ImageNet variants, 3 WILDS real-shift datasets, 3 model scales (B/L/XL), 8 baseline categories (discriminative/SSL/diffusion/AR/JEM/MD/ResNet/REPA variants), tokenizer ablations, likelihood estimation strategy ablations, comprehensive efficiency analyses, error analysis, and theoretical verification of equivalence to MD—the experimental design is exceptionally comprehensive with closely linked ablations.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear structure with a complete logical chain of motivation \(\to\) analysis \(\to\) method \(\to\) experiments \(\to\) extensions. Includes 7 figures, 7 main tables, and rich visualizations in the appendix (per-token analysis, error cases, further ablations), featuring in-depth analysis and explicit explanations of "why" behind every experimental result.
• Value: ⭐⭐⭐⭐ Pushes AR generative classifiers to a new SOTA and renders them competitive with DINOv2 for the first time; the 25x acceleration takes a major step toward making GCs practical. The order marginalization concept could inspire AR models in other discriminative tasks such as image segmentation or anomaly detection. However, the necessity of distillation indicates that the absolute efficiency of pure GCs still remains a bottleneck.

Related Papers

• [ICML 2026] A Diffusive Classification Loss for Learning Energy-based Generative Models
• [ECCV 2026] Continuous Speculative Decoding for Autoregressive Image Generation
• [ECCV 2026] Obliviate: Erasing Concepts from Autoregressive Image Generation Models
• [AAAI 2026] Symmetrical Flow Matching: Unified Image Generation, Segmentation, and Classification with Score-Based Generative Models
• [CVPR 2026] Advancing Image Classification with Discrete Diffusion Classification Modeling