Skip to content

title: >- [Paper Note] DiverseVAR: Balancing Diversity and Quality of Next-Scale Visual Autoregressive Models description: >- [ECCV 2026][Image Generation][Visual Autoregressive Models] Addresses per-prompt diversity collapse in text-conditioned visual autoregressive models by pairing condition-annealed noise injection on text embeddings with a training-free Scale-Travel latent rollback and refinement mechanism, setting the optimal diversity-quality Pareto frontier. tags: - ECCV 2026 - Image Generation - Visual Autoregressive Models - Generation Diversity - Latent Refinement date: 2026-09-19 content_hash: 045cca441940a80b

DiverseVAR: Balancing Diversity and Quality of Next-Scale Visual Autoregressive Models

Conference: ECCV 2026
Paper: ECCV Official Page
Code: https://diverse-var.github.io/
Area: Image Generation
Keywords: Visual Autoregressive Models, Generation Diversity, Latent Scale Rollback, Condition Annealing, Image Generation

TL;DR

DiverseVAR tackles the severe per-prompt diversity collapse in text-conditioned visual autoregressive models (VAR) by introducing condition-annealed noise injection to text embeddings coupled with Scale-Travel—a training-free multi-scale latent rollback and refinement scheme that substantially boosts sample diversity while preserving visual fidelity.

Background & Motivation

Visual autoregressive (VAR) models have emerged as compelling alternatives to diffusion and flow-matching models in visual generation. By shifting the generative objective from 1D raster-scan next-token prediction to multi-scale next-scale prediction over residual token pyramids, VAR naturally preserves 2D spatial locality and causal structure while generating tokens within each scale level in parallel. This paradigm has achieved parity with, and often superior sampling efficiency over, leading diffusion models, spurring large-scale scaling efforts across both text-to-image (e.g., Infinity, Switti) and text-to-video (e.g., InfinityStar) synthesis.

Despite these advances in visual fidelity and prompt alignment, text-conditioned VAR models suffer from a fundamental yet overlooked limitation: per-prompt diversity collapse. When supplied with a fixed text prompt, VAR models produce nearly identical samples across different random seeds, exhibiting repetitive layouts, identical poses, and homogeneous aesthetics. Conventional diversity-enhancing controls from language modeling or diffusion—such as adjusting nucleus sampling threshold \(p\), raising temperature \(\tau\), or scheduling classifier-free guidance (CFG) weights—yield marginal benefits or rapidly induce catastrophic distortions in VAR. While injecting Gaussian noise into text embeddings (condition-annealing) effectively loosens the conditioning signal's dominance to unlock macro-structural variation, it perturbs the latent feature manifold, leaving images marred by severe visual artifacts, mangled textures, and broken geometry.

The key tension lies in how to disrupt the premature semantic lock-in during early coarse scales without suffering permanent image degradation in the final output. The authors recognize that in next-scale prediction, early coarse scales anchor the global layout and semantic layout, whereas later fine scales construct high-frequency textures. The core idea is to establish a training-free diversify-then-refine inference framework: DiverseVAR first injects condition-annealed noise into text embeddings across coarse scales to trigger diverse global compositions, and subsequently executes Scale-Travel via the native multi-scale autoencoder to rewind the accumulated latent feature map back to an earlier clean coarse scale for pristine, noise-free autoregressive refinement.

Method

Overall Architecture

DiverseVAR is an inference-time, training-free framework designed specifically for the residual multi-scale token prediction paradigm of VAR. Given an input text prompt with embedding \(c\) and random noise seeds, the pipeline operates in two cascaded stages. During the initial coarse-scale generation, condition-annealing injects scheduled Gaussian noise into the text embedding, diluting the text condition's overwhelming dominance and allowing the causal transformer to branch into distinct structural trajectories. Once generation reaches an intermediate scale \(l\) where global semantics are stabilized, the accumulated feature map \(\mathbf{Z}_l\) is fed into the multi-scale encoder to rewind the representation back to a coarser target scale \(m\) (\(m < l\)), cleanly discarding corrupted fine-scale residual tokens. Finally, generation resumes from scale \(m+1\) to final scale \(K\) under unperturbed condition embeddings, synthesizing sharp, artifact-free textures.

The complete operational flow is illustrated below, mapping sequentially across condition-annealing, multi-scale accumulation, Scale-Travel rollback, and clean autoregressive refinement:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Text Prompt & Random Seed"] --> B["Condition-Annealing Noise Injection<br/>Perturb text embedding with schedule"]
    B --> C["Noisy Forward Next-Scale Generation<br/>Generate up to intermediate scale l to form Z_l"]
    C --> D["Scale-Travel Latent Rollback<br/>Multi-scale encode Z_l and rewind to scale m"]
    D --> E["Clean Autoregressive Scale Refinement<br/>Autoregressively generate from scale m+1 to K"]
    E --> F["Output High-Fidelity & Diverse Image/Video"]

Key Designs

1. Condition-Annealed Noise Injection: Breaking Early Semantic Monopolies for Diverse Compositions

Standard VAR integrates text conditioning via cross-attention layers and the <SOS> token. Because the conditioning signal acts with uniform potency across all scale stages, the initial residual tokens (e.g., \(1\times 1\) or \(2\times 2\)) immediately lock onto a deterministic visual prototype. After systematically evaluating CFG scheduling, <SOS> perturbation, and text-embedding noise injection, the authors find that perturbing text embeddings provides the most pronounced diversity boost. Specifically, during the first \(k_{\max}\) scales, the nominal condition embedding \(c\) is perturbed into \(\hat{c}_k\): $\(\hat{c}_k = \sqrt{1 - \alpha(k)}\, c + \sqrt{\alpha(k)}\, \epsilon, \quad \epsilon \sim \mathcal{N}(0, \mathbf{I})\)$ where \(\alpha(k)\) follows a cosine annealing schedule starting at maximum variance at \(k=1\) and decaying to zero by scale \(k_{\max}\). This perturbation prevents the model from collapsing into a single mode in early stages, resulting in varied scene compositions and object interactions, though direct continuation to the final scale \(K\) produces broken contours and degraded fidelity.

2. Scale-Travel Latent Rollback: Utilizing the Multi-Scale Autoencoder for Structural Rewinding

To eliminate visual artifacts without discarding the newly discovered semantic compositions, DiverseVAR capitalizes on the intrinsic invertibility of VAR's multi-scale representations. At an intermediate scale \(l\) (empirically set to \(l=8\), where macro-level layout is established), the model accumulates all predicted residual token grids into a composite feature map \(\mathbf{Z}_l = \sum_{i=1}^l \text{up}(\mathbf{r}_i, \mathbf{s}_K)\). Instead of applying an external denoising network, Scale-Travel passes \(\mathbf{Z}_l\) directly through VAR's pre-trained multi-scale encoder, downsampling and quantizing the residual feature differences only up to a chosen coarser scale \(m\) (\(m < l\)): $\(\tilde{\mathbf{r}}_k = \mathcal{Q}\left(\text{down}(\mathbf{Z}_l - \mathbf{Z}_{k-1}, \mathbf{s}_k)\right), \quad k = 1, \dots, m\)$ where \(\mathcal{Q}\) denotes vector quantization and \(\text{down}(\cdot, \mathbf{s}_k)\) denotes bilinear downsampling to grid dimension \(\mathbf{s}_k\). Discarding the corrupted finer tokens (\(m+1\) to \(l\)) while retaining the re-encoded coarse grid \(\tilde{\mathbf{r}}_{1:m}\) effectively rewinds the generative trajectory along the scale axis to a clean, low-resolution anchor.

3. Clean Autoregressive Scale Refinement: Seamless High-Fidelity Detail Reconstruction

With the cleaned, coarse-scale sequence \(\tilde{\mathbf{r}}_{1:m}\) serving as the new causal prefix, the conditioning noise is fully deactivated (\(\alpha(k)=0\), using the pristine prompt embedding \(c\)). The transformer then autoregressively predicts subsequent residual token grids from scale \(m+1\) up to the terminal scale \(K\): $\(\tilde{\mathbf{r}}_{m+1:K} \sim \prod_{k=m+1}^K p(\mathbf{r}_k \mid \tilde{\mathbf{r}}_1, \dots, \tilde{\mathbf{r}}_m, \mathbf{r}_{m+1}, \dots, \mathbf{r}_{k-1})\)$ Conditioned on uncorrupted embeddings and coarse structural constraints, the model leverages its learned prior to restore crisp contours, natural lighting, and coherent textures, thereby overcoming the diversity-quality trade-off induced by condition annealing.

Loss & Training

DiverseVAR is entirely training-free and operates solely at test time, requiring zero fine-tuning of the VAR transformer or VQ-VAE codebooks. Typical hyperparameters select \(k_{\max} \in \{2, 3, 4, 5, 6\}\) with maximum noise scale \(\alpha(1) \in \{0.5, 1.0\}\) under a cosine schedule. Scale-Travel initiates rollback at scale \(l = 8\), targeting a rollback destination of \(m \in \{4, 5, 6\}\).

Key Experimental Results

Main Results

The framework was comprehensively evaluated across text-to-image VAR architectures (Infinity and Switti) and text-to-video VAR architectures (InfinityStar-8B). Image generation was evaluated on MJHQ-30K and MS-COCO benchmarks using pairwise LPIPS distance (LPIPS-MPD \(\uparrow\)), Vendi Score (\(\uparrow\)), ImageReward (IR \(\uparrow\)), and Fréchet Inception Distance (FID \(\downarrow\)).

Model Configuration Dataset FID \(\downarrow\) ImageReward (IR) \(\uparrow\) LPIPS-MPD \(\uparrow\) Vendi \(\uparrow\) Note
Infinity (Base) MJHQ-30K 19.16 1.22 0.33 4.10 Original baseline; severe diversity collapse
Infinity + Cond. Anneal. MJHQ-30K 15.53 0.49 0.56 7.68 High diversity but dramatic quality collapse
Infinity + Scale-Travel (DiverseVAR) MJHQ-30K 15.28 1.08 0.48 6.07 Quality recovered (+120.4% IR vs Cond. Anneal.) with +45.5% MPD over base
Infinity (Base) MS-COCO 37.37 1.16 0.37 4.41 Original baseline
Infinity + Cond. Anneal. MS-COCO 22.51 0.39 0.58 8.13 Severe quality penalty (IR drops to 0.39)
Infinity + Scale-Travel (DiverseVAR) MS-COCO 28.97 1.04 0.52 6.68 Optimal balance on photorealistic benchmark
Switti (Base) MJHQ-30K 16.18 1.11 0.44 4.79 Baseline Switti
Switti + Cond. Anneal. MJHQ-30K 21.34 0.63 0.58 7.34 Only noise injection; FID degrades to 21.34
Switti + Scale-Travel (DiverseVAR) MJHQ-30K 17.30 0.93 0.56 6.68 Substantial diversity gain with strong quality preservation

Source: Original paper Table 1.

For text-to-video generation using InfinityStar on VBench benchmarks, DiverseVAR was applied to keyframe generation:

Model Setting VBench Diversity \(\uparrow\) VBench Quality Score \(\uparrow\) Note
InfinityStar (Base) 0.270 0.806 Identical camera motion and object layout
InfinityStar + Cond. Anneal. 0.627 0.794 High diversity but degraded temporal and visual quality
InfinityStar + DiverseVAR 0.487 0.804 +80.4% diversity over base with negligible quality drop (-0.002)

Source: Original paper Table 2.

Ablation Study: Alternative Test-Time Diversity Controls

The authors compared DiverseVAR against standard sampling alterations including temperature modulation (\(\tau\)), nucleus sampling (\(p\)), and CFG weight adjustments (\(\omega\)):

Strategy Sampling Setting LPIPS-MPD \(\uparrow\) Vendi \(\uparrow\) ImageReward (IR) \(\uparrow\) Observation & Behavior
Infinity (Temperature \(\tau\)) \(\tau = 0.5\) 0.284 3.500 1.305 Overly conservative, minimal variation
Infinity (Default \(\tau\)) \(\tau = 1.0\) 0.334 4.031 1.323 Baseline
Infinity (Elevated \(\tau\)) \(\tau = 5.0\) 0.414 4.813 1.108 Moderate diversity gain with visual degradation
Infinity (Extreme \(\tau\)) \(\tau = 10.0\) 0.459 4.954 -0.430 Total semantic collapse, severe negative IR
DiverseVAR (Ours) Default setting 0.470 5.833 1.216 Higher diversity than \(\tau=10\) while maintaining 1.2+ IR
Infinity (Nucleus \(p\)) \(p = 0.90 \to 1.0\) 0.323 \(\to\) 0.335 3.934 \(\to\) 4.071 1.320 \(\to\) 1.330 Varying \(p\) has virtually zero effect on VAR diversity

Source: Original paper Table 4 and Table 5.

Key Findings

  • Synergy Between Noise and Scale Rollback: Condition annealing alone increases LPIPS-MPD by +69.7% on MJHQ-30K, but degrades ImageReward from 1.22 down to 0.49. Integrating Scale-Travel reclaims ImageReward to 1.08 (+120.4% relative gain over noise injection) while maintaining a +45.5% diversity gain over baseline Infinity.
  • Superior Pareto Frontier: Across varying CFG weights (\(\omega \in [2.0, 9.0]\)), DiverseVAR's Pareto curves uniformly encompass those of linear/piecewise CFG scheduling and standalone noise injection on both MJHQ-30K and MS-COCO.
  • Inadequacy of Traditional Sampling Controls: Nucleus sampling parameter variations produce negligible change in Vendi score (\(\approx 4.0\)), while scaling temperature beyond \(\tau=2.0\) causes catastrophic degradation (plummeting to \(-0.430\) at \(\tau=10\)), demonstrating that VAR's multi-scale structure requires scale-aware interventions.

Highlights & Insights

  • Repurposing Time-Travel into Scale-Travel: While diffusion models perform trajectory rewinding along temporal denoising steps, causal autoregression has long been considered strictly unidirectional. DiverseVAR demonstrates that the spatial scale hierarchy of VAR acts as a surrogate time axis, enabling non-destructive rollback via multi-scale discrete re-encoding.
  • Completely Plug-and-Play: The technique requires zero parameter updates, external discriminator networks, or auxiliary diffusion refiners, functioning as an elegant inference loop wrapper compatible with any next-scale model.
  • Complementarity with Prompt Rewriting: Where LLM prompt rewriting struggles when inputs are already granular and specific, DiverseVAR directly perturbs visual hypothesis generation, compounding effectively with rewritten prompts without mutual interference.

Limitations & Future Work

  • Marginal Computational Overhead: Re-encoding feature map \(\mathbf{Z}_l\) and regenerating scales \(m+1\) to \(K\) incurs an additional computational cost compared to single-pass autoregression.
  • Residual Artifacts in Extreme Configurations: Under aggressive diversity settings, roughly 6% of generated samples exhibit residual visual artifacts, indicating that adaptive rollback depth selection (e.g., via a lightweight visual critic) could be a fruitful direction.
  • vs. CADS (Sadat et al., ICLR 2024): While CADS showed condition annealing benefits in continuous diffusion spaces, VAR models display heightened sensitivity to condition noise, prompting DiverseVAR's specialized Scale-Travel refinement to repair corrupted discrete token sequences.
  • vs. Guidance Schedulers: Conventional CFG schedulers adjust logit magnitudes but cannot undo early deterministic structural choices once coarse tokens are sampled; DiverseVAR explicitly rewinds and re-decodes the early scale hierarchy.

Rating

  • Novelty: ⭐⭐⭐⭐ [First systematic study of diversity collapse in VAR, introducing an innovative Scale-Travel rollback mechanism leveraging multi-scale encoders]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across image and video domains, rigorous Pareto front benchmarking, and exhaustive sampling ablations]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Clear problem formulation, transparent algorithmic exposition, and intuitive visual storytelling]
  • Value: ⭐⭐⭐⭐ [Provides an immediately actionable, training-free enhancement for visual autoregressive models and downstream inference-time search]