ARVAR: Accelerating Visual Autoregressive Model via Attention Retrospect

Conference: ECCV2026
Paper: Official paper page · PDF
Area: Visual Generation Efficiency / Visual Autoregressive Image Generation
Keywords: Attention Retrospect, cross-scale caching, token pruning, Branch Halving, FlashAttention

TL;DR

ARVAR uses low-resolution attention and activations to allocate high-resolution computation, recomputing important tokens, reconstructing other positions, and reusing the conditional branch at selected late steps to achieve 1.56x HART and 3.01x Infinity-2B speedups on one RTX3090 while preserving the original model's images better than FastVAR.

Background & Motivation

Visual autoregressive modeling predicts an entire token grid at each scale instead of generating image tokens one at a time. Small grids establish coarse layout, and later grids add increasingly fine detail. HART and Infinity extend this approach to 1024-by-1024 text-to-image generation, but having few decoding steps does not make every step cheap. The final grids contain many positions, so attention and feed-forward processing remain expensive even with KV caching and FlashAttention.

FastVAR selects positions using frequency information, a cheap signal that can nevertheless discard semantically important details. SkipVAR uses an additional decision model to adapt skipping behavior, requiring model-specific fine-tuning. Attention looks like a more suitable importance signal, but explicitly retrieving the large attention matrix at high resolution conflicts with FlashAttention's avoidance of materializing that matrix. The challenge is therefore to obtain useful importance estimates cheaply and preserve the spatial grid after pruning.

The authors observe that corresponding layers retain similar attention saliency and activation structure across nearby scales, while conditional and unconditional activations become closer at late steps. The first observation allows cheap small-grid computation to guide expensive large-grid computation; the second suggests that some branch computation is redundant. Core idea: turn cross-scale stability into reusable attention and activation caches, recompute only important high-resolution positions, and remove actual unconditional-branch computation at selected late steps.

Method

Overall Architecture

ARVAR takes the same text conditioning and scale schedule as the base VAR model and ultimately produces a full image through its existing decoder. It neither retrains the generator nor replaces its tokenizer. Instead, it reduces computation inside selected high-resolution steps. Earlier steps provide layer-matched attention and activation caches; later steps optionally apply Branch Halving before attention-guided pruning and structural reconstruction.

Two independent schedules control the method. Branch Halving, or BH, determines whether a step processes only the conditional branch. Attention Retrospect, or AR, determines which spatial positions are recomputed and how skipped positions are restored. They may operate at different steps. The diagram shows the dependencies within a step where the relevant operations are enabled, not a requirement that every late step use both modules.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Text conditioning and<br/>low-resolution steps"] --> B["Layer-matched attention<br/>and activation caches"]
    B --> C["Branch Halving<br/>Select input branches by step"]
    C --> D["Attention-guided Pruning<br/>Sparse forward on salient positions"]
    B --> D
    D --> E["Structural Reconstruction<br/>Interpolated base and scatter updates"]
    B --> E
    E --> F["Repeat conditional output if needed<br/>Restore full grid and continue generation"]

Key Designs

1. Branch Halving: avoid a real unconditional forward at selected steps

The base model uses conditional and unconditional predictions for classifier-free guidance, typically concatenating the two branches along the batch dimension. BH exploits their increasing activation similarity at late steps: it sends only the conditional branch through the Transformer blocks at a configured step. After the forward pass, it repeats the conditional output to restore the two-branch interface expected by subsequent processing. It does not compute both branches and average them, and it does not discard the text condition. The saved work is the second branch's computation and associated intermediate state.

Repeating the conditional result means that this step no longer has an independent unconditional prediction to supply a guidance difference. Similar activations are an empirical justification, not a guarantee of lossless substitution for every prompt. The paper describes a 50% FLOPs reduction for the affected step; that percentage is not an end-to-end image-generation latency reduction. Earlier steps and other components still run, and some of BH's savings overlap with the work reduced by token pruning.

2. Attention-guided Pruning: rank the current large grid using an earlier small grid

AR does not additionally materialize a large attention matrix at the current high-resolution step. Instead, it retrieves attention cached at the same layer in an earlier step. If the current step is \(T\), the attention source is \(T-M\). The method extracts the self-attention component, aggregates it into a spatial saliency score for each position, and interpolates the resulting low-resolution map to the current grid. Top-K selection supplies the indices used to gather important input tokens. Only these positions undergo the expensive attention and feed-forward processing, and their spatial indices are retained for reconstruction.

The importance signal is spatial saliency extracted from historical attention, not a token's current frequency, magnitude, or random sampling probability. Compared with frequency-based selection, it can retain more spatially coherent object regions instead of dropping scattered positions within salient structures. Figure 10 reports roughly 94%-95% overlap between historical and final top-60% important positions around \(T-3\). This supports the usefulness of nearby historical saliency, while also suggesting that retaining a much smaller set is more sensitive to ranking errors.

A reproduction caveat matters here. Equation (6) and Algorithm 1 describe column averaging and mean(..., -1), but do not sufficiently specify the cache tensor's axis convention or the normalization of the extracted self-attention block. Averaging within each row of a complete row-normalized square attention matrix would yield a constant, not a useful ranking. The cached text therefore does not justify inventing a supposedly verified summation-axis formula. The supported pipeline is historical self-attention aggregation, spatial interpolation, Top-K selection, and gather; the precise aggregation axis still needs implementation-level confirmation.

The attention source must be both inexpensive and predictive. Very early scales may miss details that matter later, whereas collecting explicit attention at a later scale costs more. AR's engineering choice is to pay for this information on a relatively small grid and retain FlashAttention during the expensive high-resolution computation. This is a spatial token-selection mechanism, not an arbitrary sparsification of the entire model, and it does not eliminate the cost of caching attention.

3. Structural Reconstruction: overwrite a historical activation base with sparse updates

The shortened sequence cannot simply be passed to the next scale because VAR requires a complete spatial grid. AR separately caches the same layer's output activations at step \(T-N\), interpolates them to the current grid, and uses that full map as the initial output. It then scatters the newly computed sparse outputs back to their original Top-K positions. Important positions receive fresh values; skipped positions retain interpolated historical values. This is the reconstruction relationship in Equations (8)-(9):

\[ Y_T^l=\operatorname{Scatter}\!\left(\operatorname{Up}(Y_{T-N}^l;h_T,w_T),\ I_T^l,\ \widehat{Y}_T^l\right). \]

Here \(I_T^l\) contains the retained spatial indices, and \(\widehat{Y}_T^l\) contains their new outputs. Scatter overwrites selected positions rather than adding old and new features. This avoids filling every skipped position with zero and differs from copying an entire old grid without any current updates. The attention cache determines where to update; the activation cache determines what to place elsewhere. Their roles are distinct.

The attention offset \(M\) and activation offset \(N\) need not match. Earlier, cheaper attention can identify salient regions, while reconstruction may benefit from more recent activations to reduce texture and feature-magnitude mismatch. The reported HART configuration uses \(M=3,N=2\), illustrating that AR is not simple whole-layer output reuse. Layer-wise activation visualizations show that layout and boundaries are often preserved, but fine textures can still differ. Reconstruction is an approximation, not an exact recovery of the skipped computation.

A Worked Example

Consider the HART configuration in Table 8. The model still executes 14 scale steps rather than reducing its decoding-step count. BH runs at \(S_{13}\), computing only the conditional branch and repeating its output. AR runs at \(S_{14}\) with a 75% pruning ratio, leaving 25% of positions for the relevant forward computation. Its attention source is \(S_{11}\) and its activation source is \(S_{12}\), corresponding to \(M=3,N=2\).

The final step interpolates the layer-matched saliency map from \(S_{11}\) to the \(S_{14}\) grid and selects important positions. It separately interpolates activations from \(S_{12}\) to the same size, then overwrites the selected positions with current sparse outputs. This configuration reports a 1.56x speedup and FID 27.67 on MJHQ30K. Applying BH at both \(S_{13}\) and \(S_{14}\) raises speedup to 1.60x but changes FID to 27.73, a configurable trade-off rather than a free improvement.

Loss & Training

ARVAR introduces no new training loss, distillation stage, or model fine-tuning. It reuses pretrained HART and Infinity-2B and their generation pipelines. The configurable choices are application steps, pruning ratios, attention and activation cache sources, and BH placement. The cached paper does not sufficiently specify attention-head aggregation or the exact interpolation mode, so this note does not invent those implementation details.

Key Experimental Results

Main Results

Experiments use one NVIDIA RTX3090 at 1024-by-1024 resolution, with FlashAttention enabled for both original and accelerated backbones. The following selection from Table 2 separates semantic performance, image fidelity, and distributional quality: GenEval accuracy evaluates prompt satisfaction; SSIM and LPIPS compare accelerated outputs with the original model's outputs; MJHQ30K FID evaluates the generated distribution. Reported speedups are taken directly from the paper rather than recomputed from rounded latency values.

Model / method	Speedup	Latency, seconds	GenEval accuracy, higher	GenEval SSIM, higher	GenEval LPIPS, lower	MJHQ30K FID, lower	MJHQ30K CLIP, higher
HART	1.00x	0.94	0.490	-	-	30.75	28.51
HART + FastVAR	1.47x	0.64	0.502	0.6970	0.1597	28.19	28.39
HART + ARVAR	1.56x	0.60	0.509	0.7455	0.1186	27.67	28.41
Infinity-2B	1.00x	2.63	0.677	-	-	31.06	28.00
Infinity-2B + FastVAR	2.80x	0.94	0.683	0.8198	0.0460	30.79	27.80
Infinity-2B + ARVAR	3.01x	0.88	0.685	0.8493	0.0316	29.65	27.91

Relative to FastVAR, ARVAR is faster and better preserves the original model's images on both backbones. Relative to the unaccelerated models, however, CLIP scores fall slightly. The main conclusion is a better speed-quality trade-off among acceleration methods, not improvement over the original generator on every quality dimension.

Ablation Study

Table 5 isolates the two modules on HART and MJHQ30K. SSIM, LPIPS, and PSNR measure fidelity to the original generator's outputs.

Configuration	Speedup	SSIM, higher	LPIPS, lower	PSNR, higher	FID, lower	CLIP, higher
Original HART	1.00x	-	-	-	30.75	28.51
AR only	1.46x	0.7232	0.1088	22.59	27.56	28.50
BH only	1.43x	0.7567	0.0523	23.90	28.01	28.35
AR + BH	1.56x	0.6904	0.1204	22.00	27.67	28.41

BH alone stays closest to the original images, AR alone has the lowest FID, and their combination is fastest but not best on fidelity metrics. The individual 1.46x and 1.43x speedups cannot simply be multiplied because the modules reduce partially overlapping high-resolution work.

Table 7 also examines the selection criterion. Self-attention yields SSIM/LPIPS/FID of 0.6904/0.1204/27.67, versus 0.6689/0.1434/28.19 for frequency-based selection and 0.4537/0.3821/31.98 for random selection. Which positions receive computation clearly matters, not just the number retained. Full attention obtains a higher CLIP score, 28.45 versus self-attention's 28.41, but a worse FID of 28.23, revealing another metric-dependent trade-off.

Key Findings

• Table 9 reports peak memory changing from 21.45 GB to 21.33 GB. The principal benefit is faster computation, not a large reduction in total model memory. The attention and activation caches occupy 0.05 GB and 0.18 GB, respectively.
• Figure 13's block-level breakdown reduces attention time from 11.23 ms to 2.33 ms and FFN time from 5.24 ms to 1.30 ms, with pruning and reconstruction adding 0.55 ms and 0.21 ms. These are not end-to-end image latencies.
• In Table 6, HART + ARVAR obtains HPSv2.1 of 27.67, below FastVAR's 27.92 and original HART's 28.72. ImageReward rises from FastVAR's 0.5329 to 0.5511 but remains below the original model's 0.5634. The prose claim of universally highest quality scores should not be repeated without this qualification.

Highlights & Insights

• Attention can be valuable without extracting the largest matrix at the moment it is needed. Collecting structural clues on small grids and using them on large grids preserves both attention guidance and FlashAttention's implementation advantages.
• Pruning and reconstruction form a paired design: saliency identifies update locations, while historical activations provide a plausible base elsewhere. The reusable idea is to allocate fresh computation spatially instead of choosing only between whole-layer recomputation and whole-layer caching.
• Independent AR and BH schedules are more useful than assuming both should always be enabled. Ablations favor different configurations for fidelity, FID, and throughput; no single configuration wins every metric.

Limitations & Future Work

• The authors report three BH failure patterns: fine attribute binding, rare attributes, and local asymmetry. Examples include left-right attribute confusion, unusual blue bread crust reverting toward a typical appearance, and loss of distinctions such as heterochromia or a rear wing on only one car. Similar overall activations can hide meaningful semantic differences.
• Historical saliency depends on cross-scale structural stability. More aggressive pruning and details that emerge only late can invalidate old rankings. Prompt-aware fallback or a cache-disagreement trigger for full computation could be investigated, but neither is an existing component of this method.
• Evidence mainly covers two text-to-image backbones, 1024-by-1024 images, and one RTX3090. It does not establish the same speedups for video, other scale schedules, hardware, or larger batches. The reported main results do not include repeated-run uncertainty bars.
• Reproduction is limited by the saliency-axis ambiguity and inconsistencies in some baseline names and citations. Table 8 changes pruning step and ratio together in some comparisons, so their effects cannot all be attributed to step placement alone. Implementation verification and more tightly controlled experiments remain necessary.

Related Work & Insights

• Versus FastVAR: Both target redundant work on large late-stage grids. FastVAR uses frequency-based selection; ARVAR uses historical self-attention and adds independently scheduled BH. The AR-only ablation already improves fidelity at similar speedup, so BH cannot explain all gains.
• Versus SkipVAR: SkipVAR uses an additional decision model for adaptive skipping. ARVAR requires no such training and instead reduces positions and branches inside existing steps, but it still needs a suitable fixed schedule for each backbone.
• Versus ToMe and generic feature caching: Token merging changes the spatial representation, while whole-layer caching may update no positions at all. ARVAR uses historical activations as a base and overwrites important locations with current results, explicitly maintaining the grid required by subsequent scales.
• Versus FlashAttention: These methods are complementary. FlashAttention improves memory access for attention computation; ARVAR reduces the positions and branches processed at high resolution and moves explicit attention statistics to smaller grids.

Rating

• Novelty: 4/5. Cross-scale attention guidance and activation reconstruction fit together well, with BH adding a separate compression direction, although the ideas build on existing pruning and caching approaches.
• Experimental Thoroughness: 4/5. Two backbones, multiple metrics, module and scheduling ablations, memory measurements, and failure examples are useful; broader deployment settings and uncertainty estimates are missing.
• Writing Quality: 3/5. The overall mechanism is clear, but saliency dimensions are underspecified and some narrative claims conflict with the tables.
• Value: 4/5. Training-free gains remain useful even when FlashAttention is already enabled; tasks requiring precise semantic detail should apply BH conservatively.