EVAR: Edge Visual Autoregressive Models via Principled Pruning

Conference: ECCV 2026
Paper: ECCV Official
Project: https://aden9460.github.io/EVAR
Code: https://aden9460.github.io/EVAR
Area: Image Generation
Keywords: Visual Autoregressive Models, Structured Pruning, Second-Order Optimization, Edge Deployment, Knowledge Distillation

TL;DR

Addressing the severe inference latency on edge devices and the cross-scale error cascading in Visual Autoregressive (VAR) models, EVAR introduces a structured pruning framework combining Pyramid-Aware second-order optimization with Progressive Scale-Aware Distillation (PSAD), pruning VAR-d16 by 40% to 230M parameters while maintaining 3.91 FID and accelerating single-image latency to 277 ms (1.8x speedup) on iOS.

Background & Motivation

Next-scale visual autoregressive (VAR) modeling departs from conventional next-token autoregressive generation by decoding visual tokens in a coarse-to-fine hierarchy, generating all tokens in parallel within each scale. This paradigm achieves generative fidelity and scalability on par with state-of-the-art diffusion models while markedly reducing decoding steps. However, existing VAR acceleration efforts predominantly optimize for throughput in large-batch server scenarios, leaving single-image, low-latency generation on resource-constrained mobile and edge devices largely unaddressed. On memory-constrained mobile devices such as smartphones, dense VAR models frequently encounter out-of-memory crashes, highlighting an urgent demand for hardware-aligned structured model compression.

Directly applying conventional second-order structured pruning methods (such as LLM-oriented Optimal Brain Surgeon / OBS frameworks) to next-scale VAR leads to severe architectural mismatch. The multi-scale generative hierarchy of VAR suffers from two inherent vulnerabilities: extreme scale-wise token imbalance and autoregressive error cascading. Coarse-scale tokens (e.g., \(1\times 1\) or \(2\times 2\)) govern the global structural appearance and semantic layout of the image, yet they constitute only a tiny fraction of the total sequence length. Conversely, fine-scale tokens dominate the total sequence count. Accumulating Hessian statistics over flattened sequences causes abundant fine-scale activations to overwhelm the Hessian, underrepresenting early structural scales. Furthermore, due to the autoregressive dependency, pruning-induced errors in early scales continuously accumulate and amplify through subsequent scales, causing irreversible geometric distortions. Similarly, during fine-tuning, standard objectives remain dominated by fine-scale token gradients, leaving coarse scales under-supervised.

To preserve global semantic structures while enabling stable closed-form OBS weight compensation, the approach rebalances multi-scale statistical contributions and coordinates scale-wise gradient allocation. Core Idea: Introduce EVAR, the first structured pruning framework tailored for next-scale VAR models, utilizing scale-normalized and decay-weighted Pyramid-Aware OBS to eliminate fine-scale dominance in Hessian estimation while ensuring positive-definiteness, paired with Progressive Scale-Aware Distillation (PSAD) to balance backpropagation gradients for low-latency, high-fidelity on-device generation.

Method

Overall Architecture

The EVAR framework consists of four primary stages: multi-scale streaming calibration, Pyramid-Aware OBS structured pruning, Progressive Scale-Aware Distillation (PSAD), and Core ML hardware-aligned execution. First, an uncorrupted calibration set of real images is pre-encoded via the VQ-VAE residual pyramid to collect activation statistics across layers via a memory-efficient streaming pipeline. Next, second-order sensitivity scores guide the structured pruning of multi-head attention heads and FFN intermediate channels, accompanied by closed-form weight compensation. Then, PSAD restores capacity via smooth soft-gating and inverse token-count weighting. Finally, the pruned model is converted to Core ML and deployed on edge hardware with optimized compute-unit scheduling.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Real Calibration Images<br/>Pre-encoded Residual Pyramid"] --> B["Streaming Calibration & Pyramid-Aware Hessian Accumulation<br/>Scale-count normalization & decay weighting"]
    B --> C["Pyramid-Aware OBS Structured Pruning<br/>Coupled attention heads & FFN channels pruned with closed-form compensation"]
    C --> D["Progressive Scale-Aware Distillation (PSAD)<br/>Coarse-to-fine soft gating & balanced gradient recovery"]
    D --> E["Core ML Edge Runtime<br/>Pinned to CPU+GPU avoiding NPU fallback overhead"]

Key Designs

1. Pyramid-Aware OBS: Rebalanced Multi-Scale Second-Order Estimation

To resolve the issue where standard Hessian accumulation is overwhelmed by fine-scale tokens and triggers catastrophic error cascading, Pyramid-Aware OBS introduces token-count normalization and scale-decay weighting into Hessian accumulation. Computing independent Hessians per scale is impractical because coarse scales contain fewer tokens (e.g., 1 or 4) than the hidden dimension \(D\), resulting in rank-deficient, non-invertible covariance matrices. EVAR constructs a unified weighted Gram matrix augmented with a damping term \(\lambda I\) to strictly guarantee positive-definiteness:

\[H_{\mathrm{pa}} = \sum_{i=1}^{K}\frac{w_i}{n_i}\left(X^{(i)}\right)^{\top}X^{(i)} + \lambda I\]

where \(X^{(i)} \in \mathbb{R}^{n_i \times D}\) denotes the input activations for scale \(i\) containing \(n_i\) tokens. The factor \(1/n_i\) cancels the disparity in token counts across scales, while the decay weight \(w_i\) assigns higher relative significance to early coarse scales. Consequently, the inversion \(H_{\mathrm{pa}}^{-1}\) emphasizes weight directions that protect coarse structural representations.

2. Structured Unit Pruning and Closed-Form Compensation: Hardware-Aligned Compression

To translate parameter reduction into actual wall-clock speedup on edge processors, EVAR groups weights into physically removable structured units, including attention-head projection dimensions and FFN intermediate channel slices. The saliency error of each structured unit \(u\) is calculated by summing column-wise OBS metrics:

\[\mathrm{score}(u) = \sum_{p \in u} \frac{\|W_{:,p}^{(S)}\|^2}{\left(H_{\mathrm{pa}}^{-1}\right)_{p,p}}\]

After zeroing out units with the smallest estimated perturbation, closed-form second-order compensation is applied to the remaining unpruned parameters:

\[\Delta^{(S)} = \sum_{p \in u} \frac{W_{:,p}^{(S)}}{\left(H_{\mathrm{pa}}^{-1}\right)_{p,p}} \left(H_{\mathrm{pa}}^{-1}\right)_{p,:}\]

For large FFN layers, EVAR adopts a dynamic grouped pruning strategy that removes channels in small groups with progressively decreasing group sizes, balancing pruning efficiency with reconstruction precision.

3. Progressive Scale-Aware Distillation (PSAD): Harmonizing Scale Gradients

Pruned compact models possess reduced parameter capacity. Standard distillation distributes equal loss weights across tokens, causing fine-scale tokens to monopolize optimization gradients while coarse-scale layouts remain under-supervised. PSAD tackles this with a smooth coarse-to-fine gating curriculum \(\tilde{\gamma}_i(t)\) and scale-balancing weights \(w_i\):

\[\tilde{\gamma}_i(t) = \min\left(1, \max\left(0, \frac{t - t_i}{\Delta_i}\right)\right), \qquad w_i = \frac{C}{n_i}\]

Here, \(t_i\) and \(\Delta_i\) define the activation onset and ramp-up duration for scale \(i\), avoiding training instability from discrete scale unlocking. Setting \(w_i = C / n_i\) cancels the token count factor in the summation, ensuring that effective backpropagated gradient norms \(g_i(t) \propto \tilde{\gamma}_i(t) w_i n_i \bar{g}_i \approx \tilde{\gamma}_i(t) C \bar{g}_i\) remain balanced across the generative hierarchy, allowing the pruned network to robustly re-learn global image topology before fine-tuning local textures.

Loss & Training

The overall training loss for EVAR fine-tuning is: $\(\mathcal{L}_{\mathrm{total}}(t) = \mathcal{L}_{\mathrm{task}}(t) + \mathcal{L}_{\mathrm{PSAD}}(t)\)$ where \(\mathcal{L}_{\mathrm{task}}(t)\) is the autoregressive cross-entropy loss modulated by the scale gates, and \(\mathcal{L}_{\mathrm{PSAD}}(t)\) is the temperature-scaled KL divergence: $\(\mathcal{L}_{\mathrm{PSAD}}(t) = \tau^2 \sum_{i=1}^{K}\tilde{\gamma}_i(t)\,w_i\,\mathcal{L}_{\mathrm{KL}}^{(i)}\)$ Fine-tuning is conducted for 40 epochs using AdamW on a single 8 \(\times\) NVIDIA RTX 5090 node, taking approximately 1 hour per epoch for pruned VAR-d16.

Key Experimental Results

Main Results

On class-conditional ImageNet \(256 \times 256\), EVAR demonstrates competitive synthesis quality against leading autoregressive models and baselines:

Type	Model	Parameters	Pruning Rate	Steps	FID ↓	IS ↑	Precision ↑	Recall ↑
VAR	VAR-d16	310M	–	10	3.55	274.4	0.84	0.51
VAR	VAR-d20	600M	–	10	2.95	302.6	0.83	0.56
VAR	VAR-d24	1.0B	–	10	2.33	312.9	0.82	0.59
AR	LlamaGen-L	343M	–	576	3.07	256.1	0.83	0.52
AR	AiM-L	350M	–	256	2.83	244.6	0.82	0.55
Collab / Linear	CoDe (N=8)	2.3B	–	10	2.26	300.4	0.81	0.58
Collab / Linear	FastVAR (d24)	1.0B	–	10	2.64	284.4	0.80	0.58
EVAR (Ours)	VAR-d16 (20% sparse)	270M	20%	10	3.67	267.5	0.81	0.51
EVAR (Ours)	VAR-d16 (40% sparse)	230M	40%	10	3.91	259.1	0.81	0.51
EVAR (Ours)	VAR-d24 (20% sparse)	875M	20%	10	2.47	310.2	0.82	0.57
EVAR (Ours)	VAR-d24 (40% sparse)	748M	40%	10	2.52	304.4	0.82	0.57

Comparison across structured pruning baselines on VAR-d16 (Tab. 2):

Method	40% Sparse (Training-Free) FID ↓	40% Sparse (Training-Free) IS ↑	20% Sparse (Training-Free) FID ↓	20% Sparse (Training-Free) IS ↑	40% Sparse (1-Epoch FT) FID ↓	40% Sparse (1-Epoch FT) IS ↑
SlimGPT	138.91	6.39	56.87	19.35	10.62	44.91
Magnitude	180.82	3.90	152.29	5.77	12.69	25.84
LLM-pruner	153.82	5.62	35.21	28.95	10.82	41.35
OBA	142.82	6.35	30.54	29.48	9.85	44.35
Taylor	145.66	6.78	27.30	31.48	8.35	46.87
EVAR (Ours)	58.21	21.12	7.85	49.21	4.86	55.61

Ablation Study

Ablation on calibration and Hessian accumulation for VAR-d16 at 40% sparsity (Tab. 3):

Calibration Configuration	Params (M)	FID ↓	Note
Dense Baseline	310	3.55	Unpruned dense model
First scale only	230	124.21	Coarse tokens only; severe feature underrepresentation
Fifth scale only	230	102.54	Single intermediate scale fails across full hierarchy
Last scale only	230	86.51	Fine tokens only; coarse structural fidelity destroyed
Direct token concatenation	230	69.21	Flat token assembly; dominated by fine-scale tokens
Pyramid-Aware Accumulation (Ours)	230	58.21	Scale-count normalization & decay weighting; best one-shot quality

Ablation of distillation components and weighting schemes (40% sparsity, 230M, Tab. 4):

Fine-Tuning & Distillation Scheme	Params (M)	FID ↓	Note
Pruned (Training-free)	230	58.21	Post-pruning starting point
+ Standard Fine-Tuning (FT)	230	4.25	Task loss only without teacher guidance
+ Vanilla KD	230	4.26	Uniform token weighting across all scales
+ Exponential Increase	230	5.82	Heavier fine-scale weights exacerbate coarse degradation
+ Linear Increase	230	5.45	Still over-emphasizes fine scales
+ Exponential Decay	230	4.36	Over-suppresses fine tokens, starving texture details
+ PSAD (Ours)	230	3.91	Curriculum soft gating & inverse weighting achieve top fidelity

Core ML operator assignment and runtime latency on iPad Pro (M4) (Tab. 5):

Compute Units	Latency (ms)	#CPU ops	#GPU ops	#NPU ops	Scheduling Analysis
CPU + NPU	1090 ± 2	274	0	8554	274 unsupported ops cause extensive CPU-NPU data copies and high latency
All (CPU + GPU + NPU)	494 ± 2	129	3103	5596	Frequent 3-unit handoffs inflate synchronization latency
CPU + GPU	277 ± 2	10	8818	0	Clean two-unit scheduling avoids NPU fallback, achieving 1.8x speedup

Key Findings

• Crucial Role of Early-Scale Hessian Rebalancing: Under 20% sparsity training-free conditions, EVAR achieves an FID of 7.85 compared to SlimGPT's 56.87 and Magnitude's 152.29, proving that preventing coarse-scale error compounding is far more effective than unweighted second-order compensation.
• Strict Need for Inverse Token-Count Weighting: Tab. 4 verifies that increasing fine-scale distillation weights worsens FID (5.45–5.82), while aggressive exponential decay starves fine details (4.36 FID). The theoretical balance \(w_i = C/n_i\) in PSAD provides optimal recovery (3.91 FID).
• Asymmetric Speedup in Compute-Bound Edge Environments: While batch-size-1 inference on RTX 5090 shows identical latency across sparsity levels due to memory-bandwidth and launch overhead constraints, mobile devices (iPad Pro M4) are compute-bound, allowing EVAR's structured width reduction to yield a direct 1.8x physical speedup (494 ms to 277 ms) and prevent OOM failures on smartphones (iPhone 16 Pro Max).

Highlights & Insights

• Pyramid-Aware Second-Order Statistics: Solves the dilemma of rank deficiency from single-scale partitioning versus fine-scale dominance from flat token concatenation, establishing a mathematically sound Gram formulation with damping.
• Curriculum-Gated Multi-Scale Distillation: Employs continuous soft gating to smoothly phase in fine resolutions during fine-tuning, avoiding abrupt gradient spikes and stabilizing the capacity recovery of pruned architectures.
• Empirical Profiling of Mobile Core ML Runtimes: Disproves the common assumption that NPUs are unconditionally optimal for transformer deployment; because custom positional embeddings and sampling operations lack NPU support, executing exclusively on CPU+GPU eliminates 274 inter-unit round-trips and reduces latency from 1090 ms to 277 ms.

Limitations & Future Work

• Extension to Text-to-Image Cross-Attention: Current evaluations are centered on class-conditional ImageNet generation; extending EVAR to open-domain text-to-image autoregressive transformers requires handling joint pruning of text conditioning cross-attention.
• Scaling to Higher Resolutions: As next-scale models scale to \(512 \times 512\) with more pyramid stages, late-stage token lengths grow even faster, potentially necessitating dynamic per-scale pruning ratios.
• Synergy with Low-Bit Quantization: Combining structured channel pruning with INT4/FP8 post-training quantization could further reduce memory footprint on edge devices.

Related Work & Insights

• vs SlimGPT / LLM-pruner: These methods flatten tokens along a 1D sequence and ignore hierarchical visual scale dynamics; EVAR explicitly structures Hessian accumulation and recovery across scales, delivering superior fidelity.
• vs CoDe / FastVAR: CoDe relies on dual-model collaborative drafting, and FastVAR prunes KV caches without altering model weights; EVAR provides orthogonal structured parameter compression that can be combined with these algorithmic inference techniques.

Rating

• Novelty: ⭐⭐⭐⭐☆ Pioneers structured second-order pruning and scale-aware distillation tailored to next-scale visual autoregressive architectures.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluations across sparsity regimes, training-free vs fine-tuned baselines, multi-component ablations, and real iOS on-device hardware profiling.
• Writing Quality: ⭐⭐⭐⭐⭐ Well-structured narrative with sound mathematical formulation and clear engineering insights.
• Value: ⭐⭐⭐⭐⭐ Provides a practical blueprint for deploying high-fidelity visual autoregressive models on edge consumer devices.