MEPA: Multi-Scale Representation Alignment for Visual Autoregressive Modeling with Mixture of Experts

Conference: ECCV 2026
arXiv: 2607.00371
Code: None
Area: Image Generation
Keywords: Visual Autoregressive Modeling, Mixture of Experts, Multi-Scale Representation Alignment, Semantic Guidance, Image Generation

TL;DR

To address the issues of multi-scale representation learning conflicts and early-scale semantic error propagation caused by a shared architecture in Visual AutoRegressive (VAR) models, MEPA proposes a Scale-aware Token-routed Mixture of Experts (STMoE) to decouple model capacity across different scales. Furthermore, it incorporates self-supervised visual features (DINOv3) to perform semantic alignment on the accumulated residual aggregated representations of early scales. On ImageNet 256x256, MEPA outperforms the VAR baseline using only half the training epochs, reducing FID by 0.63–0.69.

Background & Motivation

Visual Autoregressive Modeling (VAR) achieves coarse-to-fine multi-scale autoregressive image generation via the "next-scale prediction" paradigm, exhibiting excellent performance in both generation quality and inference speed. VAR compresses images into multi-scale residual token sequences, predicting them sequentially from small to large scales and decoding them in parallel within each scale, which significantly improves efficiency.

However, VAR suffers from two inherent limitations in multi-scale representation learning. First, learning objectives across different scales vary significantly: smaller scales primarily model global semantic layouts, whereas larger scales focus on fine-grained textual details. Yet, all scales share a single Transformer architecture, leading to conflicting optimization objectives—the model is forced to learn both "sketching outlines" and "drawing details" within the same set of parameters. Second, error propagation along the autoregressive causal chain: since the generation process sequentially depends on scales from small to large, if semantic predictions at early scales are inaccurate (e.g., misaligned object positions or class confusion), these errors will be propagated as conditional inputs to all subsequent scales, amplifying layer by layer and ultimately degrading generation quality. Empirical analysis in the paper (Fig. 1) validates these two points: feature spaces at different scales exhibit distinct distributions, and early semantic errors indeed lead to unacceptable outputs.

The core insight of this paper is to utilize a Mixture of Experts (MoE) architecture to let different scales and tokens adaptively select distinct FFN experts, achieving decoupled and specialized representation learning. Concurrently, pre-trained self-supervised visual features (DINOv3) are employed to perform semantic regularization on the aggregated representations of early scales in VAR, blocking error propagation from the source. This is the first work to systematically investigate representation alignment under the VAR paradigm.

Method

Overall Architecture

MEPA introduces two core modules to the next-scale prediction framework of VAR: the Scale-aware Token-routed MoE (STMoE) layer and the semantic guidance mechanism. An input image is quantized by the VAR tokenizer into \(K\) scales of residual token sequences \(\{r_1, r_2, \ldots, r_K\}\), which are sequentially fed into the Transformer from the smallest to the largest scale. MEPA replaces the standard FFN in the Transformer with the STMoE layer, where each token incorporates scale embeddings and uses a router to select the top-k experts. During training, output features of small and medium scales are extracted from the intermediate layers of the Transformer, projected via MLP, and then progressively aggregated and upsampled from small to large scales. These aggregated representations are aligned with the complete image features extracted by a frozen DINOv3 encoder using cosine similarity, forming the semantic guidance loss \(\mathcal{L}_{SG}\). The final loss is a weighted sum of the semantic guidance loss and the original cross-entropy loss. Inference remains standard autoregressive generation using only the STMoE Transformer without any extra overhead.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image"] --> B["VAR Tokenizer<br/>Multi-scale Residual Quantization"]
    B --> C["STMoE Transformer<br/>Scale-aware Token Routing"]
    C --> D["Intermediate Features<br/>Small/Medium Scale Outputs"]
    D --> E["Residual Feature Aggregation<br/>Scale-wise Accumulation + Upsampling"]
    E --> F["Alignment with DINOv3 Features<br/>Cosine Similarity Maximization"]
    C --> G["Multi-scale Token Prediction"]
    G --> H["Cross-Entropy Loss"]
    F --> I["Final Loss<br/>L_SG + λ·L_CE"]
    H --> I

Key Designs

1. Scale-aware Token-routed Mixture of Experts (STMoE): Evolution from Scale Routing to Token Routing

The feature spaces of different scales in VAR vary significantly. A shared FFN forces the model to compromise across all scales, hindering the learning of specialized representations for each scale. The most intuitive solution is to allocate experts based on scale—i.e., Scale-routed MoE (SMoE): learning a parameterized scale embedding for each scale and computing the scale-to-expert affinity using a routing matrix \(S = \text{Softmax}_E(SE(scale(x)) W)\), where all tokens of the same scale share the same expert activation pattern.

However, SMoE has two critical flaws. First, the number of tokens varies drastically across scales (fewer tokens at smaller scales and significantly more at larger scales), which causes severe load imbalance among experts—as shown in Table 1, the load variance of 8 experts in SMoE is as high as 6.62, while STMoE is only 2.41. Second, different tokens within the same scale possess distinct semantic importance and spatial characteristics due to different spatial positions (e.g., boundary tokens vs. center tokens). Applying the same expert activation to all tokens of the identical scale in SMoE results in homogeneous routing, limiting the model's fine-grained specialization capabilities.

To solve this, STMoE injects scale embeddings directly into the token representations and performs routing on a per-token basis: \(S = \text{Softmax}_E((SE(scale(x)) + x) W)\). In practice, the VAR Transformer itself already adds positional and scale embeddings to the input tokens, allowing STMoE to naturally reuse this information. Consequently, each token independently selects its experts based on the combination of its content and scale information. This allows the model to perceive scale-wise variations (e.g., larger scale tokens leaning towards certain experts, while smaller scale tokens prefer others) and execute fine-grained routing according to token-level semantics (e.g., boundary and center tokens at the same scale might select different experts). Routing heatmaps (Fig. 4) validate this behavior: in STMoE at scale 10, Expert 3 prefers boundary tokens, Expert 6 prefers non-boundary tokens, and Expert 1 prefers tokens in the first row of the feature map. Additionally, a standard load-balancing loss \(\mathcal{L}_b = M \sum_{j=1}^{m}(\frac{1}{N}\sum_{i=1}^{N}\mathbb{I}(G_{i,j}\neq 0))(\frac{1}{N}\sum_{i=1}^{N}S_{i,j})\) is introduced to encourage uniform use of all experts by penalizing the product of selection frequency and routing scores, safeguarding distributed training efficiency.

2. Residual Feature Aggregation: Bridging VAR Residual Space and Self-Supervised Full Representation Space

Directly aligning an individual residual representation in VAR with self-supervised features performs poorly due to a fundamental mismatch between their representation spaces. Specifically, VAR's \(r_k\) is the residual increment of scale \(k\) relative to the previous \(k-1\) scales, whereas self-supervised encoders like DINO/DINOv3 output complete patch-wise representations of the full image. Residual features are local and incremental, while self-supervised features are global and complete. Enforcing direct alignment leads to feature mismatch, failing to inject effective semantics and potentially interfering with VAR's generative learning.

MEPA's design is to project the intermediate outputs of the VAR Transformer using an MLP \(m_\phi\) to the channel dimension of the DINOv3 features, and then accumulate and upsample them sequentially to the target resolution: \(z_j = \sum_{i=1}^{j} \text{Up}(m_\phi(h_\theta)_i, sz_K)\). This operation transforms the residual space into a progressively aggregated representation space—where \(z_1\) contains only the coarsest global structural information, \(z_3\) aggregates semantic inputs from the first three scales, and \(z_K\) forms the complete aggregated representation. This progressive accumulation from small to large scales naturally aligns with the causal generation order of VAR, ensuring that the informational richness of the aggregated features increases monotonically with scale.

3. Semantic Guidance at Early Scales: Blocking Error Propagation from the Source

With the aggregated representations, a crucial question is "at which scales should the alignment be applied." A naive approach is to align only the final fully aggregated representation \(z_K\) (denoted as +SG-Last). However, this only constrains the final generation outcome without directly reinforcing the semantic accuracy of earlier scales. Due to the causal dependency structure of VAR, where large-scale predictions are strictly conditioned on the feature maps of earlier scales, any early semantic error propagates and amplifies along the generation chain once it occurs.

MEPA chooses to apply semantic guidance only on early and intermediate aggregated representations. Let \(F\) be the set of selected early scales (empirically determined as the aggregated features of scales 1-5, 1-6, and 1-7, denoted as +SG-M). The patch-wise cosine similarity is maximized between each selected \(z_j (j \in F)\) and the DINOv3 feature \(g\):

\[\mathcal{L}_{SG}(\theta, \phi) = -\mathbb{E}_x\left[\frac{1}{N}\sum_{i=1}^{N}\sum_{j \in F} \text{sim}(g^i, z_j^i)\right]\]

This approach directly strengthens the semantic correctness of the early-to-mid scale stages, forcing the model to produce accurate object categories, spatial layouts, and global structures while generating coarse semantic sketches. Since fine-detailed generations are conditioned on these early features, a more accurate semantic foundation naturally guides more reliable fine-grained detailed rendering, minimizing error propagation from the ground up. Ablation studies (Table 6) confirm that +SG-M (the medium-scale group) achieves the best FID of 3.59, outperforming +SG-Last (3.81) and +SG-S (the smallest three scales, 3.64), aligning perfectly with the motivation of "reinforcing early semantic foundations."

Loss & Training

The final training objective is a weighted combination of the semantic guidance loss and the original VAR cross-entropy loss: \(\mathcal{L} = \mathcal{L}_{SG} + \lambda \mathcal{L}_{CE}\), where \(\lambda = 0.5\). In addition, the STMoE layer requires an auxiliary load-balancing loss \(\mathcal{L}_b\). Optimization uses the AdamW optimizer with a batch size of 96, weight decay of 0.05, \(\beta_1=0.9, \beta_2=0.95\), and the base learning rate linearly decayed from \(1\times10^{-4}\) to \(1\times10^{-5}\), consistent with the original VAR work. All experiments are conducted on ImageNet-1K 256x256, defaulting to 200 epochs of training (100-epoch efficiency results are also reported). Semantic guidance employs a frozen DINOv3 as the target encoder. Crucially, aligning features extracted from intermediate layers of the VAR Transformer, rather than the final layer, yields superior performance.

Key Experimental Results

Main Results

MEPA is compared with mainstream methods on the ImageNet 256x256 class-conditional generation benchmark. MEPA-d16 (585M parameters, 10 generation steps) achieves an FID of 2.32 after 200 training epochs, significantly outperforming VAR-d16 (3.55) and VAR-d20 (2.95), yet utilizing fewer parameters. With only 100 training epochs, MEPA-d16 achieves an FID of 2.65, which already surpasses the 200-epoch VAR-d16 baseline (3.55), demonstrating an approximate 2x training acceleration.

Type	Model	FID↓	IS↑	Params	Steps	Epochs
GAN	StyleGAN-XL	2.30	265.1	166M	1	-
Diffusion	DiT-XL/2	2.27	278.2	675M	250	-
AR	MAR-B	2.31	281.7	208M	64	800
VAR	VAR-d16	3.55	280.4	310M	10	200
VAR	VAR-d20	2.95	302.6	600M	10	250
VAR	FlexVAR-d20	2.41	299.3	600M	10	250
VAR	SpectralAR-d16	3.02	282.2	310M	64	200
VAR	MEPA-d16 (100 epochs)	2.65	304.6	585M	10	100
VAR	MEPA-d16 (200 epochs)	2.32	311.3	585M	10	200

Ablation Study

Configuration	FID↓	IS↑	Description
VAR-d16 baseline (100 epochs)	4.10	241.6	No STMoE, No SG
+STMoE only	2.99	284.2	Only STMoE added, FID reduced by 1.11
+SG only (DINOv3)	3.59	269.0	Only Semantic Guidance added, FID reduced by 0.51
+STMoE + SG (Full MEPA)	2.65	304.6	Joint optimal, FID reduced by 1.45

MoE Type	FID↓	IS↑	Description
VAR Baseline	4.10	241.6	No MoE
+MoEEC	4.06	234.9	General MoE, no scale awareness, negligible gain
+MoE++	3.39	271.7	Improved version of generic MoE
+SMoE	3.11	280.7	Scale routing, homogeneous routing at the same scale
+STMoE	2.99	284.2	Token-level routing + scale awareness, optimal

Key Findings

• STMoE and SG are complementary rather than redundant: Individually introducing STMoE or SG yields significant improvements, and their combination further bolsters performance (reducing FID from 4.10 to 2.65). This indicates that they address distinct bottlenecks in multi-scale representation learning—STMoE decouples model capacity, while SG enhances early semantics.
• STMoE exhibits much greater routing flexibility than SMoE: Routing heatmaps show that STMoE activates all 8 experts (compared to only 6 activated by SMoE), and different tokens at different spatial locations within the same scale select distinct experts, enabling finer-grained specialization.
• Intermediate scale aggregation performs best for alignment: +SG-M (scales 1-5/1-6/1-7) achieves an FID of 3.59, which is superior to +SG-Last (only final complete representation, FID 3.81) and +SG-S (only the three smallest scales, FID 3.64), validating that "reinforcing early semantic foundations" is more effective than "constraining the generation endpoint."
• Semantic guidance accelerates convergence: To reach equivalent performance levels, models equipped with SG require only about 100 epochs to match the 200-epoch baseline, achieving approximately a 2x training speedup (Fig. 5).
• MEPA's inference overhead is manageable: Compared to VAR-d20, MEPA-d16 increases inference time from 0.50s to 0.85s (approx. 0.7x increase), with only a 4.5% increase in training time per epoch. However, it surpasses the 200-epoch baseline with only 100 epochs, rendering the actual total training time shorter.

Highlights & Insights

• Decoupled multi-scale learning from an architectural perspective: Utilizing MoE to let different experts adaptively specialize in processing different scale and semantic tokens is more elegant than manually designing multi-branch or multi-head architectures. The router automatically learns "which expert to direct which token to," eliminating the need for manual, hard-coded scale bisections.
• "Residual accumulation before alignment" is a key engineering insight: Direct alignment of raw residual features and self-supervised features fails due to representational space mismatches. By cumulatively summing features along the causal direction of VAR, the residual space is "translated" into a progressively completed representation space, naturally bridging the discrepancy. This strategy can be extended to establish self-supervised representation guidance in any residual generative architectures (e.g., cascaded diffusion models, progressive GANs).
• Aligning the origin rather than the endpoint: The superiority of +SG-M over +SG-Last reveals an intuitive but crucial finding—in autoregressive generation, constraining intermediate representations of early stages is more effective than constraining the final output. This provides valuable insights for other autoregressive models (e.g., intermediate-layer representation alignment for LLMs, intermediate-step supervision for reasoning chains).
• A win-win for training efficiency and generation quality: Instead of trading computational overhead for better generation quality like many prior works, MEPA achieves training acceleration (2x) through superior representation learning while still reaching higher final generation quality, demonstrating that "learning faster" and "learning better" can be simultaneously realized.

Limitations & Future Work

• The authors acknowledge that due to limited computing resources, the model has not been fully trained on the 512x512 resolution (only 66 epochs completed), and its scaling behavior at higher resolutions remains to be validated.
• The additional parameters introduced by STMoE (585M vs. 310M in VAR-d16, though active parameters are fewer) make a direct, parameter-equivalent comparison to dense baselines challenging. Although it maintains inference efficiency advantages (10 steps vs. 250 steps of Diffusion), the single-step inference time is increased by 0.7x.
• Semantic guidance currently only explores discriminative self-supervised features from DINO/DINOv3. Whether other pre-trained representations (such as multimodal features from CLIP or reconstructive features from MAE) can provide complementary semantic signals remains unexplored.
• The routing strategies of the MoE still have room for improvement. While the current model employs a fixed top-k selection, future work could explore dynamic top-k selection, uncertainty/difficulty-aware routing, or gradually growing the number of experts during training.

Related Work & Insights

• vs. REPA (Representation Alignment for Diffusion Models): REPA aligns self-supervised features during the denoising process of diffusion models, while MEPA is the first to perform representation alignment in the VAR paradigm. The key difference lies in VAR's multi-scale residual feature space—direct alignment (akin to diffusion models) fails here, necessitating cumulative residual aggregation before selectively aligning early scales. MEPA's FID gain (21.36%) significantly outperforms REPA's gain on SiT (12.62%), proving that alignment strategies specifically tailored for VAR's characteristics are much more effective.
• vs. Generic MoE (MoEEC / MoE++）: These generic MoE methods directly replace FFN with MoE layers without incorporating scale-aware routing designs, showing negligible improvements on VAR (FID 4.06 vs. 4.10). This confirms that in multi-scale architectures like VAR, scale information is a prerequisite for MoE to function effectively.
• vs. FlexVAR / SpectralAR: These studies improve VAR from different perspectives (e.g., flexible scale strategies or frequency-domain modeling) and remain orthogonal to MEPA's MoE + representation alignment, indicating potential for integration.

Rating

• Novelty: ⭐⭐⭐⭐☆ This is the first work to introduce MoE and representation alignment into the VAR paradigm. Both the evolution of STMoE from SMoE to token-level routing and the residual aggregation alignment strategy exhibit originality, though MoE and representation alignment have precedent in NLP and diffusion models, respectively.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Ablation studies comprehensively cover multiple dimensions, including MoE types, routing strategies, self-supervised encoder selections, aggregated scale group selections, load balancing, and comparisons with REPA. The routing heatmap visualizations and convergence curve analyses are highly convincing.
• Writing Quality: ⭐⭐⭐⭐☆ The logic is clean, and the progression from motivation (empirical analysis in Fig. 1) to method (the two modules) to experiments (progressive ablations) is complete, though some equations are slightly cluttered.
• Value: ⭐⭐⭐⭐☆ It delivers a win-win in both training efficiency (2x speedup) and generation quality, offering practical value to the AR image generation community. The "residual aggregation before alignment" approach is highly inspiring for broader residual generative models.

Related Papers

• [ICLR 2026] Multi-Subspace Multi-Modal Modeling for Diffusion Models: Estimation, Convergence and Mixture of Experts
• [ICLR 2026] MVAR: Visual Autoregressive Modeling with Scale and Spatial Markovian Conditioning
• [ICLR 2026] SSG: Scaled Spatial Guidance for Multi-Scale Visual Autoregressive Generation
• [ICML 2026] Visual Implicit Autoregressive Modeling
• [CVPR 2026] Mixture of Style Experts for Diverse Image Stylization