SPAR: Semantic-Pixel Self-Alignment and Adaptive Routing for Unified Multimodal Models¶

Conference: ECCV2026
Paper: ECCV Paper
Project: SPAR
Area: Multimodal VLM
Keywords: unified visual tokenizer, semantic-pixel alignment, dynamic token routing, image generation, image editing

TL;DR¶

SPAR first reconciles semantic preservation and pixel reconstruction with an asymmetric dual-stream tokenizer, then conditions diffusion through token-wise cross-layer routing and internal representation alignment, reaching 0.91 on GenEval and 0.64 on WISE with SPAR-3B while largely preserving the original visual understanding capabilities.

Background & Motivation¶

Vision-language models recognize objects, understand relationships, and answer questions, but their visual encoders do not necessarily retain enough information to reconstruct an input image. Semantic learning encourages ignoring texture differences that do not affect categories or meaning, whereas generation and editing depend on precisely these local textures and spatial details. Conventional diffusion models therefore use reconstruction-oriented VAE latent spaces, while understanding models use high-dimensional semantic features; they do not naturally share the same visual representation. Methods such as RAE explore modeling directly in semantic spaces, but this paper argues that attaching a decoder to a frozen semantic encoder still tends to produce blur and structural distortions.

Fully unfreezing the encoder is not an immediate solution either: pixel reconstruction gradients can rewrite its discriminative structure and cause forgetting of understanding capabilities. If one transformation stream must both preserve abstract semantics and supply reconstruction details, the two objectives also compete for representational capacity. Moreover, supervising diffusion with an external DINO-like teacher may align generation to a space different from the model's own visual input interface. Finally, even with a better representation space, using only the final MLLM layer ignores different tokens' needs for shallower structural information and deeper semantics.

The paper consequently separates the construction of the representation space, extraction of generation conditions, and supervision of the generator into three connected components. It does not require every module to become one network; instead, it makes understanding features, reconstruction latents, and intermediate generation representations compatible through coordinated interfaces. Core Idea: preserve discriminative information with a lightweight semantic stream, learn reconstruction mappings with a heavier pixel stream, and guide generation with token-wise cross-layer conditions and representation supervision from the tokenizer itself.

Method¶

Overall Architecture¶

Inputs can be text alone or text with a reference image; understanding tasks produce text, while generation and editing tasks produce images. SPAR first trains a semantic-pixel dual-stream tokenizer, then uses its optimized visual encoder in InternVL3 and connects it to the SANA diffusion Transformer (DiT). Dynamic token routing constructs generation conditions from multiple hidden layers of the frozen MLLM, while internal representation self-alignment constrains intermediate DiT features during training. These connections are distinct: the former is an inference-time data path, whereas the latter is supervision that depends on real training images. Solid edges below represent conditioning or generation data flow, and dashed edges represent training supervision; the tokenizer's reconstruction training is explained afterward.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Text / reference image /<br/>training target image"] --> Tok["Semantic-pixel<br/>dual-stream tokenizer"]
    Input -->|Text and learnable queries| MLLM["Frozen MLLM<br/>multi-layer hidden states"]
    Tok -->|Reference image encoder features| MLLM
    MLLM --> DTR["Dynamic token routing"]
    DTR -->|Connector and conditional projection| DiT["DiT generation process"]
    Tok -.->|Training target encoder features| Align["Internal representation<br/>self-alignment"]
    DiT -.->|Intermediate features during training| Align
    DiT --> Output["Compact latents<br/>pixel decoder produces image"]

Key Designs¶

1. Semantic-pixel dual-stream tokenizer: allocate different capacities to semantic preservation and reconstruction adaptation

Both streams start from the semantic encoder's features; inference does not run separate semantic and pixel encoders and simply concatenate their outputs. The input image passes through the semantic encoder to produce spatially arranged high-dimensional features of dimension \(D_s\). The semantic stream uses only residual blocks and an MLP projection to map them into a smaller compact space of dimension \(D_p\), where \(D_p \ll D_s\). The implementation uses 3 residual blocks in this stream, seeking minimal semantic perturbation while changing the interface dimension rather than learning another reconstruction network. The pixel stream additionally uses a 6-layer Transformer to integrate spatial positions through global self-attention, followed by residual blocks and an MLP mapping to the same compact dimension. It receives more computational capacity because it must adapt discriminative features to the pixel decoder's native latent space.

The two compact features are concatenated along channels, projected back to the compact dimension by a fusion layer, and reconstructed through the DC-AE pixel decoder. Understanding uses the optimized visual encoder's features, while fused compact latents serve reconstruction and diffusion modeling; these are not the same layer output. Thus, a unified tokenizer does not mean using unmodified high-dimensional semantic features directly as diffusion latents. Asymmetry alone does not guarantee freedom from semantic forgetting: the semantic anchor also includes a frozen encoder copy and feature constraints during subsequent training. Likewise, recovering details should not be interpreted as losslessly inverting information that has already been strictly discarded; earlier stages learn a mapping, and the final stage fine-tunes the encoder to retain more pixel cues. This is an interpretation of the mechanism's boundary: the paper demonstrates improved reconstruction metrics, not information-theoretic invertibility.

2. Dynamic token routing: choose a different combination of MLLM layers for each position

Learnable query embeddings occupy image-generation positions and enter the MLLM together with text tokens and optional reference-image tokens. Queries toward the end of the sequence can access preceding instructions and reference images through causal self-attention, becoming implicit conditions for the target image. Different MLLM layers contain different abstraction levels; taking only the final layer or using identical fusion weights for every token loses this flexibility. DTR uniformly samples \(K\) layers from the hidden-state collection that includes the embedding output, with \(K=4\) by default, and computes cross-layer weights for each token. The routing network takes that token's feature from the deepest sampled layer because it has the fullest context for deciding what information the position needs. A temperature-controlled softmax produces layer weights, learnable per-layer scales calibrate feature magnitudes, and a final linear projection produces representations for the connector.

\[ w_i=\operatorname{softmax}\!\left(\frac{g(H_i^{(K)})}{\tau}\right),\qquad \widehat H_i=W_p\left(\sum_{k=1}^{K}w_i^{(k)}\alpha^{(k)}H_i^{(k)}\right). \]

Here \(H_i^{(k)}\) is token \(i\)'s feature at sampled layer \(k\), \(\alpha^{(k)}\) is a learnable layer scale, and the default temperature is \(\tau=1.0\). Routing operates along the layer dimension; it neither selects different generation experts nor claims to skip computation for unselected MLLM layers. Weights can differ across tokens, but the paper does not hard-code a rule such as assigning edges to shallow layers and semantics to deep layers. Weight visualization is proposed as a way to interpret layer preferences; the available main text does not provide quantitative analysis sufficient to establish specific preferences for token categories.

3. Internal representation self-alignment: supervise intermediate DiT layers with the optimized visual encoder

After dual-stream training, the visual encoder supports semantic preservation and pixel representation, so the authors use it as an internal teacher for generation training. During training, the target image provides teacher-side encoder features, while the DiT provides features from a designated intermediate layer during generation training. An MLP alignment head projects DiT features into a comparable space, and the objective maximizes their position-wise cosine similarity with teacher features. The paper expresses this as minimizing negative mean cosine similarity, where \(N\) is the number of aligned positions.

\[ \mathcal L_{\mathrm{align}}=-\frac{1}{N}\sum_{i=1}^{N} \cos\!\left(\phi_a(f_{\mathrm{dit},i}),f'_{v,i}\right). \]

Here \(f'_{v,i}\) denotes projected teacher-side encoder features, rather than directly using the fused dual-stream latent as the cosine target. Diffusion supervision and the visual understanding interface therefore come from the same optimized tokenizer, without adding a separate semantic feature network. However, the absence of an external teacher applies specifically to this generation-alignment stage, not to the entire system's dependence on pretrained modules or teacher copies. Tokenizer training still uses a frozen pixel encoder for latent anchors and a frozen semantic encoder copy for self-distillation targets. At inference time there is no real target image for alignment, so the model uses the learned DiT parameters and DTR conditioning path without computing this loss online.

A Worked Example¶

Consider the instruction in Figure 3 to add sunglasses to the pictured cat: the reference image first becomes input features through the optimized visual encoder. The instruction, reference-image tokens, and image-generation queries form one sequence, with queries aggregating the edit request and source-image semantics under causal attention. DTR combines features from 4 sampled layers for each position, and the connector conditions the DiT rather than directly predicting a sunglasses location or pixel mask. The DiT generates an edited-image latent in the compact space, which the pixel decoder reconstructs into a visible image. For a training example, the real edited image additionally supplies teacher features that constrain intermediate DiT representations; this supervision path is absent in actual use. This example explains the data flow, not an independently reported success rate or local-preservation score for that sample.

Loss & Training¶

There are two separate training schedules with 3 stages each; unfreezing the tokenizer's encoder must not be confused with unfreezing the language model during unified-model training. Tokenizer Stage I freezes the visual encoder and pixel decoder and trains only the dual-stream and associated fusion modules. Alongside pixel MSE and LPIPS reconstruction losses, it minimizes squared error between the pixel-stream latent and the frozen pixel encoder's output. This explicit pixel anchor first teaches the new stream to enter a space the existing decoder can handle, rather than letting both the decoder and mapping drift freely. Stage II keeps the visual encoder frozen, unfreezes the pixel decoder, and uses only MSE and LPIPS to adapt decoding to the fused latent distribution. Stage III unfreezes the visual encoder, constrains it by squared feature error against its frozen copy, and sets its learning rate to \(0.1\times\) the global learning rate. Reconstruction losses remain active, and a GAN loss is added after a certain number of steps; the cached main text does not specify that activation step or numerical loss weights.

Unified-model Stage I freezes both the MLLM and DiT, training the connector, conditional projection, query embeddings, and DTR with flow matching on generation data. Stage II keeps the MLLM frozen, jointly trains the connector and DiT, and adds weighted internal representation self-alignment to flow matching using mixed generation and editing data. Stage III retains the Stage II parameter configuration for supervised fine-tuning on high-quality instruction data. SPAR-1B uses InternVL3-1B, Qwen2.5-0.5B, and SANA-0.6B; SPAR-3B uses InternVL3-2B, Qwen2.5-1.5B, and SANA-1.6B. Generation data comprises BLIP3o's 27M recaptioned examples, a 5M subset of CC12M, and 4M synthetic images from JourneyDB. Editing training uses GPT-Image-Edit, while instruction tuning uses BLIP3o-60K and ShareGPT-4o-Image. The authors emphasize that no additional understanding-task training data is required, which does not mean the model lacks understanding knowledge inherited from InternVL3 pretraining.

Key Experimental Results¶

Main Results¶

Reconstruction results are from Table 1, page 10: the ImageNet 50k validation set at \(256\times256\) resolution; lower rFID and higher PSNR and SSIM are better. Ratio retains the paper's spatial compression ratio and should not be read as a standardized parameter count or compute budget.

Model	Type	Ratio	rFID ↓	PSNR ↑	SSIM ↑
RAE	Generation only	16	0.49	19.23	0.620
DC-AE	Generation only	32	0.69	23.85	0.660
VA-VAE	Generation only	16	0.28	27.96	0.790
DualToken	Unified	16	0.54	23.56	0.742
UniLIP	Unified	32	0.79	22.99	0.747
SPAR	Unified	32	0.27	26.65	0.856

SPAR leads these unified tokenizers on all three metrics, but does not dominate every generation tokenizer: VA-VAE's PSNR of 27.96 exceeds SPAR's 26.65. Text-to-image results are from Table 3, page 12 and report benchmark scores, not percentages; data and training budgets are not standardized across systems.

Model	GenEval Counting ↑	GenEval Position ↑	GenEval Overall ↑	WISE Overall ↑
FLUX.1-dev	0.75	0.68	0.82	0.50
Janus-Pro	0.59	0.79	0.80	0.35
BAGEL	0.81	0.64	0.82	0.52
OpenUni-L	0.77	0.75	0.85	0.52
SPAR-1B	0.83	0.85	0.89	0.57
SPAR-3B	0.84	0.87	0.91	0.64

Ablation Study¶

The following results come from Table 5, page 14; reconstruction again uses ImageNet 50k at \(256\times256\), while understanding scores come from the corresponding multimodal benchmarks. The full configuration's understanding scores match SPAR-1B in the main table and should not be presented as SPAR-3B ablations.

Config	rFID ↓	PSNR ↑	SSIM ↑	MME-P ↑	MMB ↑	MMVP ↑
Full model	0.27	26.65	0.856	1500	73.0	68.9
Without pixel stream	0.31	24.62	0.788	1499	72.6	68.7
Without semantic stream	0.29	25.28	0.804	709	18.4	50.0
Frozen encoder	6.14	16.26	0.572	1492	72.6	67.3

Removing the pixel stream reduces PSNR by 2.03, while removing the semantic stream lowers MMB from 73.0 to 18.4, supporting their respective roles in detailed reconstruction and semantic preservation. Freezing the encoder severely harms reconstruction, showing that subsequent fine-tuning matters; however, the text does not fully specify this ablation's per-stage budget, so one row cannot characterize every frozen-encoder strategy. Table 6, page 15 reports GenEval/WISE/ImgEdit of 0.89/0.57/3.85 for the full model, 0.86/0.54/3.73 without DTR, and 0.86/0.53/3.78 without self-alignment. The prose describes removing DTR as causing the largest overall drop, but this is not true for every metric: WISE drops more without self-alignment, and GenEval is identical for both removals.

Key Findings¶

Understanding is largely preserved, not individually lossless: Table 2, page 11 reports SPAR-1B MMMU of 43.2, slightly below InternVL3-1B's 43.4, while MME-P rises from 1492 to 1500.
Editing is strong: Table 4, page 14 reports ImgEdit Overall of 4.01 for SPAR-3B, 3.44 for OmniGen2, and 4.20 for GPT-4o.
Page 13 states a gain of +0.58 over OmniGen2, but subtracting the table values gives 0.57; both the original values and the conflict are retained rather than silently reconciled.

Highlights & Insights¶

Asymmetric capacity allocation is more targeted than simply stacking two encoders. Semantics primarily needs preservation and constraints, while reconstruction mappings need additional spatial computation, a distinction supported by the ablations.
Sharing the source of the alignment teacher and understanding interface is a transferable idea. It reduces interface differences introduced by an extra feature network, but does not establish that every distribution mismatch has been empirically eliminated.
DTR turns cross-layer fusion from global parameters into input-dependent, position-dependent decisions. It is relevant when connecting frozen backbones to generators, although the present evidence concerns performance ablations rather than causal proof of layer specialization.

Limitations & Future Work¶

The authors do not provide a dedicated systematic limitations section; the following restrictions primarily reflect the evidence available in the main text rather than explicitly acknowledged failures.
Training steps, complete loss weights, sampling budgets, hardware costs, and repeated-run uncertainty are missing, limiting full cost reproduction and assessment of the statistical meaning of small understanding gains.
Parameter columns differ between Tables 2 and 3: SPAR-3B is listed as 2B in the former and 3B in the latter; these should not be treated as standardized total parameter counts for efficiency comparisons.
Budget-matched external-teacher comparisons and thorough ablations of DTR layer count, temperature, and static multi-layer fusion are absent, so all gain sources cannot yet be separated.
Useful follow-up experiments would independently control the tokenizer, teacher source, and routing scheme, and report errors and latency for high-resolution details and preservation of non-target editing regions.

Compared with RAE: both value semantic representations for generation, but SPAR adds a compact dual-stream mapping and constrained encoder fine-tuning, focusing on reconstruction detail and understanding preservation.
Compared with REPA: both align intermediate generator features, but SPAR obtains its teacher from its own optimized tokenizer rather than an independent external discriminative encoder.
Compared with UniLIP: both target unified understanding, generation, and editing; SPAR emphasizes asymmetric responsibilities and token-wise cross-layer conditioning. Same-ratio reconstruction comparisons are promising but do not replace matched-data experiments.
A research extension: using semantic-preservation metrics to control encoder update magnitude might be more robust than a fixed \(0.1\times\) learning rate; this is a reader hypothesis, not a method validated in the paper.

Rating¶

Novelty: 4/5. Asymmetric dual-stream capacity, an internal teacher, and token-wise routing form a coherent design, although their underlying mechanisms have precedents.
Experimental Thoroughness: 4/5. Reconstruction, understanding, generation, and editing are covered with key ablations, but costs, randomness, and finer controls are missing.
Writing Quality: 3/5. The central argument is clear, but there are arithmetic discrepancies, inconsistent parameter reporting, and claims stronger than the tables support.
Value: 4/5. The paper offers an empirically supported design for adding generation while retaining existing MLLM capabilities.