Towards Scalable Pre-training of Visual Tokenizers for Generation¶

Conference: ECCV2026
Paper: ECCV Paper
Authors: Jingfeng Yao, Yuda Song, Yucong Zhou, Xinggang Wang
Area: Visual Tokenizer / Image Generation
Keywords: joint pretraining, image-text contrastive learning, self-supervised learning, latent space, diffusion model

TL;DR¶

VTP trains a visual tokenizer for reconstruction, visual semantics, and image-text alignment together, allowing additional tokenizer pretraining to improve downstream diffusion generation and achieving guided gFID 1.11 on ImageNet 256×256 with a 675M generator trained for 600 epochs.

Background & Motivation¶

Latent diffusion models first compress images with a visual tokenizer, learn the distribution of the compressed representations, and finally reconstruct pixels with a decoder. This division reduces the spatial resolution seen by the generator, but it also lets the first stage determine which representation the second stage must model. Conventional autoencoders receive supervision mainly through reconstruction error and therefore prioritize textures, colors, and local details. Preserving those details more accurately does not necessarily organize objects, layouts, and semantics into a latent space that is easier to generate. The paper provides direct evidence: continued reconstruction-only tokenizer training improves reconstruction rFID while worsening downstream generation gFID under a fixed training budget.

Previous methods already introduce semantic priors: VA-VAE aligns with vision foundation features, REPA aligns representations during diffusion training, and RAE directly uses a pretrained representation encoder. These findings establish that perception features are useful, but do not automatically answer how tokenizer training can keep benefiting from more data, larger models, and more compute. Simply attaching a fixed foundation model to a generator leaves the representation constrained by an existing pretraining objective. Conversely, scaling reconstruction alone may spend compute on increasingly fine low-level variation without making downstream modeling easier. The central problem is therefore not further reducing token counts, but making tokenizer pretraining compatible with generation.

VTP treats the visual tokenizer as an independently scalable representation learner while preserving pixel reconstruction. Image-text contrastive learning provides global semantic alignment, self-supervised learning provides cross-view consistency and local spatial perception, and reconstruction keeps the representation decodable. The authors hold the downstream DiT configuration fixed to test whether changing the first stage alone improves generation. Core Idea: first train a generation latent space for both semantic structure and reconstructability, then scale tokenizer pretraining instead of expecting lower reconstruction error to automatically yield better generation.

Method¶

Overall Architecture¶

VTP uses an all-ViT autoencoder: an image passes through an encoder and a low-dimensional bottleneck to produce spatial latents, which a pixel decoder reconstructs into an image. The tokenizer here encodes continuous visual representations; it is not necessarily a text-like tokenizer with a discrete codebook. Training uses image-text pairs, with global, local, and masked image views supplying different forms of supervision. The pipeline applies heterogeneous batch sampling, joint perception-reconstruction pretraining, and decoder refinement with a frozen encoder to improve pixel fidelity. The resulting tokenizer and downstream diffusion model are trained separately, so DiT gradients do not update the encoder.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    INPUT["Image-text pairs and image views"] --> BATCH["Heterogeneous batch sampling"]
    BATCH --> JOINT["Joint perception-reconstruction<br/>pretraining"]
    JOINT --> REFINE["Decoder refinement<br/>with a frozen encoder"]
    REFINE -.->|Frozen encoder supplies latents| TRAIN["Downstream DiT training"]
    TRAIN --> SAMPLE["Diffusion sampling produces latents"]
    SAMPLE --> OUTPUT["Pixel decoding and generated image"]
    REFINE -.->|Refined decoder| OUTPUT

Joint pretraining contains supervision paths through a text encoder, an EMA teacher, and a pixel decoder, rather than a sequence of inference modules. The text encoder and EMA teacher shape the representation; they are not components that must be executed sequentially to generate each new image. At generation time, the trained DiT samples latents and the pixel decoder renders them, without an original image to reconstruct. Reconstruction and generation evaluation therefore share a decoding capability but obtain their input latents differently.

Key Designs¶

1. Heterogeneous batch sampling: give each objective an appropriate sample budget

Image-text contrastive learning relies on in-batch negatives and usually benefits from large batches that distinguish matching and nonmatching pairs. Self-supervised learning processes multiple crops, while reconstruction additionally runs a pixel decoder, making equally large batches expensive for these objectives. VTP therefore keeps a shared large batch of image-text pairs, uses every pair for CLIP, and randomly selects subsets for self-supervision and reconstruction. This allocates different compute budgets over a shared data source rather than preparing entirely separate datasets for the three tasks. The main scaling experiments use a CLIP batch of 16k, a self-supervised batch of 4k, and a reconstruction batch of 2k. These settings do not imply that every image receives all three forms of supervision in every training step.

The arrangement allows semantic learning to see more samples without increasing decoding cost proportionally. It also shows that joint training requires appropriate conditions for each objective, not merely adding loss terms together. The dedicated sampling ablation in Table 3 on page 14 instead uses default batches of 8k for CLIP and 1k for reconstruction; these must not be conflated with the main settings. Its CLIP batch-size results support stronger semantic supervision, but do not establish that every objective benefits from indefinitely larger batches.

2. Joint perception-reconstruction pretraining: learn global and local semantics in one decodable representation

Encoder outputs pass through a low-dimensional bottleneck rather than exposing unchanged high-dimensional perception features to the generator. Section 3.2 specifies the image and latent spaces as follows:

\[ I\in\mathbb{R}^{3\times H\times W},\qquad Z\in\mathbb{R}^{d\times H/16\times W/16}. \]

The downsampling factor is 16; the main analysis uses bottleneck dimension 64 and also tests dimension 256 to check dependence on representation capacity. The decoder lifts low-dimensional latents into feature space, processes them with ViT blocks, and restores the pixel arrangement using pixel shuffle. The reconstruction branch uses L1 and perceptual losses to preserve specific appearance rather than retaining only category-level information. GAN loss is excluded from pretraining because the authors observed large gradient norms and instability when combining it with this ViT architecture.

The image-text branch compares visual and textual features across a batch in the CLIP manner. Matching pairs are brought together and nonmatching combinations are distinguished, requiring image representations to encode recognizable semantic content from captions. This is a global cross-modal constraint, not direct supervision assigning a word to every pixel location. The text-to-image experiments further associate CLIP training with improved text rendering; this is an empirical finding, not evidence of dedicated OCR supervision.

The self-supervised branch follows DINOv2 by combining masked image modeling with self-distillation. For masked image modeling, an EMA teacher processes an unmasked global view, while the student encoder processes its masked version and learns missing-region representations from teacher signals. For self-distillation, the student processes global and local views and the teacher processes global views, with cross-entropy between pseudo-label predictions enforcing cross-view consistency. The student must therefore make local observations compatible with global understanding, rather than only recognizing the whole image. The EMA teacher is updated during training and should not be described as a fixed external DINOv2 model copied directly into the latent space.

These objectives supply complementary information, but arbitrary weights are not guaranteed to work equally well. The paper sets the reconstruction weight to 0.1 and the self-supervised and contrastive weights to 0 or 1 in corresponding ablations, testing individual and combined perception objectives. Lower reconstruction weights improve generation at some cost to reconstruction fidelity, so concurrent improvement does not imply that the objectives never compete. Operators in Equations (1) through (3) are incompletely extracted in the cache; this note explains their components from readable prose instead of presenting repaired expressions as exact author equations.

3. Decoder refinement with a frozen encoder: improve fidelity without disturbing the latent space

After joint pretraining, the authors freeze the visual encoder and fine-tune only the pixel decoder with a GAN objective. This postpones potentially unstable adversarial training until the latent space has formed while allowing the decoder to improve textures and visual realism. Since the encoder is frozen, decoder refinement does not change the latent representation that the downstream model must learn by updating the encoder. Its role is to improve output pixels, not to redefine semantic structure through an adversarial objective.

This stage also explains why reconstruction scores alone cannot establish generative potential. A decoder can improve its conversion of supplied representations into pixels while the generator must still learn to produce meaningful latents from noise. VTP's central benefit comes from shaping representations in the preceding stage, with decoder refinement ensuring that those representations yield high-quality images. Separating these roles distinguishes better image representation from better rendering of a representation.

A Worked Example¶

For a 256×256 input with the f16d64 configuration used in the main analysis, compression produces 16×16 spatial positions with 64 channels each. The image must be reconstructed when included in the reconstruction sub-batch, while its caption supplies paired semantics in the larger image-text batch. Different crops and masked views from the same data source provide self-supervised inputs to the teacher and student. These objectives train the encoder jointly; the pipeline does not generate text, then local images, and finally assemble the original image. Once the encoder is frozen, the downstream DiT learns the distribution of these 16×16 spatial latents, samples them from noise, and decodes them into an image. This example explains f16d64 tensor dimensions, not a universal 64-channel setting: the text-to-image curves in Figure 8 are labeled f16d24.

Loss & Training¶

The main pretraining dataset is an internally filtered DataComp-1B subset containing 277M samples. The model uses QKNorm for stability; its text encoder is a 12-layer Transformer with hidden dimension 768. Fast scaling experiments use a ViT-L encoder and a 4-layer ViT-L pixel decoder, while final scaling experiments also study symmetric VTP-S, VTP-B, and VTP-L architectures. The standard diagnostic protocol fixes LightningDiT-B, trains it on ImageNet for 80 epochs, and reports FID-10k. Understanding is measured by ImageNet Top-1 linear probing of bottleneck features without the multi-layer feature enhancements commonly used in the DINO family. This more directly evaluates the low-dimensional representation actually received by the generator, rather than the strongest features elsewhere in the encoder. Table 4 separately compares larger generators and longer training schedules, so its best gFID is not a result from the standard small-model diagnostic protocol.

Key Experimental Results¶

Main Results¶

The following selection comes from Table 4 on page 15 and evaluates ImageNet 256×256 generation; lower rFID and gFID are better. Parameters refer to the generator and epochs to downstream generator training; "Not reported" does not mean zero or an unsupported setting.

Method	Generator parameters	Epochs	rFID	Unguided gFID	Guided gFID
VA-VAE	675M	80	0.28	4.29	Not reported
REPA-E	675M	80	0.28	3.46	1.67
RAE	675M	80	0.57	4.28	Not reported
RAE	835M	80	0.57	2.16	Not reported
VTP	675M	80	0.36	2.62	1.44
VTP	1.0B	80	0.36	2.03	Not reported
VTP	675M	600	0.36	1.85	1.11

VTP's reconstruction rFID of 0.36 is worse than VA-VAE's 0.28, but its unguided generation gFID is lower in the corresponding 675M, 80-epoch rows. This shows why reconstruction cannot replace downstream generation evaluation, rather than establishing that VTP leads on every individual metric. Table 4 also reports VTP ImageNet zero-shot accuracy of 78.2% and linear probing accuracy of 85.7%. The 1.0B model's 2.03 should not be interpreted as an improvement over 675M models at identical generation compute. The prose on page 14 reports 2.60, the Figure 9 caption reports 2.61, and Table 4 reports 2.62; this note uses Table 4 for the 675M, 80-epoch result while retaining the inconsistency.

Ablation Study¶

The following data-scale analysis comes from Figure 7(a) on page 11 and Section 4.5 on page 12, not another final-model leaderboard. It compares VTP-ViT-Large and AE-ViT-Large, processes 1.1 billion samples per pretraining configuration, and holds downstream DiT training FLOPs fixed. Dataset sizes denote distinct-image subset sizes rather than cumulative sample exposures.

Pretraining subset size	Reconstruction-only AE gFID	VTP gFID
100K	58.37	47.59
1M	58.39	40.57
10M	57.23	30.98
100M	56.71	27.45

At fixed cumulative exposure, VTP benefits substantially from more distinct images, whereas AE improves only slightly. This design partly separates data diversity from simply training longer, since smaller datasets are revisited more often. These endpoints cannot be compared directly with Table 4's 1.11 because downstream models and training configurations differ.

Key Findings¶

Figure 4 on page 8 and the prose on page 10 show that increasing reconstruction-only training changes gFID from 55.04 to 58.56, worsening generation.
The three-objective configuration in Figure 6 on page 10 reaches gFID 27.8 and linear probing accuracy 74.9%, supporting complementary perception objectives.
Page 12 reports that scaling the VTP encoder improves gFID from 31.28 to 26.12, with decoder scaling further reaching 24.08.
Figure 8 reports faster convergence and better text rendering on LAION, but the cache lacks a reliable complete table of endpoint values, so exact benchmark gains are not inferred.

Highlights & Insights¶

The object of scaling is moved upstream to the tokenizer. Holding downstream configurations fixed helps distinguish its contribution from generator scaling itself.
Understanding is evaluated directly at the bottleneck. This tests the representation actually used by the generator rather than extra information available to an auxiliary classifier.
Semantic learning and reconstruction are not treated as mutually exclusive routes. Reconstruction preserves details that direct perception-encoder transfer can miss, while semantic objectives reshape the latent space.
Heterogeneous batch sampling is a reusable engineering choice. It lets expensive decoding objectives use smaller batches without sacrificing the negative-sample scale needed by contrastive learning.

Limitations & Future Work¶

The authors demonstrate empirical scaling trends within tested configurations, not a universal law proven for arbitrary data and generators.
Fixed downstream FLOPs do not mean fixed total cost. Larger tokenizers and additional pretraining should be assessed against reuse frequency and the total training budget.
The internally filtered 277M dataset raises a reproducibility barrier. The cache does not provide enough detail to reconstruct the complete filtering outcome.
Correlation between understanding and generation does not establish a single causal mechanism. Further work could isolate category semantics, spatial structure, and low-level frequency content.
The paper primarily covers image generation; the supplied full text does not establish the same trend for video or all unified multimodal tasks.
Reporting contains the 2.60, 2.61, and 2.62 discrepancy above; damaged equations and Figure 8 curves should not be the sole basis for exact reproduction settings.

Compared with VA-VAE: both address the reconstruction-generation tension, but VTP emphasizes learning representations through scalable joint pretraining rather than only aligning them with existing vision features.
Compared with RAE: RAE uses an existing perception encoder and trains a decoder; VTP includes semantic learning and reconstruction in encoder pretraining, allowing suitability for generation to influence representation formation.
Compared with REPA / REPA-E: these methods use representation alignment or coupled adaptation during diffusion training, whereas VTP's core experiments explicitly separate tokenizer pretraining from DiT training.
Research direction: compare extra tokenizer compute against extra DiT compute at equal total training FLOPs, then examine amortization when one tokenizer serves multiple downstream models; this is a reader proposal, not an experiment completed in the paper.

Rating¶

Novelty: 4/5. The contribution centers on generation-oriented joint pretraining and systematic scaling analysis rather than a fundamentally new individual loss.
Experimental Thoroughness: 4/5. Compute, parameter, data, and text-to-image studies are covered, but total-cost analysis and reproduction details remain incomplete.
Writing Quality: 4/5. The problem and controlled experiments are clear, although inconsistent result values complicate precise citation.
Value: 5/5. It offers a concrete, testable direction for independently pretraining reusable visual tokenizers.