WinTok: A Win-Win Hybrid Tokenizer via Decomposing Visual Understanding and Generation with Transferable Tokens¶
Conference: ECCV 2026
Paper: ECCV Official Link
Code: https://github.com/WeChatCV/WinTok
Area: Model Compression
Keywords: Unified Tokenizer, Hybrid Encoding, Transferable Tokens, Asymmetric Distillation, Unified Multimodal Models
TL;DR¶
WinTok presents a hybrid visual tokenizer that incorporates learnable semantic tokens into a unified ViT encoder alongside asymmetric distillation from foundation models, fundamentally decoupling visual understanding from generation to attain an 82.0% ImageNet-1K classification accuracy and an rFID of 0.41 using only 50M training images.
Background & Motivation¶
Building unified multimodal models (UMMs) under the next-token prediction paradigm is a central aspiration for general artificial intelligence. Nevertheless, jointly accommodating visual understanding and visual generation introduces a fundamental representational dilemma. Visual understanding fundamentally favors high-level, continuous, and semantically rich feature representations that align smoothly with textual spaces. In stark contrast, visual generation relies predominantly on low-level, discrete, and spatially dense tokens that preserve high-frequency pixel details and local texture patterns. Early unified multimodal systems (such as Janus and Bagel) circumvent this tension by deploying dedicated dual encoders—one semantic encoder for perception and one pixel encoder for synthesis. Although functional, this approach substantially inflates parameter counts, incurs heavy computational latency, and undermines the architectural unity of multimodal systems.
To eliminate structural redundancy, recent endeavors have explored single-encoder unified visual tokenizers (e.g., UniTok and VILA-U). These architectures compel a single latent space to simultaneously optimize pixel reconstruction objectives and high-level semantic alignment objectives. Unfortunately, such monolithic coupling inevitably triggers severe optimization conflict: pushing the encoder toward fine-grained pixel reconstruction ruins semantic abstraction and discriminative class boundaries, whereas enforcing semantic alignment blurs local pixel fidelity and degrades visual generation quality. Consequently, existing unified tokenizers exhibit compromised discriminative capability on ImageNet classification (e.g., UniTok achieves only 70.8%), demonstrating a steep performance penalty compared to specialized perception backbones.
This paper breaks through this trade-off by reconsidering the decomposition of roles within a single backbone network. Rather than splitting the model into separate physical encoders, the authors propose a hybrid tokenization strategy that decouples task representations at the token sequence level. Core idea: supplement image patch tokens with a set of learnable semantic tokens within a unified ViT encoder, allowing cross-token attention to naturally exchange context while guiding the semantic tokens via asymmetric distillation from visual foundation models, thereby enabling pixel tokens to focus on detail reconstruction and semantic tokens on high-level comprehension.
Method¶
Overall Architecture¶
WinTok adopts a unified encoder dual latent architecture. Given an input image, it is first patchified and mapped into local pixel tokens, while an additional set of learnable semantic tokens is introduced. These two token sets are concatenated along the sequence dimension and fed into a single Vision Transformer (ViT) encoder. Within self-attention layers, pixel tokens supply fine-grained details to semantic tokens, while semantic tokens provide global contextual cues to pixel tokens. Upon exiting the encoder, the two representations follow distinct paths: pixel tokens undergo multi-codebook vector quantization (MCQ) and are decoded by a ViT decoder for high-fidelity image reconstruction, whereas semantic tokens are globally pooled and supervised by pre-trained foundation model embeddings via an asymmetric distillation loss.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Image X"] --> B["Input Decomposition & Token Concatenation<br/>Patch Embedding + M Learnable Tokens"]
B --> C["Unified Encoder Interaction<br/>Joint Self-Attention ViT Backbone"]
C --> D["Dual Latent Token Routing"]
D --> E["Pixel Token Quantization & Reconstruction<br/>MCQ Quantization + ViT Decoder"]
D --> F["Semantic Token Asymmetric Distillation<br/>Global Pooling + Teacher Supervision"]
E --> G["Reconstructed Image X_hat"]
F --> H["Multimodal Downstream Applications"]
Key Designs¶
1. Input Decomposition and Token Concatenation: Eliminating Single-Representation Bottlenecks
Existing single-encoder architectures force every visual patch to serve both dense reconstruction and abstract classification, which inevitability causes representational interference. WinTok projects the input image \(\mathbf{X}\) into patch-level pixel tokens \(\mathbf{P}^o = \{\mathbf{P}_1^o, \dots, \mathbf{P}_N^o\}\), and appends \(M\) learnable vectors \(\mathbf{S}^o = \{\mathbf{S}_1^o, \dots, \mathbf{S}_M^o\}\) (default \(M=256\)). The combined sequence is processed simultaneously by the unified encoder \(\mathcal{E}\): $\([\mathbf{P}, \mathbf{S}] = \mathcal{E}([\mathbf{P}^o, \mathbf{S}^o])\)$ This mechanism preserves the structural compactness and parameter sharing of a single ViT encoder while establishing independent latent channels for distinct objectives, freeing pixel tokens to specialize in local textures and empowering learnable tokens to aggregate holistic visual concepts.
2. Asymmetric Token Distillation: Transferring Foundation Discriminative Power Without Inference Overhead
To equip the randomly initialized tokens \(\mathbf{S}\) with rich semantic understanding, WinTok introduces an asymmetric token distillation mechanism. A pre-trained visual foundation model \(\mathcal{T}\) (e.g., SigLIP2-So400M) serves as an off-the-shelf teacher network to extract \(K\) semantic tokens \(\mathbf{T}\) from the input image. Because the student token length \(M\) differs from the teacher sequence length \(K\), WinTok performs global average pooling across both sequences to obtain unified global semantic vectors \(\mathbf{s} = \text{Pool}(\mathbf{S})\) and \(\mathbf{t} = \text{Pool}(\mathbf{T})\), optimizing them using a cosine distance loss: $\(\mathcal{L}_{sem} = 1 - \frac{\mathbf{s} \cdot \mathbf{t}}{\|\mathbf{s}\|_2 \|\mathbf{t}\|_2}\)$ This asymmetric pooling strategy decouples token sequence lengths, enabling arbitrary pre-trained vision models (such as CLIP, DINOv2, or SigLIP2) to supervise the learnable tokens without structural modification. During downstream inference, the teacher model is discarded entirely, ensuring zero additional computational cost.
3. Pixel Token Quantization and Reconstruction: Preserving Fine-Grained Fidelity via Multi-Codebook Vector Quantization
The contextualized pixel tokens \(\mathbf{P}\) are dedicated exclusively to image synthesis and are exempt from semantic loss gradients. A multi-codebook quantization (MCQ) module converts continuous tokens into discrete indices \(\mathbf{Q} = \mathcal{Q}(\mathbf{P})\), which are subsequently fed into a ViT decoder \(\mathcal{D}\) to reconstruct the image \(\hat{\mathbf{X}} = \mathcal{D}(\mathbf{Q})\). The quantizer and decoder are optimized end-to-end using straight-through estimator (STE) gradient approximation, combined with pixel-level \(L_1\) loss, codebook alignment loss, perceptual loss \(\mathcal{L}_{per}\), and adversarial patch loss \(\mathcal{L}_{adv}\) to enforce crisp texture synthesis.
Loss & Training¶
The overall training objective combines the generation and understanding losses in an end-to-end manner: $\(\mathcal{L} = \mathcal{L}_{pix} + \lambda_{sem} \mathcal{L}_{sem}\)$ where the pixel objective is formulated as: $\(\mathcal{L}_{pix} = \|\mathbf{X} - \hat{\mathbf{X}}\|_1 + \beta \|\mathbf{P} - \text{sg}[\mathbf{Q}]\|_2^2 + \lambda_{per}\mathcal{L}_{per} + \lambda_{adv}\mathcal{L}_{adv}\)$ Here \(\text{sg}[\cdot]\) represents the stop-gradient operator. The encoder is initialized from SigLIP2-So400M, and the decoder is trained from scratch. WinTok is trained on a 50M subset randomly drawn from open-source datasets (ImageNet-1K, COYO-700M, CC12M, etc.) for 5 epochs with a global batch size of 256, utilizing a learning rate of \(2 \times 10^{-4}\) with cosine decay.
When integrated into downstream unified multimodal modeling, WinTok pairs with Qwen3-8B: continuous semantic tokens \(\mathbf{S}\) are mapped through a lightweight linear projection to feed into the autoregressive LLM for visual question answering, while discrete tokens \(\mathbf{Q}\) are modeled autoregressively for conditioned image generation.
Key Experimental Results¶
Main Results¶
On the ImageNet-1K validation benchmark (\(256 \times 256\) resolution), WinTok demonstrates a strong balance between reconstruction fidelity (rFID) and high-level classification accuracy.
| Tokenizer Category / Method | Downsampling Ratio | Training Data Scale | Codebook Capacity | rFID ↓ | Top-1 Accuracy (%) ↑ |
|---|---|---|---|---|---|
| Semantic Only | |||||
| CLIP-L/14 | - | WIT-400M | - | - | 75.5 |
| DINOv2-L | - | LVD-142M | - | - | 86.3* |
| SigLIP2-So/16 | - | WebLI-10B | - | - | 83.4 |
| Reconstruction Only | |||||
| LlamaGen | 16 | IN-1K | \(2^{14}\) | 2.19 | - |
| WeTok | 16 | IN-1K | \(2^{32}\) | 0.61 | 18.3 |
| MGVQ | 16 | IN-1K | \(2^{88}\) | 0.49 | - |
| Unified Tokenizers | |||||
| VILA-U | 16 | CY-700M | \(2^{14}\) | 1.80 | 73.3 |
| QLIP-L | 16 | DC-1B | - | 1.46 | 79.1 |
| DualToken | 16 | CC-12M | - | 0.54 | 81.6 |
| TokenFlow | 16 | LA+CY-700M | \(2^{15}\) | 1.37 | - |
| UniTok | 16 | DC-1B | \(2^{96}\) | 0.41 | 70.8 |
| WinTok (Ours) | 16 | Mix-50M | \(2^{48}\) | 0.41 | 82.0 |
Furthermore, when evaluated in downstream multimodal LLM settings, WinTok consistently outperforms existing unified tokenizers across visual comprehension and generation benchmarks:
| Benchmark / Task | Metric | UniTok (7B) | TokenFlow-L (13B) | WinTok (8B) | Gain |
|---|---|---|---|---|---|
| POPE (Hallucination) | Accuracy (%) | 83.2 | 85.0 | 86.5 | +3.3% vs UniTok |
| TextVQA | Accuracy (%) | 51.6 | 54.1 | 55.2 | +3.6% vs UniTok |
| MMBench | Score | - | 60.3 | 74.9 | +14.6% vs TokenFlow |
| GQA | Accuracy (%) | 61.1 | 60.3 | 62.4 | +1.3% vs UniTok |
| GenEval (T2I Alignment) | Overall Score | 0.59 | 0.55 | 0.76 | +0.17 vs UniTok |
| DPG-Bench | Overall Score | 81.18 | 73.38 | 83.36 | +2.18 vs UniTok |
Ablation Study¶
The authors perform thorough ablations to dissect the design choices, including task-role inversion, token capacity scaling, and teacher foundation model variations:
| Configuration / Variant | rFID ↓ | Top-1 Acc (%) ↑ | Core Findings / Note |
|---|---|---|---|
| WinTok (Default, M=256) | 0.41 | 82.0 | Full model: pixel tokens for reconstruction, learnable tokens for semantics |
| LoseTok (Role Inversion) | 27.39 | 79.2 | Swapping roles: learnable for reconstruction, pixel for semantics; reconstruction collapses |
| Learnable Tokens M=50 | ~0.60 | ~79.0 | Insufficient semantic token capacity restricts gradient decoupling |
| Learnable Tokens M=100 | ~0.53 | ~80.2 | Performance scales smoothly with the number of semantic tokens |
| Teacher = CLIP-L/14 | - | 77.2 | Viable semantic distillation, but lower downstream performance than SigLIP2 |
| Teacher = DINOv2-L | - | 80.8 | Strong visual features, but lacks explicit language-grounded alignment |
| Decoder Size = ViT-B | 0.60 | 82.0 | Smaller decoder degrades fine reconstruction details while preserving semantic accuracy |
Key Findings¶
- Role Inversion Catastrophe (LoseTok): Inverting the assignments by forcing learnable tokens to undergo vector quantization for image reconstruction and using pixel tokens for semantic distillation causes rFID to collapse catastrophically from 0.41 to 27.39. Learnable tokens lack localized spatial inductive biases essential for dense pixel synthesis, confirming that spatial grid tokens must handle reconstruction while global tokens absorb semantic supervision.
- Scaling Learnable Token Quantity: Expanding learnable tokens from 50 to 256 systematically boosts classification accuracy from 79.0% to 82.0% and improves reconstruction rFID from 0.60 to 0.41. Ample semantic capacity allows the ViT encoder to route high-level gradients away from pixel representations, avoiding mutual interference.
- Data and Codebook Efficiency: Compared with UniTok, which consumes 1 billion DataComp images and \(2^{96}\) codebook combinations, WinTok matches its 0.41 rFID and surpasses its classification accuracy by 11.2% (82.0% vs. 70.8%) using a substantially smaller \(2^{48}\) codebook and only 50M training samples.
Highlights & Insights¶
- Streamlined Architecture Without Dual Encoders: Rather than maintaining two disparate encoder backbones or constructing complicated multi-stage training pipelines (e.g., VQRAE), WinTok accomplishes clean task orthogonalization simply by appending learnable tokens into standard ViT self-attention blocks.
- Asymmetric Distillation as a Universal Knowledge Conduit: Global average pooling neatly resolves sequence-length discrepancies between student and teacher models, allowing WinTok to effortlessly digest pre-trained knowledge from arbitrary foundation vision models (CLIP, DINOv2, SigLIP2) without structural re-engineering.
- Insights for Unified Multimodal Design: The results reveal that performance drops in unified multimodal models on perception tasks do not stem from the capacity limits of the LLM backbone, but from degraded visual tokens produced by conflicted tokenizers. Supplying cleanly decoupled semantic tokens instantly elevates downstream reasoning metrics.
Limitations & Future Work¶
- Acknowledged Limitations: WinTok is pre-trained on a modest 50M open-source dataset, leaving billion-scale scaling behaviors unexplored. In addition, downstream unified modeling is primarily validated on Qwen3-8B without extensive exploration across diverse LLM families.
- Identified Research Constraints: The visual generation branch remains constrained to fixed-resolution (\(16 \times 16\)) discrete token grids, lacking native support for dynamic resolutions or arbitrary aspect ratios. Furthermore, global average pooling in semantic distillation abstracts away spatial coordinates, which may slightly diminish fine-grained bounding-box localization capabilities.
- Future Directions: Developing variable-length semantic token allocation schemes, exploring localized multi-level feature distillation, and generalizing the hybrid tokenization paradigm to unified video and 3D multimodal synthesis.
Related Work & Insights¶
- vs. UniTok: UniTok coerces a single discrete codebook to serve both pixel reconstruction and contrastive alignment, resulting in compromised semantic understanding (70.8% ImageNet accuracy). WinTok introduces transferable semantic tokens to achieve complete objective decoupling, boosting accuracy to 82.0% and maintaining a 0.41 rFID with 95% less training data.
- vs. Janus / Bagel: Standard decoupled paradigms deploy separate physical encoders for understanding and generation, leading to redundant parameter overhead and disjoint feature spaces. WinTok unifies the encoder backbone and allows cross-token contextual interaction during encoding.
- vs. VQRAE: While VQRAE also explores hybrid concepts, it relies on a complex, multi-stage training recipe. WinTok provides a simpler, one-stage end-to-end framework powered by asymmetric foundation model distillation.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ (Simple yet highly effective architectural decoupling using learnable tokens and asymmetric distillation)
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Evaluated across 10 diverse benchmarks with comprehensive ablations, LoseTok sanity checks, and t-SNE visualizations)
- Writing Quality: ⭐⭐⭐⭐⭐ (Clear logical progression, mathematically sound formulations, and intuitive diagrams)
- Value: ⭐⭐⭐⭐⭐ (Offers a robust, highly efficient tokenizer foundation for next-generation unified multimodal systems)