Let ViT Speak: Generative Language-Image Pre-training¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/YanFangCS/GenLIP
Area: Multimodal VLM
Keywords: Vision-Language Pre-training, Generative Pre-training, Vision Transformer, Gated Attention, Multimodal Large Language Models

TL;DR¶

GenLIP dispenses with contrastive dual encoders and cascaded auxiliary text decoders by training a single Vision Transformer to predict language tokens directly from visual prefix sequences via autoregressive next-token prediction, matching or surpassing SigLIP2 with only one-fifth of the pretraining data.

Background & Motivation¶

Multimodal Large Language Models (MLLMs) typically consist of three primary building blocks: a vision encoder for visual perception, a modality connector, and an autoregressive large language model (LLM) serving as the central reasoning engine. Among them, the visual representations extracted by the vision encoder fundamentally establish the performance ceiling of downstream multimodal reasoning. Conventional vision-language pre-training (VLP) approaches are heavily dominated by dual-encoder contrastive frameworks such as CLIP and SigLIP. However, contrastive objectives focus exclusively on global discriminative alignment across separate modality spaces, which suffers from an inherent objective mismatch when integrated into downstream generative LLMs driven by next-token prediction. Diagnostic perplexity analyses reveal that vision encoders trained under discriminative contrastive objectives yield noticeably higher generation perplexity when attached to LLMs than encoders trained under generative objectives.

To reconcile this cross-paradigm gap, generative VLP frameworks such as CapPa, AIMv2, and OpenVision2 connect an image encoder to an auxiliary text decoder, supervising the vision backbone indirectly through caption generation losses. Furthermore, hybrid architectures such as CoCa and SigLIP2 introduce additional text encoders to simultaneously optimize contrastive and captioning objectives. Nevertheless, cascading multiple towers and balancing disparate objectives complicates the optimization landscape and forces gradients to flow indirectly through auxiliary text modules, substantially constraining pretraining efficiency and architectural simplicity.

This paper advocates a return to a minimalist design philosophy: rather than engineering complex multi-tower networks, one should let the Vision Transformer "speak" directly. Core idea: train a single unified Transformer to jointly model visual patch tokens and text tokens using exclusively a standard autoregressive language modeling loss from scratch, introducing a lightweight gated attention mechanism to eliminate multimodal attention sinks and spatial representation collapse.

Method¶

Overall Architecture¶

GenLIP establishes a unified generative modeling paradigm for vision encoder pretraining. Given an image-text pair, the image is decomposed into non-overlapping patches and projected into patch embeddings via a standard convolutional layer, while the text caption is parsed into subword tokens using an off-the-shelf text tokenizer. These two streams are concatenated along the sequence dimension into a visual-prefix sequence and processed through a single Transformer backbone. During pretraining, the model is trained strictly with an autoregressive language modeling objective over the text tokens. When deployed as a vision encoder in downstream MLLMs, the language-specific components (the text tokenizer and the LM head) are discarded, and the backbone reduces to standard bidirectional full attention to output rich visual features into the downstream projector.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Paired image-text data"] --> B["Multimodal Sequence Concatenation<br/>Visual patch prefix + Text token sequence"]
    B --> C["Unified Prefix-LM Attention<br/>Bidirectional for vision + Causal for text"]
    C --> D["Gated Attention Mechanism<br/>Per-head input-adaptive gating to prevent sink"]
    D --> E["Two-Stage Pretraining Strategy<br/>S1 fixed 224x224 + S2 native aspect ratio adaptation"]
    E --> F["Downstream Vision Feature Extraction<br/>Discard LM head, standard full attention output"]

Key Designs¶

1. Unified Prefix-LM Architecture: Eliminating Auxiliary Towers and Decoders

Traditional generative VLP systems stack an auxiliary text decoder on top of an image encoder, which forces gradients to backpropagate through deep text layers and delays cross-modal interaction. GenLIP directly concatenates image patch embeddings and text token embeddings into a single unified sequence \(S = [v_0, \dots, v_M, t_0, \dots, t_L]\). To accommodate mixed spatial and temporal structures without absolute position bias, GenLIP discards absolute position embeddings in favor of Multimodal Rotary Position Embedding (MRoPE). The attention mechanism is configured with Prefix-LM masks: visual tokens attend to each other bidirectionally to capture rich spatial context, whereas text tokens attend to all visual patches and preceding text tokens causally. This architecture realizes direct early fusion within a single Transformer tower without negative batch construction or cross-attention decoders.

2. Gated Attention Mechanism: Eliminating Multimodal Attention Sinks and Collapse

When pretraining a unified Transformer with causal text generation conditioned on a visual prefix, a severe shortcut emerges: because language tokens only observe the visual prefix causally, the network easily collapses global visual information onto the very first visual token, turning it into an artificial image-level summary sink. This attention sink phenomenon causes the first token to absorb the majority of attention mass, resulting in severe loss spikes during training and destroying the spatial diversity of patch representations. Consequently, the frozen visual representations suffer dramatic degradation on discriminative and dense visual tasks (e.g., ImageNet linear probing accuracy plummets from 84.3% to 76.2%). GenLIP resolves this failure mode by incorporating an input-dependent, per-head gated attention mechanism:

\[G = \sigma(X W_g + b_g), \quad \widetilde{A} = G \odot A\]

where \(X\) represents the input hidden states, \(W_g\) and \(b_g\) are learnable projection parameters, \(\sigma\) is the sigmoid activation, and \(A = \text{Attn}(X)\) is the original attention output. By modulating attention outputs element-wise prior to the residual addition, the gating factor prevents attention weights from excessively collapsing onto a single sink token and forces text tokens to leverage spatially distributed visual details, stabilizing scaling and eliminating loss spikes.

3. Two-Stage Pretraining Strategy with Native-Aspect-Ratio Adaptation

To balance pretraining compute against fine-grained visual comprehension demands, GenLIP organizes pretraining into two progressive stages. Stage 1 executes fixed-resolution pretraining (S1) at \(224 \times 224\) (196 visual tokens) on Recap-DataComp-1B across 8.0B seen samples (8 epochs), efficiently instilling foundational cross-modal alignment and linguistic grounding. Stage 2 executes diverse-resolution adaptation (S2) on 39M high-quality image-text samples featuring rich, long captions (subsets from Infinity-MM, BLIP3o-Long-Caption, and CapRL) for only 1 epoch. Crucially, images are kept at their native aspect ratios and resized dynamically to maintain total patch counts within \([16, 1024]\). Leveraging MRoPE's native support for variable 2D grids, this stage rapidly adapts the vision encoder to dense document layouts, OCR text, and high-resolution chart structures with minimal training cost.

Loss & Training¶

GenLIP is trained using exclusively the standard autoregressive next-token negative log-likelihood loss on the text token sequence:

\[\mathcal{L}_{\text{LM}} = - \sum_{k=0}^{L} \log P(t_k \mid \{v_j\}_{j=0}^M, \{t_i\}_{i=0}^{k-1}; \theta)\]

No masked image modeling (MIM) or pixel reconstruction objectives are involved. For deep models (such as GenLIP-g with 40 layers), Layer Scale and Drop Path regularization are applied to stabilize gradient propagation under cosine learning rate decay schedules.

Key Experimental Results¶

Main Results¶

Following the Cambrian evaluation protocol, visual representations are evaluated by keeping the vision encoder strictly frozen while fine-tuning the downstream MLLM (LLaVA-NeXT equipped with Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct) on the LLaVA-OneVision 3M dataset. Benchmarks span Document & OCR (ChartQA, OCRBench, DocVQA, TextVQA, AI2D, InfoVQA, SEED-2), General VQA (VQAv2, GQA, SQA, MME-P), and Captioning (NoCaps, COCO, TextCaps).

The following table summarizes frozen evaluation results under LLaVA-NeXT-Qwen2.5-1.5B from Table 2 of the original paper:

Model	Architecture	Pretrain Samples	ChartQA	DocVQA	OCRBench	MME-P	TextCaps (CIDEr)	Overall Avg (ALL AVG)
CLIP (Radford et al.)	ViT-L/14	12.8B	24.8	38.9	23.7	1218	117.9	53.1
AIMv2 (Fini et al.)	ViT-L/14	12.0B	26.3	37.7	25.2	1157	122.4	55.7
OpenVision2 (Liu et al.)	ViT-L/16	12.8B	30.7	43.3	45.6	1230	127.4	58.7
SigLIP (Zhai et al.)	ViT-L/16	40.0B	30.2	47.3	41.0	1203	120.7	56.9
SigLIP2 (Tschannen et al.)	ViT-L/16	40.0B	33.4	45.1	45.7	1165	127.8	58.7
GenLIP (Ours)	ViT-L/16	8.0B	41.2	51.1	51.1	1258	131.4	61.5
SigLIP2 (Tschannen et al.)	ViT-So/16	40.0B	35.2	46.4	47.2	1220	131.5	60.6
GenLIP (Ours)	ViT-So/16	8.0B	40.8	51.9	51.5	1215	129.5	62.6
SigLIP2 (Tschannen et al.)	ViT-g/16	40.0B	35.3	47.6	47.3	1284	134.5	61.5
GenLIP (Ours)	ViT-g/16	8.0B	45.0	57.0	55.6	1256	135.4	65.2

GenLIP-L/16 achieves an overall average of 61.5 using merely 8.0B training samples, outperforming SigLIP2-L/16 (58.7) trained on 40.0B samples by 2.8 points. At the ViT-g/16 scale, GenLIP outperforms SigLIP2-g/16 by 3.7 points overall, exhibiting substantial leaps in fine-grained benchmarks such as DocVQA (57.0 vs 47.6) and ChartQA (45.0 vs 35.3).

Ablation Study¶

The following table presents ablations on model architecture, initialization, attention sinks, and discriminative probing, consolidated from Tables 7, 8, and 9 of the original paper:

Configuration / Variant	Core Description	ChartQA	OCRBench	DocVQA	IN-1K Probe	Overall Avg (ALL AVG)
SAIL (Qwen3-0.6B Init)	Pretrained LLM initialization, standard attention	31.6	30.2	39.3	-	53.6
SAIL-g (Qwen3-0.6B Init)	Pretrained LLM initialization + Gated Attention	30.2	29.5	39.4	-	54.8
SAIL-g (From Scratch)	Random initialization + Gated Attention	32.5	36.0	43.6	-	56.0
GenLIP-So/16 (Default)	ViT from scratch + Gated Attention	34.6	34.2	44.1	84.3 (S1)	56.3
1 Register Token	1 Register token to absorb attention sinks	30.5	32.5	37.2	-	53.3
4 Register Tokens	4 Register tokens to absorb attention sinks	34.5	30.8	41.0	-	55.1
GenLIP-So/16 (w/o GA)	Remove Gated Attention (Table 9)	-	-	-	76.2	-

Key Findings¶

Generative pretraining delivers huge advantages on text-rich visual reasoning: Across all scales, the performance margin of GenLIP over contrastive models is most pronounced on Doc&OCR tasks, verifying that word-by-word language modeling builds much finer-grained semantic-spatial grounding than global cosine similarity.
Language model pretraining priors provide negative transfer for visual feature extraction: Training from scratch outperforms initializing from a pretrained Qwen3-0.6B checkpoint (56.0 vs 53.6), indicating that pure text attention patterns do not transfer cleanly to visual patch extraction.
Gated attention is superior to register tokens: While introducing 1 or 4 register tokens partially mitigates sink issues, gated attention achieves higher overall accuracy (56.3 vs 53.3/55.1) and prevents degradation in linear probing (84.3% vs 76.2% without GA).

Highlights & Insights¶

Radical architectural simplicity: Completely sheds contrastive negative batch constructions and secondary text decoders; a single Vision Transformer seamlessly handles visual perception and language generation.
Tackling multimodal attention sinks: Identifies that conditional causal generation on visual prefixes induces severe attention collapse onto the first visual patch, elegantly fixing it via an input-dependent sigmoid gate.
High transferability: The gated prefix-LM attention formulation and variable aspect ratio adaptation scheme can be readily integrated into unified multimodal transformers and native generative foundation models.

Limitations & Future Work¶

Limitations acknowledged by authors: Empirical validations are primarily carried out within the academic LLaVA-NeXT framework, leaving generalization across frontier proprietary MLLMs to be explored; pretraining data volume remains at the 1B scale without verifying scaling laws at tens of billions; the pipeline relies heavily on high-quality synthetic recaptioning datasets.
Remaining gap: In purely discriminative classification benchmarks (such as ImageNet-1K linear probe), GenLIP slightly trails contrastive methods like SigLIP2 (84.3% vs 88.9%) due to the absence of explicit contrastive clustering loss.
Future directions: Exploring multi-image interleaved pretraining and extending GenLIP to unified video-language generative modeling.

vs CLIP / SigLIP / SigLIP2: Contrastive methods train two independent encoders via discriminative loss, causing downstream objective mismatch; GenLIP uses a single unified transformer trained via next-token prediction, demonstrating much higher data efficiency and lower perplexity in downstream LLMs.
vs AIMv2 / OpenVision2: Prior generative VLP methods cascade a separate text decoder on top of the vision encoder; GenLIP removes the auxiliary decoder entirely, allowing the ViT to generate text directly.
vs SAIL / Chameleon: While SAIL adapts a pretrained LLM into a native multimodal model, GenLIP focuses on pretraining a scalable vision encoder from scratch for modular MLLMs and resolves representation collapse via gated attention.

Rating¶

Novelty: ⭐⭐⭐⭐☆ Eliminates unnecessary dual encoders and decoders, establishing a direct generative paradigm with targeted gated attention.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations covering two LLM sizes, frozen and unfrozen protocols, fine-grained ablations, and discriminative probing.
Writing Quality: ⭐⭐⭐⭐⭐ Clearly written with transparent motivations, cohesive narratives, and insightful diagnostic figures.
Value: ⭐⭐⭐⭐⭐ Sets a strong precedent for moving modular MLLM vision encoders toward pure, data-efficient generative pretraining.