VIVAS: Vitalizing Visual Perception in VLM Pre-training via Vision-language Unified Autoregressive Supervision¶
Conference: ECCV 2026
Paper: ECCV 2026 Poster 4767
Area: Multimodal VLM
Keywords: Vision-Language Model, Visual Perception, Unified Token Space, Autoregressive Supervision, Multimodal Pre-training
TL;DR¶
To overcome the severe deficiency in fine-grained visual perception caused by text-dominant pre-training bias, VIVAS incorporates a dense-structural-semantic vision tokenizer and a unified token space into large-scale pre-training, enabling unified autoregressive supervision over both visual details and text with only 3.8% additional inference latency.
Background & Motivation¶
Vision-Language Models (VLMs) have emerged as the foundational paradigm for general multimodal artificial intelligence. Typical architectures employ a vision encoder along with an alignment module to map visual representations into a continuous latent space, leveraging a Large Language Model (LLM) backbone for conditional sequence decoding. Despite their impressive versatility in high-level conversation and general visual question answering, contemporary VLMs exhibit a critical and persistent vulnerability: a severe deficiency in fine-grained visual perception. They frequently stumble on tasks demanding precise spatial localization, high-frequency boundary discernment, and dense grounding, effectively bottlenecking multimodal comprehension to a superficial, coarse level.
This fundamental flaw originates from the pervasive text-dominant optimization bias inherent in standard training regimes. When a model is optimized exclusively under text-generation objectives (such as cross-entropy over target response tokens), visual signals are treated merely as passive conditioning prefixes. Under this objective, the network is implicitly incentivized to shortcut high-frequency geometric contours and localized visual details that appear redundant for generating coarse textual semantics. While recent works have attempted to inject visual supervision during post-training (such as SFT or reinforcement learning phases), their efficacy is inherently marginal. By the post-training phase, the underlying representation space has already crystallized under trillions of text-centric pre-training tokens, rendering late visual supervision merely an auxiliary alignment constraint that often interferes with conversational instruction following.
Therefore, the decisive window for instilling fine-grained perception resides fundamentally in pre-training. Realizing effective pre-training visual supervision, however, encounters two formidable hurdles: architectural instability and supervision signal quality. Decoupled architectural designs—such as appending external Diffusion Transformers (DiT) or regression heads to output hidden states—suffer from catastrophic training instability and divergence during early pre-training when visual capacities are not yet established. Furthermore, supervision signals derived purely from pixel-level reconstruction (like VQ-VAE) introduce a severe cross-modal semantic gap, whereas standard contrastive embeddings (like SigLIP) lack spatial granularity and boundary sharpness. Core idea: integrate complementary structural features from DINOv3 and semantic representations from SigLIP2 into a dense-structural-semantic visual codebook within a unified token space, performing end-to-end vision-language unified autoregressive supervision over both visual details and linguistic tokens during pre-training.
Method¶
Overall Architecture¶
VIVAS is grounded in a unified token space paradigm, entirely circumventing the optimization instabilities that plague architectural separation schemes. The overall framework consists of two core components: an offline-trained Dense-Structural-Semantic Vision Tokenizer and an end-to-end Unified Vision-Language Autoregressive Pre-training pipeline.
During the training workflow, each input image is processed through two pathways. On the continuous input side, it is encoded by a standard visual encoder (SigLIP2-so400m-patch16-naflex) and projected into the LLM latent space to serve as visual prefix conditioning. On the discrete supervisory side, the image is passed through dual frozen expert backbones (DINOv3 and SigLIP2) and cross-attention fusion to generate discrete visual supervisory tokens \(V = \{v_1, \dots, v_M\}\) via Index Backpropagation Quantization (IBQ). The textual sequence \(T = \{t_1, \dots, t_N\}\) and visual tokens are interleaved into a unified sequence \(S\). During end-to-end pre-training, the LLM backbone (Youtu-LLM) is optimized autoregressively to predict both subsequent textual and visual tokens via standard cross-entropy loss, compelling the model to actively internalize dense visual perception without degrading language fluency.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Image & Text Instructions"] --> B["Dual Expert Feature Extraction<br/>DINOv3 Geometry + SigLIP2 Semantics"]
B --> C["Dense-Structural-Semantic Vision Tokenizer<br/>Cross-Attention Fusion + IBQ Discretization"]
C --> D["Unified Vision-Language Vocabulary<br/>Expand Text Vocabulary with 150k Visual Codebook"]
D --> E["Unified Autoregressive Pre-training<br/>Joint Cross-Entropy over Visual & Text Tokens"]
E --> F["Progressive Multi-Stage Curriculum<br/>Language Pre-train → Multimodal Pre-train → Text-Only Post-train"]
F --> G["Fine-Grained Perception & Multimodal Reasoning Output"]
Key Designs¶
1. Dense-Structural-Semantic Vision Tokenizer: Fusing geometric boundaries with language-grounded semantics Existing discrete visual representations face an intrinsic dilemma: text-decoupled visual vocabularies (e.g., standard VQ-VAE or self-supervised DINOv3) capture fine-grained geometry and high-frequency textures but lack multimodal alignment, causing severe modality disconnect when used as targets; conversely, contrastive vision-language representations (e.g., SigLIP2) provide high-level category semantics but suffer from coarse spatial resolution and boundary bleeding. To resolve this dilemma, VIVAS decouples and recombines their strengths: frozen DINOv3 features \(H_{\text{geo}} \in \mathbb{R}^{N \times d_d}\) serve as structural queries (\(Q\)) to enforce spatial templates, while frozen SigLIP2 features \(H_{\text{sem}} \in \mathbb{R}^{N \times d_s}\) provide semantic keys (\(K\)) and values (\(V\)) to supply aligned linguistic attributes. Projected into a shared latent space, the synergistic representation is synthesized via cross-attention: $\(Z_{\text{syn}} = \text{Softmax}\left(\frac{Q K^\top}{\sqrt{d_k}}\right) V\)$ \(Z_{\text{syn}}\) is then concatenated with the structural features \(H_{\text{geo}}\) along the channel dimension, projected through an MLP, and quantized via Index Backpropagation Quantization (IBQ) into a learnable codebook \(\mathcal{C}\) of vocabulary size \(K = 150{,}000\) and embedding dimension \(D = 768\), producing the discrete supervisory visual vocabulary \(\mathcal{V}_{\text{img}}\).
2. Texture-Bias-Free Perceptual-Adversarial Quantization: Suppressing high-frequency noise for 97.7% codebook utilization Standard vector quantization commonly relies on pixel-level \(\ell_1\) reconstruction objectives. However, \(\ell_1\) minimization inevitably induces a severe "texture bias," compelling the codebook to allocate its representational capacity toward memorizing high-frequency pixel noise and subtle illumination fluctuations rather than learning abstract geometric semantics. VIVAS deliberately excludes the \(\ell_1\) loss and optimizes the tokenizer solely through perceptual similarity \(\mathcal{L}_{\text{lpips}}\) and adversarial discriminator loss \(\mathcal{L}_{\text{gan}}\), coupled with vector quantization loss \(\mathcal{L}_{\text{vq}}\) and entropy regularization \(\mathcal{L}_{\text{ent}}\): $\(\mathcal{L}_{\text{tok}} = \lambda_p \mathcal{L}_{\text{lpips}} + \lambda_g \mathcal{L}_{\text{gan}} + \mathcal{L}_{\text{vq}} + \lambda_e \mathcal{L}_{\text{ent}}\)$ with loss weights set to \(\lambda_p = 1, \lambda_g = 1, \lambda_e = 0.1\). Bypassing pixel memorization compels the discrete codes to encode high-level structural semantics, yielding an exceptional codebook utilization rate of 97.7%.
3. Unified Token Space with Joint Autoregressive Supervision: Eradicating architectural training divergence In pre-training with visual supervision, architectural separation paradigms that attach an external Diffusion Transformer (DiT) to output hidden states trigger severe training instability and non-convergence, as immature visual representations destabilize the joint regression loss. VIVAS instead expands the textual vocabulary with the 150k discrete visual codebook into a single unified vision-language vocabulary. Both visual tokens \(V\) and textual tokens \(T\) are supervised under a unified sequence objective: $\(\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{text}} + \lambda \mathcal{L}_{\text{vision}}\)$ where \(\lambda\) is set empirically to 0.5. By casting visual supervision into standard next-token prediction cross-entropy, the training dynamics remain exceptionally smooth and identical to standard LLM pre-training. Crucially, because visual tokens are only predicted during training and the model outputs standard text at inference, the expanded vocabulary incurs only a nominal 3.8% additional inference latency.
4. Progressive Four-Stage Training Curriculum: Targeting the optimal window for visual supervision Ablation reveals that visual supervision acts as a double-edged sword across different lifecycle stages: while indispensable during foundational pre-training, activating visual supervision during post-training degrades instruction-following capabilities. VIVAS establishes a structured four-stage curriculum across 12.4T tokens: - Stage I (Language Backbone Pre-training, ~10T tokens): Trains the Youtu-LLM backbone on commonsense, math, STEM, and code corpora using \(\mathcal{L}_{\text{text}}\) to build robust reasoning abilities. - Stage II & III (Multimodal Pre-training & Task Adaptation, 2.4T tokens): Simultaneously trains the vision encoder, projector, and LLM end-to-end under the unified objective \(\mathcal{L}_{\text{text}} + \lambda \mathcal{L}_{\text{vision}}\), internalizing dense structural perception on image-caption pairs, vision-centric data, and diverse multimodal instructions. - Stage IV (Multimodal Post-training SFT & RL, ~4.2B tokens): Disables visual supervision (\(\lambda = 0\)) and trains purely on \(\mathcal{L}_{\text{text}}\) across curated instruction datasets and three RL domains (perception RL, reasoning RL, general VQA RL) to achieve optimal human alignment.
Loss & Training¶
The overall training objective during multimodal pre-training (2.4T tokens) is defined over the unified sequence \(S\): $\(\mathcal{L}_{\text{total}} = -\sum_{i \in \mathcal{I}_t} \log P(t_i \mid s_{<i}; \theta) - \lambda \sum_{j \in \mathcal{I}_v} \log P(v_j \mid s_{<j}; \theta)\)$ where \(\mathcal{I}_t\) and \(\mathcal{I}_v\) denote the positional indices of text and visual tokens, respectively. End-to-end optimization of the vision encoder, projector, and LLM backbone ensures that continuous visual embeddings and discrete supervisory predictions mutually reinforce each other across all Transformer layers.
Key Experimental Results¶
Main Results¶
VIVAS was thoroughly evaluated across 7 categories and 39 multimodal benchmarks against leading open-source models at the 4B scale, including Qwen3-VL-4B-Instruct and InternVL-3.5-4B. The representative results are summarized below:
| Task Category / Benchmark | Metric | Qwen3-VL-4B-Instruct | InternVL-3.5-4B | VIVAS w/o Vision Sup. (4B) | VIVAS (Ours, 4B) | Gain over w/o VS |
|---|---|---|---|---|---|---|
| MMBench-EN | Score | 83.9 | 80.3 | 83.2 | 83.9 | +0.7 |
| MME | Total Score | 2309* | 2272 | 2360 | 2384 | +24 |
| VisuLogic | Acc (%) | 19.0 | - | 23.1 | 25.7 | +2.6 |
| VLMsAreBlind | Acc (%) | 71.9 | - | 85.4 | 88.9 | +3.5 |
| HallusionBench | Score | 57.6 | 44.8 | 57.3 | 59.1 | +1.8 |
| CRPE_relation | Acc (%) | 71.0* | 75.0 | 69.5 | 72.2 | +2.7 |
| RefCOCO val | [email protected] | 90.7 | 92.5 | 88.1 | 93.6 | +5.5 |
| RefCOCO+ val | [email protected] | 82.9 | 87.6 | 87.6 | 90.1 | +2.5 |
| RefCOCOg val | [email protected] | 87.3 | 89.6 | 89.5 | 92.2 | +2.7 |
| ScreenSpot Pro | Acc (%) | 59.5 | - | 57.1 | 59.6 | +2.5 |
| OSWorld | Success (%) | 26.2 | - | 36.4 | 38.8 | +2.4 |
| CharXiv DQ | Acc (%) | 76.2 | 71.1 | 76.5 | 79.4 | +2.9 |
( indicates scores reproduced via the authors' internal evaluation platform. All models evaluated at the 4B scale)*
Ablation Study¶
Ablations on visual supervision strategies, tokenization signals, and training stages were conducted under a controlled 0.5T token pre-training regime with 5-shot evaluation:
1. Impact of Visual Supervision Signals (0.5T pre-training, 5-shot evaluation):
| Visual Supervision Strategy | VisuLogic | MMBench-EN | MME | CRPE_rel | HallusionBench | RefCOCO val |
|---|---|---|---|---|---|---|
| w/o Visual Supervision | 22.2 | 53.0 | 48.2 | 51.0 | 55.3 | 62.1 |
| Text-decoupled vocabularies | 21.6 | 51.2 | 46.9 | 49.3 | 53.9 | 60.7 |
| Vision-language semantic vocabularies | 23.6 | 53.6 | 47.7 | 51.4 | 54.8 | 63.0 |
| VIVAS (Dense-Structural-Semantic, Ours) | 24.4 | 54.9 | 49.8 | 56.3 | 56.5 | 65.2 |
2. Impact of Visual Supervision Across Training Stages (Table 2):
| Pre-train VS | Post-train VS | VisuLogic | MMBench-EN | CRPE_relation | HallusionBench |
|---|---|---|---|---|---|
| \(\times\) | \(\times\) | 23.1 | 83.2 | 69.5 | 57.3 |
| \(\checkmark\) | \(\times\) | 25.7 | 83.9 | 72.2 | 59.1 |
| \(\times\) | \(\checkmark\) | 23.3 | 82.9 | 69.9 | 57.7 |
| \(\checkmark\) | \(\checkmark\) | 23.6 | 83.3 | 70.1 | 58.1 |
3. Impact of \(\ell_1\) Loss in Codebook Optimization (Table 7):
| \(\ell_1\) Loss Inclusion | VisuLogic | MMBench-EN | CRPE_relation | HallusionBench | Remarks |
|---|---|---|---|---|---|
| Without \(\ell_1\) (Ours) | 24.4 | 54.9 | 59.0 | 56.5 | Avoids texture bias, focuses on structure and semantics |
| With \(\ell_1\) | 23.7 | 54.2 | 57.9 | 56.1 | Memorizes high-frequency noise, impairs visual reasoning |
Key Findings¶
- Pre-training is the decisive window: Applying visual supervision during pre-training yields remarkable gains (VisuLogic jumps from 23.1 to 25.7). In contrast, applying it solely during post-training offers negligible improvements (23.3), and applying it across both pre-training and post-training actually harms performance (23.6) due to interference with instruction tuning.
- Purely decoupled visual codes degrade VLM capabilities: Supervising with text-decoupled VQ-VAE vocabularies actively hurts multimodal comprehension across every single benchmark (MMBench-EN drops from 53.0 to 51.2; RefCOCO drops from 62.1 to 60.7), underscoring the severe hazard of unaligned visual noise.
- Input feature concatenation is insufficient: Feeding combined DINOv3 + SigLIP2 representations into the input encoder without visual-token supervision achieves only 60.9 on RefCOCO val (inferior even to the baseline's 62.1). Under text-dominant loss, models discard extra visual inputs unless explicitly forced to predict them.
- Predictable power-law scaling: Pre-training across 2.4T multimodal tokens demonstrates smooth, uninterrupted neural scaling dynamics (\(L(C) \propto C^{-\alpha}\), with \(\alpha \approx 0.102\) in Stage 2 and \(\alpha \approx 0.079\) in Stage 3), with 27-benchmark average accuracy steadily increasing from 0.43 to over 0.74.
Highlights & Insights¶
- Unified Token Space vs. Architectural Separation: Solves the notorious training instability of diffusion-based visual supervision heads by formulating visual supervision as straightforward discrete next-token cross-entropy.
- Asymmetric Cross-Attention Feature Synthesis: Treating DINOv3 geometric features as structural queries and SigLIP2 as semantic key/values represents a clean, intuitive mechanism to bridge spatial granularity and language alignment.
- Deliberate Omission of \(\ell_1\) Loss: Revealing that traditional pixel-level reconstruction losses induce counterproductive texture bias in VLM tokenization is a crucial empirical insight for discrete vision modeling.
- Negligible Inference Cost: Since the visual tokenizer is discarded at inference time and the model decodes standard text tokens, the framework incurs only a 3.8% increase in inference latency.
Limitations & Future Work¶
- Computational Overhead of Dual-Expert Preprocessing: Offline extraction of dual DINOv3 and SigLIP2 representations across billions of image-text pairs requires substantial storage and multi-pass forward inference.
- Fixed Patch Granularity: The current visual tokenization operates on fixed patch grids, which may constrain adaptability to arbitrary aspect ratios or extremely dense document images without dynamic tiling.
- Future Directions: Exploring online self-distillation to eliminate the dependency on frozen teacher backbones, and extending unified autoregressive supervision to high-frame-rate video understanding and embodied manipulation.
Related Work & Insights¶
- vs. Text-Centric VLMs (LLaVA-OV, Qwen2.5-VL, InternVL): Prior models rely solely on text cross-entropy loss, allowing fine-grained spatial information to vanish in deep layers; VIVAS introduces unified discrete supervision, achieving substantial gains on fine-grained grounding (+5.5% on RefCOCO val).
- vs. Post-Training Visual Supervision (PixelLM, Volo, VILA-U): Post-training methods add auxiliary vision heads late in the pipeline when representations are already fixed; VIVAS establishes that foundational pre-training is the critical stage, completely disabling visual supervision during post-training.
- vs. Continuous Diffusion Heads (Transfusion, Show-o, Chameleon): Continuous diffusion heads require complex flow-matching or denoising regression that frequently diverge during early multimodal training; VIVAS's discrete unified token space ensures perfectly stable optimization dynamics.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Pioneering demonstration of stable, large-scale (12.4T token) unified autoregressive visual supervision that successfully overcomes text-dominant representation collapse.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorously tested across 39 benchmarks and 7 task domains, supported by controlled 0.5T ablations, out-of-distribution tests, and neural scaling law analysis.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative, elegant mathematical formulations, and compelling conceptual figures detailing the motivation and empirical observations.
- Value: ⭐⭐⭐⭐⭐ Delivers a highly practical, generalizable, and low-latency pre-training blueprint for next-generation foundation vision-language models.