Skip to content

ViQ: Text-Aligned Visual Quantized Representations at Any Resolution

Conference: ECCV 2026
arXiv: 2606.27313
Code: https://github.com/yuxumin/ViQ
Area: Multimodal VLM / VLM Efficiency
Keywords: Quantized Visual Encoder, Discrete Representation, Training Acceleration, Any Resolution, Text Alignment

TL;DR

ViQ proposes a two-stage visual quantization framework—first performing text-aligned pre-training to enrich continuous features with semantics, then progressively compressing and using the FSQ quantizer for low-dimensional discretization—enabling discrete visual encoders to reach or even surpass the performance of continuous encoders for the first time across 9 multimodal benchmarks, while delivering 20%-70% training acceleration and a storage compression ratio of nearly 1/96.

Background & Motivation

Current mainstream Multimodal Large Language Models (MLLMs) almost entirely rely on continuous visual encoders such as CLIP, SigLIP, and InternViT: images are mapped to high-dimensional floating-point feature vectors and then fed into the LLM through a projection layer. Although successful, this paradigm introduces a fundamental representation mismatch—the visual feature stream is a continuous floating-point tensor at its base, while the text token sequence is a sequence of discrete integer indices. To make the two interact, MLLMs need to bridge a projection layer between the high-dimensional continuous visual stream and the discrete text space. This not only increases computational overhead and hardware pressure but also makes the natural goal of "modeling both vision and language under the same discrete framework" difficult to achieve. Some pioneering works such as QLIP and UniTok have attempted to discretize visual signals using Vector Quantization (VQ), but these solutions encounter a thorny dilemma: reconstruction-heavy autoencoders (like VQ-VAE) retain low-level details after quantization but lose high-level semantics, while semantic-heavy visual encoders severely lose fine-grained details when quantized. As a result, the gap between quantized encoders and continuous encoders on image-text understanding benchmarks has remained wider than 20-30 percentage points for a long time.

The Key Challenge of this dilemma lies in the fact that discretization is inherently information compression, and the direction of compression dictates the type of information loss. Reconstruction-oriented compression tends to preserve pixel-level details, but pixel-level features are naturally insensitive to semantic alignment. Semantic-oriented compression preserves conceptual similarity but discards details like resolution and texture as "noise." What MLLMs require is precisely "being able to read images and answer questions (semantics) while also seeing size-8 fonts in documents (details)"—two aspects that naturally conflict in discrete representations, forcing existing methods to choose one or the other.

The Key Insight of ViQ is: since the conflict stems from "compressing high-dimensional continuous features into low-dimensional discrete space at once," why not decompose this compression process into multiple steps with targeted supervision signals for each step? Core Idea: Decouple quantization learning into three stages: "text-aligned pre-training \(\rightarrow\) progressive feature compression \(\rightarrow\) parameter-free low-dimensional quantization". This first injects rich semantics into continuous features through language supervision, then gradually compresses the feature space to control information loss using Proximal Representation, and finally performs final discretization in the low-dimensional space via parameter-free FSQ—allowing discrete codes to carry both high-level semantics (from pre-training alignment) and low-level details (from progressive retention).

Method

Overall Architecture

The visual quantization training of ViQ is split into two major phases: Stage 1 (Text-Aligned Pre-training) and Stage 2 (Visual Quantization Learning). Stage 2 is further divided into Stage 2-1 (Proximal Representation Learning) and Stage 2-2 (FSQ Quantization). The entire process uses SigLIP2-g (1.1B parameters) as the visual encoder backbone. Stage 1 teaches the encoder to handle raw images of any resolution and obtain semantically aligned features through LLM text supervision. Stage 2-1 compresses the 1536-dimensional features progressively to 128 dimensions using a bottleneck layer and \(L_\infty\) norm constraints. Stage 2-2 further compresses the features to 6 dimensions, discretizes them using FSQ, and maintains quantization quality with multi-head attention and 2D RoPE.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image<br/>Any Resolution"] --> B["Stage 1: Text-Aligned Pre-training<br/>SigLIP2-g Encoder + Any-Resolution Adaptation"]
    B --> C["Text Supervision L_text<br/>Self-Distillation L_distill<br/>Maintaining Semantic Stability"]
    C --> D["Stage 2-1: Proximal Representation Learning<br/>Bottleneck Layer C→D=128 Dim<br/>+ L∞ Norm Constraint"]
    D --> E["Stage 2-2: Quantization Learning<br/>Multi-Head Attention + 2D RoPE<br/>FSQ d=6 Dim Discretization"]
    D -->|Reconstruction Branch| F["VAE Latent Space Supervision L_recon<br/>Retaining Low-level Details"]
    E --> G["Output: Discrete Visual Codes<br/>Codebook 64,000 / Per patch×4"]

Key Designs

1. Proximal Representation: Progressively Compressing Feature Space

Directly quantizing the 1536-dimensional high-dimensional features output by SigLIP2-g causes severe precision loss—when similar features closely packed in the continuous space are hard-assigned to the same discrete bin, a large amount of discriminative information is erased. ViQ's solution is not to compress features in a single step but to "turn one step into two": it first reduces the features to 128 dimensions via a bottleneck fully connected layer, and then imposes an \(L_\infty\) norm constraint (\(\|f_1\|_\infty = 1\)) on these 128-dimensional intermediate features, projecting all features onto the surface of a hypercube. This allows feature points to first "gather" into a compact and regularly shaped region in space while retaining their relative distance relationships, paving the way for subsequent hard quantization—much like bundling items neatly before squeezing them into a narrow-necked bottle. The ablation study (Table 3a) clearly shows the value of this progressive approach: directly performing SimVQ quantization from continuous features yields only 60.9 points, while adding a bottleneck layer and progressively increasing constraints (BN \(\rightarrow\) \(l_2\) \(\rightarrow\) \(l_\infty\)) improves the score from 66.6 to 68.7. The core insight is that the \(L_\infty\) norm constraint is better suited for the grid quantization geometry of FSQ than \(L_2\) (Table 3a comparison: 68.7 vs 67.9), because \(L_\infty\) projects features onto the hypercube surface, which naturally corresponds to the discrete grid structure of FSQ.

2. Multi-Head Finite Scalar Quantization (Multi-Head FSQ): The Counter-intuitive Advantages of Parameter-Free Quantization

ViQ chooses FSQ (Finite Scalar Quantization) as the quantizer instead of SIMVQ, LFQ, IBQ, or other methods that require learning an additional codebook. The FSQ approach is straightforward: it maps each of the 6 feature dimensions to 5-8 discrete values (levels [8,8,8,5,5,5]), and directly rounds them to complete quantization—fixing the codebook size to \(8 \times 8 \times 8 \times 5 \times 5 \times 5 = 64,000\). Compared to SimVQ (which requires maintaining a learnable \(2^{15}\) codebook), FSQ is entirely parameter-free throughout, renders more stable training, and consistently wins under the ViQ framework (Table 3c: FSQ 68.7 > SIMVQ 66.6 > original VQ 65.7). Why does parameter-free quantization perform better instead? The key lies in the fact that ViQ's progressive feature constraint has already "pre-shaped" the space into a form close to the FSQ grid—the embedding vectors in the learnable codebook compete with each other during training, which sometimes disrupts this already aligned spatial structure. Experiments in Appendix C support this interpretation: replacing SimVQ's \(L_2\) regularization with \(L_\infty\) drops the score from 66.6 to 66.1, indicating that the regularization must match the quantization geometry.

To compensate for the limited expressive capacity of a single discrete token, ViQ introduces a multi-head extension mechanism: the 6-dimensional features of each image patch are up-sampled to 24 dimensions via linear projection, and then reshaped into 4 independent 6-dimensional sub-tokens. Self-attention is performed within the 4 sub-tokens of the same patch (no interaction between different patches), which are independently quantized, concatenated, and restored to 6 dimensions via linear projection. This "1 patch \(\rightarrow\) 4 codes" design multiplies the code capacity by 4 without increasing the downsampling rate, contributing significantly to detail-dense tasks like document reading and OCR.

3. 2D Rotary Position Embedding and Any-Resolution Quantization

Discrete codes inherently carry no spatial position information—the "top-left patch" and "bottom-right patch" become indistinguishable after quantization. ViQ inserts 2D Rotary Position Embedding (RoPE) before the quantization step, performing rotational modulation on token features \(f_m\) located at coordinate \((h,w)\) in 2D space: $\(\tilde{f}_m = f_m \odot e^{i(h\theta_h + w\theta_w)}\)$ where \(\theta_h\) and \(\theta_w\) are frequency parameters for the height and width dimensions. Compared to learnable position embeddings (65.7) or omitting position information (65.3), 2D RoPE reaches 68.7 (Table 3d) and naturally supports arbitrary resolutions—since RoPE is based on continuous frequency encoding, there is no need to preset a maximum resolution limit during training. This coordinates with Stage 1's use of dynamic position parameter techniques borrowed from NaViT and OryxViT: the encoder can extract visual tokens at any aspect ratio, and the quantization side can maintain the accurate positional relationships of these tokens. It is this any-resolution capability that enables ViQ to obtain continuous-encoder-level performance on high-resolution document tasks such as InfoVQA (+10.4 vs InternViT-6B) and DocVQA (+3.2).

4. VAE Latent Space Reconstruction Supervision: Subtle Balance of Semantics and Details

ViQ's training objective encompasses three losses: text supervision \(\mathcal{L}_\text{text}\) (cross-entropy, aligning high-level semantics), self-distillation \(\mathcal{L}_\text{distill}\) (cosine similarity, maintaining semantic stability), and reconstruction loss \(\mathcal{L}_\text{recon}\). The choice of reconstruction loss is critical: if direct pixel-level MSE + LPIPS reconstruction is performed, although image reconstruction metrics improve, pixel-level rigid constraints compete with high-level semantic features, hindering multimodal understanding (Table 3e: MSE+LPIPS only yields 67.0 vs 68.7 for the full model). ViQ chooses to supervise in the latent space of a pre-trained VAE: the latent representations of images extracted by a frozen Qwen-Image encoder are used as targets, and ViQ's feature predictions are trained to approach them (\(\mathcal{L}_\text{recon} = \text{NLL}(\hat{f}, \text{Encoder}(x))\), equivalent to MSE). The VAE latent space is itself a "semanticized detail representation"—it removes pixel noise while retaining perceptually relevant high-frequency structures. Ablation analysis (Table 3f) shows that VAE latent supervision incurs only a 1.3× time overhead (compared to 1× without reconstruction), while reaching the highest score of 68.7; in contrast, DiT-based reconstruction requires 4× overhead and performs worse (67.6 with frozen DiT, and only 65.8 unfrozen). The core insight is: choosing which space to perform reconstruction in is more important than whether to rebuild or not.

A Concrete Example

Suppose a 768×1024 receipt scan (containing table numbers and handwritten signatures) is input. First, the SigLIP2-g encoder patchifies the image according to its raw aspect ratio (every 16×16), producing approximately 768 visual tokens, each corresponding to a 1536-dimensional continuous feature. Stage 1 uses Qwen2.5-0.5B for text supervision (short QA pairs) to enrich these 768 features with key semantics—the model learns that "this patch is text, and that patch is a signature." Stage 2-1 connects the bottleneck layer to compress each 1536-dimensional feature to 128 dimensions, followed by \(L_\infty\) constraints: all vectors are projected onto the surface of a hypercube of radius 1. Stage 2-2 up-projects the 128 dimensions to 24 dimensions (4×6), encodes position information via 2D RoPE, and performs FSQ quantization in 6-dimensional space—where each dimension is mapped to [0,7] or [0,4]. Ultimately, each original patch yields 4 discrete integer codes (each 16 bits, about 2 bytes). The entire image generates 768×4=3072 codes, requiring only ~6KB of storage (1/96 of the original image). During downstream training, this group of pre-computed discrete codes and a light MLP projection layer are loaded and directly fed into the LLM—completely skipping the forward pass of the visual encoder, which is the source of the 20%-70% training speedup.

Loss & Training

Overall objective function: $\(\mathcal{L}_\text{total} = \lambda_\text{text}\mathcal{L}_\text{text} + \lambda_\text{distill}\mathcal{L}_\text{distill} + \lambda_\text{recon}\mathcal{L}_\text{recon}\)$

Training adopts a three-stage progressive strategy: - Stage 1 (Text-Aligned Pre-training): ~3B VL tokens, with resolution progressively scaled from 384² to 768². Only the visual encoder is fine-tuned using LoRA, with Qwen2.5-0.5B providing text supervision. Concurrently, a frozen SigLIP2-g acts as a self-distillation teacher to ensure that the student model's global features maintain cosine similarity to the original fixed-resolution teacher features, preventing multimodal fine-tuning from damaging pre-trained generalization. - Stage 2-1 (Proximal Representation Learning): 1B tokens, 768 resolution, 128-dimensional bottleneck + \(L_\infty\) regularization. The reconstruction branch uses a prediction head (3-layer MHSA + convolutional upsampling) with VAE latent space loss. - Stage 2-2 (Quantization Learning): 30B tokens, where regularization is replaced by the FSQ quantization module (6-dimensional levels [8,8,8,5,5,5]), incorporating multi-head attention and 2D RoPE. All parameters are trained with lr=5e-5, using a cosine scheduler and annealing to 1e-5.

Key Experimental Results

Main Results

The following results are based on the Qwen2.5-1.5B / 7B backbones, trained on 2000K fixed data.

Dataset Metric ViQ (1.3B) InternViT-2.5-6B QLIP (0.3B) UniTok (0.3B)
MMStar Acc 47.8 / 54.2 48.5 / 55.3 39.9 / - 41.0 / -
MMMU Acc 42.6 / 49.1 42.1 / 48.1 36.9 / - 36.1 / -
SimpleVQA Acc 26.0 / 28.5 23.7 / 28.4 13.7 / - 15.5 / -
InfoVQA Acc 41.6 / 55.3 35.2 / 44.9 14.8 / - 15.9 / -
TextVQA Acc 74.3 / 78.5 75.5 / 79.9 45.1 / - 39.7 / -
DocVQA Acc 84.2 / 88.9 80.1 / 85.7 12.2 / - 11.6 / -
ChartQA Acc 65.2 / 72.8 67.8 / 77.4 14.1 / - 43.8 / -
Avg Acc 57.2 / 63.9 57.0 / 63.8 29.7 / - 33.0 / -

With 1.3B parameters (including the codebook), ViQ outperforms the 6B InternViT-2.5 on the 9-benchmark average score, whereas the previous strongest quantized encoder UniTok scored below 33 points—marking a generational leap. ViQ's gains are most significant in OCR/document-related tasks (InfoVQA +10.4, DocVQA +3.2), while remaining baseline-comparable to continuous encoders in general knowledge tasks.

Ablation Study

Configuration Eight-Task Avg Description
Continuous \(\rightarrow\) BN+\(L_\infty\) \(\rightarrow\) FSQ (Full) 68.7 ViQ Default
Direct Quantization (w/o Progressive) 60.9 Loss of 7.8
L2 instead of \(L_\infty\) 67.9 Regularization must match FSQ grid
SimVQ instead of FSQ 66.5-66.6 Learnable codebook performs worse than parameter-free
w/o Position Embedding 65.3 RoPE contributes 3.4 points
RoPE \(\rightarrow\) Learnable PE 65.7 Learnable PE increases optimization difficulty
Text Loss Only 61.3 Self-distillation + reconstruction cumulative gain of 7.4
+ Self-Distillation (w/o Reconstruction) 66.8 Self-distillation contributes significantly
+ MSE+LPIPS Reconstruction 67.0 Pixel-level constraints erode semantics
+ VAE Latent Space Reconstruction 68.7 Optimal (1.3× cost)

Key Findings

  • Progressive compression (Proximal Representation) yields the highest single-point gain: direct quantization drops the performance by 7.8 points, whereas adding a bottleneck layer and proper regularization recovers nearly all of it.
  • Parameter-free quantization (FSQ) consistently outperforms parameterized quantization (SimVQ/LFQ/original VQ), in contrast to mainstream literature. This is because ViQ's progressive feature pre-compression already aligns the spatial representation with the FSQ grid; a learnable codebook introduces competitive noise instead. The compatibility of the regularization with the quantization geometry is more crucial than the learnability of the codebook.
  • ViQ demonstrates its largest gains on InfoVQA / DocVQA / ChartQA (which correspond to "high detail, requiring text and chart reading"), while matching or performing slightly below continuous encoders on MMStar / MMMU—indicating that the differentiated advantage of quantized visual encoders lies in detail-dense tasks, whereas general reasoning relies more on the LLM itself.
  • Training efficiency: at a sequence length of 4k, Qwen2.5-7B + ViQ delivers a 46% forward acceleration and a 20%+ step acceleration; at 16k, this accelerates to 65% forward and 40%+ step. The speedup comes from "offline pre-computation of discrete codes + skipping the visual encoder during training."
  • VAE latent space supervision, with extremely low cost (1.3×) and the best downstream performance, is the optimal form of reconstruction loss. Pixel-level supervision (MSE+LPIPS) and DiT-based supervision either erode semantics or explicitly drain the encoder's expression capacity into the generative branch.

Highlights & Insights

  • Progressive compression is the most exquisite design of the paper: instead of compressing from 1536 to 6 in a single step, it uses a 1536 \(\rightarrow\) 128 (bottleneck) \(\rightarrow\) 6 (quantization) route with \(L_\infty\) acting as a soft landing. Each compression tier is supported by targeted regularization and supervision. This "hierarchical bottleneck + spatial pre-shaping" approach can be broadly transferred to other fields like model compression, feature distillation, and KV caches.
  • The counter-intuitive finding that parameter-free quantization outperforms parameterized quantization is highly inspiring. Researchers often focus on designing more complex codebook update strategies, but ViQ proves that once the feature space is sufficiently aligned, simple rounding is enough. This suggests that "how features are preprocessed before quantization" is more important than "how complex the quantizer itself is"—an insight with low additional cost.
  • 2D RoPE simultaneously resolves both position ambiguity and any-resolution adaptation, a brilliant design that kills two birds with one stone. In contrast, learnable position embeddings require pre-setting a maximum resolution (contradicting ViQ's any-resolution promise), and using fixed-table position encoding schemes does not scale cleanly—making RoPE's continuous frequency encoding the only choice truly aligned with "any-resolution" expression.
  • The role of VAE latent space supervision is precisely targeted: the latent space itself is already a semanticized detail representation. Supervised L2 regression in this space acts as a soft guide telling the encoder "what details to preserve"—neither demanding pixel-level alignment (protecting semantics) nor completely giving up reconstruction (protecting details), finding a delicate balance between the two competing objectives.

Limitations & Future Work

  • ViQ has been validated only on LLMs \(\leq\) 7B; its compatibility with super-large backbones of 70B+ remains untested. Whether the quantization accuracy of the visual encoder can still match when the backend LLM's own prior knowledge is sufficiently strong warrants further exploration. Moreover, the projection layer between quantized codes and ultra-large LLMs may become a new bottleneck.
  • On OCRBench, ViQ (711/636) still falls short of continuous encoders of comparable size (such as InternViT-2.5-6B, which reaches 757). The authors attribute this to systematic high-frequency loss caused by discretization—certain subtle character-level features cannot be recovered once quantized into coarse bins. Multi-scale quantization or Residual Quantization (RQ) might offer orthogonal paths to mitigate this.
  • A more practical bottleneck is that Stage 2-2 requires 30B tokens and 256 A100 GPUs for training—this resource barrier is unfriendly to small teams and academic labs. Developing efficient progressive strategies or achieving comparable performance with less data (e.g., 5-10B) constitutes a valuable path forward for reproducibility.
  • vs QLIP: QLIP optimizes quantization using contrastive learning objectives, resulting in a higher reconstruction PSNR (23.16) than ViQ (22.73) but performing poorly on understanding tasks (29.7 vs ViQ's 57.2). This proves that "quantization optimized for reconstruction \(\ne\) quantization optimized for multimodal understanding." ViQ's progressive design ensures the two are no longer mutually exclusive.
  • vs UniTok: UniTok relies on joint training of reconstruction and contrastive learning. Its reconstruction metrics (PSNR 25.32, rFID 0.37) are the best among all quantized solutions, but its understanding score is only 33.0—far below ViQ's 57.2-63.9. The competition between strong reconstruction objectives and high-level semantic features culminates in a unilateral sacrifice of understanding in UniTok. ViQ's VAE latent space supervision demonstrates that explicit balancing is easier to manage than jointly optimized multi-objectives.
  • vs InternViT-2.5: InternViT is currently one of the strongest mainstream continuous encoders. ViQ achieves competitive performance in a discrete format with fewer parameters (1.3B vs 6B), validating that the "text-aligned pre-training + quantization" pipeline has the potential to catch up with or even surpass continuous encoders. However, ViQ's codebook data and training pipeline impose higher hardware demands, making InternViT more practical for smaller teams.

Rating

  • Novelty: ⭐⭐⭐⭐ Systematically combines progressive compression, parameter-free quantization, and VAE latent space supervision to deliver a "quantization comparable to continuous" breakthrough; both the design concepts and ablation insights are highly valuable to the community.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated on 9 benchmarks + 2 LLM backbones + 5 sets of independent ablations (proximal / quantization methods / position encoding / loss combinations / reconstruction strategies) + efficiency + storage + reconstruction, providing a highly complete verification chain.
  • Writing Quality: ⭐⭐⭐⭐ The manuscript is clear, training setups are transparent, and charts match the tables well. A minor limitation is that the core conclusion (parameter-free consistently outperforming parameterized quantization) is scattered across the Table 3c footnote and Appendix C, leaving the analysis of this counter-intuitive result in the main text somewhat understated.
  • Value: ⭐⭐⭐⭐⭐ Elevates discrete visual encoders to the performance baseline of continuous encoders for the first time while delivering practical training speedups and storage compression—a significant advancement with implications for both academia (unified representation) and industry (training cost reduction).