---

title: >- [Paper Note] NaVLM-PVC: Progressive Visual Compression for Efficient Native-Resolution Encoding in MLLMs description: >- [ECCV2026][Multimodal VLM][Native-Resolution Encoding] NaVLM-PVC introduces Progressive Visual Compression (PVC) with Refined Patch Embedding and Windowed Token Compression into ViT, achieving 1.9x TTFT reduction over Qwen2-VL while maintaining competitive multimodal performance. tags: - ECCV2026 - Multimodal VLM - Native-Resolution Encoding - Visual Token Compression - Vision Transformer date: 2026-09-19 content_hash: 4e5dd3e807b38f76

NaVLM-PVC: Progressive Visual Compression for Efficient Native-Resolution Encoding in MLLMs

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/AI9Stars/NaVLM-PVC
Area: Multimodal VLM
Keywords: Native-Resolution Encoding, Progressive Visual Compression, Vision Transformer, Windowed Token Compression, Refined Patch Embedding

TL;DR

Addressing the core tension between semantic fragmentation in slice-based encoding and excessive compute in global native-resolution encoding, NaVLM-PVC proposes Progressive Visual Compression (PVC)—combining Refined Patch Embedding (RPE) and hierarchical Windowed Token Compression (WTC)—to deliver 1.9× to 2.4× TTFT latency reductions while preserving fine-grained multimodal performance.

Background & Motivation

Multimodal Large Language Models (MLLMs) have achieved remarkable breakthroughs in fine-grained document comprehension, high-resolution remote sensing, and embodied AI agents. To process arbitrary aspect ratios and high-resolution visual inputs, early mainstream architectures (such as LLaVA-UHD and MiniCPM-V) predominantly relied on slice-based visual encoding (SBE), which splits high-resolution images into individual sub-crops and encodes them independently before concatenating features. However, physical image partitioning fragments global visual context across slice boundaries. This introduces severe semantic discontinuities and impairs long-range spatial relational reasoning across objects that span multiple slices. In response, cutting-edge foundation models such as Qwen2-VL, Oryx, and MiMo-VL increasingly adopt global native-resolution encoding (GNE), which processes the whole image in a single forward pass and demonstrates superior contextual coherence and spatial orientation reasoning.

Nevertheless, global native-resolution encoding incurs prohibitive computational costs. The computational complexity of standard self-attention in Vision Transformers scales quadratically (\(O(N^2)\)) with visual token sequence length. High-resolution inputs exponentially blow up the number of tokens, resulting in massive computational overhead inside the vision encoder and injecting thousands of visual tokens into the LLM's prefill stage. This inflates time-to-first-token (TTFT) latency and GPU memory consumption. Existing workarounds either rely on late-stage compression in the projector (such as Q-Formers, Resamplers, or Pixel-Unshuffle), leaving the vision encoder backbone fully overloaded at every layer, or turn to linear attention and RNN-style architectures from scratch, sacrificing the rich representations of widely pretrained ViT checkpoints.

Controlled probe experiments reveal that while slice-based encoding suffers from a systematic cross-shaped directional bias and spatial anisotropy, global native encoding maintains holistic attention distributions. The critical challenge is how to retain native global context while drastically trimming token sequence redundancy within the encoder. Core idea: equip standard pretrained Vision Transformers with Progressive Visual Compression (PVC)—which modulates Refined Patch Embedding (RPE) for fine-grained tokenization and hierarchically inserts content-adaptive Windowed Token Compression (WTC) across intermediate layers—to progressively condense visual tokens while preserving full native-resolution spatial awareness.

Method

Overall Architecture

NaVLM-PVC follows the canonical three-component paradigm of MLLMs: a vision encoder, a feature projector, and a large language model. Its central innovation lies in reconfiguring a pretrained Vision Transformer into an efficient native-resolution encoder, termed ViT-PVC. Input images are fed at their native resolution and original aspect ratio without any slicing. The architecture first applies RPE to tokenize the image into fine-grained visual representations. Next, the ViT backbone is partitioned into \(J\) stages separated by WTC layers, which progressively downsample local \(2 \times 2\) token windows using content-adaptive weights. The condensed visual tokens from the final layer are mapped into the LLM embedding space via a lightweight two-layer MLP projector and passed to Qwen2-7B bounded by <image> and </image> tags.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Native-Resolution Input Image<br/>Arbitrary Aspect Ratio"] --> B["Refined Patch Embedding (RPE)<br/>Equivalent Weight Resizing P→P_hat"]
    B --> C["ViT Stage 1 (Transformer Blocks)<br/>N Dense Visual Tokens"]
    C --> D["Windowed Token Compression (WTC 1)<br/>Content-Adaptive 2x2 Aggregation"]
    D --> E["ViT Stage 2 (Transformer Blocks)<br/>Condensed to N/4 Tokens"]
    E --> F["Windowed Token Compression (WTC 2)<br/>Content-Adaptive 2x2 Aggregation"]
    F --> G["ViT Stage 3 (Transformer Blocks)<br/>Condensed to N/16 Tokens"]
    G --> H["MLP Projector<br/>Feature Dimension Alignment"]
    H --> I["Large Language Model (LLM)<br/>Efficient Prefill & Autoregressive Decoding"]

Key Designs

1. Refined Patch Embedding (RPE): Preserving Pretrained Equivalence for Finer Granularity

Enlarging input resolution blindly inflates sequence length, while using coarse patches discards crucial high-frequency boundaries and character strokes. RPE scales down the patch size \(P\) of a pretrained ViT (e.g., from \(14 \times 14\) or \(16 \times 16\) to \(10 \times 10\) or \(8 \times 8\)) to enrich visual granularity, while preserving representation equivalence with original embeddings so that extensive pretraining is retained.

Let the original patch embedding kernel be \(W \in \mathbb{R}^{D \times (C \cdot P \cdot P)}\) and the refined target kernel be \(\hat{W} \in \mathbb{R}^{D \times (C \cdot \hat{P} \cdot \hat{P})}\) where \(\hat{P} < P\). For an image patch vector \(t\) and its finer counterpart \(\hat{t}\), there exists a linear transformation matrix \(B\) such that \(\hat{t} = t B\). To guarantee representational equivalence between original patch embeddings \(t W^\top\) and refined embeddings \(\hat{t} \hat{W}^\top\), the system solves \(B \hat{W}^\top \approx W^\top\). Using the least-squares pseudo-inverse, the refined weights are obtained in closed form: $$ \hat{W} = (B^\top)^+ W $$ This weight transformation initializes the patch embedder with finer tokenization granularity, feeding rich fine-grained visual primitives into the initial transformer layers.

2. Windowed Token Compression (WTC): Content-Adaptive Local Aggregation

Although RPE produces dense, detailed visual tokens, propagating them through all transformer layers creates severe computational bottlenecks. WTC introduces lightweight compression layers between ViT stages to reduce sequence length hierarchically over non-overlapping \(2 \times 2\) windows.

While plain average pooling provides stable training convergence, it treats background tokens and key semantic edges with uniform weight, causing significant drops in OCR and chart recognition. Parameterized alternatives like pixel-unshuffle suffer from severe optimization instability when inserted into early ViT layers. WTC resolves this with Content-Adaptive Pooling (CA-Pooling). For the four tokens \(\{x_i\}_{i=1}^4\) in a \(2 \times 2\) window, it computes the local mean \(x_{\text{avg}} = \frac{1}{4} \sum_{i=1}^4 x_i\), forms augmented tokens \(\hat{x}_i = [x_i; x_{\text{avg}}] \in \mathbb{R}^{1 \times 2D}\), and predicts attention logits via a lightweight MLP \(f_\theta\): $$ x_{\text{compressed}} = \sum_{i=1}^{4} \frac{\exp(a_i)}{\sum_{j=1}^{4}\exp(a_j)} x_i, \quad a_i = f_\theta(\hat{x}_i) $$ Critically, \(f_\theta\) is zero-initialized at step zero. This causes the initial attention weights to fall back to uniform \(0.25\) (exact average pooling), guaranteeing smooth optimization in early training. As training progresses, the network learns content-adaptive weights that preserve salient features like text strokes and object boundaries.

3. Joint Modulation and Pre-Alignment Strategy: Optimizing Insertion Depth and Manifold Stability

The trade-off between speed and representation depends on patch size \(P\), the number of WTC layers \(J\), and their insertion layer indices \(j\). Systematic profiling shows that inserting WTC into very shallow layers (e.g., layers 1–3) disrupts low-level feature extraction, whereas placing them too deep fails to save self-attention FLOPs. A configuration of 3 WTC modules inserted at layers 4, 18, and 27 delivers the optimal Pareto frontier.

Because modifying patch kernels and inserting intermediate downsamplers shifts the pretrained ViT feature distribution, the authors introduce a dedicated Pre-alignment Stage using 4M samples. With the LLM frozen, this stage stabilizes the altered visual representation space before full joint multi-modal pretraining and supervised fine-tuning.

Key Experimental Results

Main Results

NaVLM-PVC (using Qwen2-7B as the LLM backbone) was evaluated across 15 standard multimodal benchmarks, spanning general multimodal understanding, scientific knowledge, OCR & chart reading, and complex spatial reasoning.

Table 1: Performance on General and Knowledge Benchmarks (from original Table 1)

Model	LLM Backbone	Pretraining Data (#Data)	Compression Ratio	MME	MMBench	SEED-Image	MMStar	MMMU	AI2D	MathVista
LLaVA-OneVision	Qwen2-7B	9.4M	4	1998.0	80.8	75.4	61.7	48.8	81.4	63.2
MiniCPM-V-2.6	Qwen2-7B	460.0M	16	2348.4	78.0	74.1	57.5	49.8	82.1	60.6
Qwen2-VL	Qwen2-7B	~700.0M	4	2326.8	80.7	75.3*	60.7	54.1	83.0	58.2
InternVL2	InternLM2.5-7B	~100.0M	4	2210.3	79.5	76.2*	60.7	54.1	83.0	58.3
NaVLM-PVC (Ours)	Qwen2-7B	20.1M	64	2183.6	81.3	77.2	60.5	50.2	82.9	64.2

Table 2: Performance on Visual Reasoning and OCR & Chart Benchmarks (from original Table 2)

Model	Compression Ratio	HallusionBench	RealWorldQA	CV-Bench-2D	OCRBench	DocVQA	ChartQA
LLaVA-OneVision	4	31.6	66.3	69.7*	62.2	87.5	80.0
MiniCPM-V-2.6	16	48.1	65.5*	69.7*	85.2	90.8	79.4*
Qwen2-VL	4	50.6	70.1	76.0*	86.6	94.5	83.0
NaVLM-PVC (Ours)	64	52.2	70.3	73.5	82.7	92.8	82.8

Ablation Study

The ablation experiments benchmark individual components using SigLIP2-SO-400M and Qwen2-7B under a fixed 1024×1024 resolution.

Table 3: Ablation on RPE and WTC Modules (from original Table 3)

Method / Config	Patch Size	# WTC	Layer Indices	Visual Tokens	TTFT Latency (ms)	Overall Avg	OCR & Chart	Visual Reasoning
Baseline	16	1	27	1024	233	62.1	61.8	55.7
+ WTC (Avg-pooling)	16	2	4, 18	256	82	59.2	53.4	53.2
+ WTC (CA-pooling)	16	2	4, 18	256	83	60.7	55.6	55.2
+ WTC (Pixel-unshuffle)	16	2	4, 18	256	83	57.5	53.5	52.9
+ RPE + WTC (Full PVC)	8	3	4, 18, 27	256	160	63.0	64.6	55.6

Table 4: Controlled Comparison of MoonViT vs. ViT-PVC with Pre-alignment (from original Table 4)

Strategy	Vision Encoder	PVC Module	Max Tokens	Overall Avg	MME	POPE	HallusionBench
GNE	MoonViT	✗	776	64.11	1522.97	85.38	39.94
GNE with PVC (Ours)	ViT-PVC	✓	380	64.26	1532.33	88.25	39.11

Key Findings

1. Exceptional Data and Token Efficiency: Operating at an aggressive compression ratio of 64 and trained on only 20.1M image-text pairs (over an order of magnitude less than commercial baselines), NaVLM-PVC achieves 81.3 on MMBench and 64.2 on MathVista, outperforming Qwen2-VL (80.7 and 58.2) which was trained on ~700M samples.
2. Complementarity of RPE and CA-Pooling: Introducing plain average-pooling WTC cuts TTFT latency from 233 ms to 82 ms, but causes OCR & Chart accuracy to plunge from 61.8 to 53.4. Content-adaptive pooling recovers 2.2 points, and combining CA-pooling with RPE (\(P=8\)) drives OCR & Chart performance up to 64.6 (+2.8 over baseline) while sustaining a 1.5× inference speedup.
3. Mitigating Spatial Directional Bias: On the ShapeGrid probe benchmark, slice-based models suffer severe accuracy degradation along the central vertical and horizontal slicing axes (forming a cross-shaped error pattern). In contrast, global native encoding maintains uniform directional attention, lifting spatial relative direction accuracy by 11.0% (73.6% vs. 62.6%).

Highlights & Insights

• Zero-Initialized Residual Gating: Initializing the content-adaptive MLP weights to zero preserves exact average pooling at step zero. This prevents gradient instability when modifying intermediate representations of frozen or fine-tuned ViTs.
• Pseudo-Inverse Weight Transformation: Adapting patch kernels via closed-form pseudo-inverse enables flexible patch-size scaling without discarding pretrained weights or requiring expensive architectural re-training from scratch.
• Early-Stage In-Network Compression: Compressing tokens progressively inside the ViT tackles the quadratic complexity of self-attention directly at intermediate layers, cutting both ViT compute and LLM prefill latency simultaneously.

Limitations & Future Work

• Fixed Window Partitioning: Current WTC modules enforce rigid \(2 \times 2\) grid windows; exploring deformable or attention-directed non-local token clustering could further improve representation for irregular object boundaries.
• Layer Allocation Heuristics: The selection of insertion layer indices (e.g., layers 4, 18, and 27) was determined empirically; automated neural architecture search could optimize placement across diverse model sizes and depths.
• Extension to Video Modalities: While demonstrated on high-resolution images, extending Progressive Visual Compression to 3D spatio-temporal video token streams represents an impactful avenue for future exploration.

Related Work & Insights

• vs Slice-based Encoding (LLaVA-UHD, MiniCPM-V): SBE incurs artificial spatial fragmentation and cross-shaped directional bias. NaVLM-PVC retains a unified global representation and achieves lower TTFT than slice-based architectures via hierarchical token merging.
• vs Native-Resolution Encoders (Qwen2-VL, MoonViT): Uncompressed native encoders incur heavy compute and slow prefill. ViT-PVC cuts token counts by half or more (e.g., 380 vs. 776 tokens) while matching or exceeding task accuracy.
• vs Late Projector Compressors (Pixel-Unshuffle, Resampler, Honeybee): Projector-level compression only relieves the LLM, leaving the ViT backbone overwhelmed by quadratic token scaling. PVC's in-network compression accelerates both the vision backbone and language model.

Rating

• Novelty: ⭐⭐⭐⭐ [Identifies cross-shaped bias in slice-based encoding and introduces seamless in-network progressive compression for ViTs]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Systematic spatial probes on ShapeGrid, extensive 15-benchmark evaluation, and granular latency/convergence ablations]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, self-consistent terminology, well-structured mathematical and empirical formulations]
• Value: ⭐⭐⭐⭐⭐ [Offers an actionable, high-efficiency paradigm for industrial and academic native-resolution MLLM deployment]

Related Papers

• [ECCV 2026] Magic-MM-Embedding: Towards Visual-Token-Efficient Universal Multimodal Embedding with MLLMs
• [ECCV 2026] Staying VIGILant: Mitigating Visual Laziness via Counterfactual Visual Alignment in MLLMs
• [ECCV 2026] Global-Local Dual Perception for MLLMs in High-Resolution Text-Rich Image Translation
• [ICLR 2026] ERGO: Efficient High-Resolution Visual Understanding for Vision-Language Models
• [ECCV 2026] Evaluating and Enhancing Negation Comprehension in Remote Sensing MLLMs