UltraViT: Latency-Optimized On-device Vision Encoder for Large Vision-Language Models

Conference: ECCV 2026
Paper: ECCV Official Page
Area: Multimodal VLM
Keywords: Vision Encoder, On-device Efficiency, Large Vision-Language Models, Heterogeneous Spatial Mixers, Generative Pre-training

TL;DR

Addressing the heavy computational burden and latency bottleneck of vision encoders in edge-deployed LVLMs, UltraViT constructs a macro-block heterogeneous pyramidal vision encoder informed by physical on-device NPU latency measurements, combined with a two-stage generative pre-training framework that delivers 1.7× to 19.1× on-device speedups while establishing state-of-the-art multimodal accuracy.

Background & Motivation

Large Vision-Language Models (LVLMs) have driven remarkable advancements across visual reasoning and multimodal understanding, yet their colossal parameter counts and substantial computational footprints fundamentally impede deployment on resource-constrained mobile and edge devices. To mitigate high inference costs, existing efforts targeting efficient LVLMs predominantly concentrate on two trajectories: aggressively shrinking the visual token sequence at the language model input (via training-aware token compression or post-hoc pruning and sparsification), and coupling smaller language model backbones with lightweight cross-modal projection layers. However, the architectural design of the vision encoder itself has been largely overlooked. Most existing frameworks continue to treat computationally heavy networks like CLIP-ViT-L or SigLIP-400M as fixed feature extractors, creating a severe operational bottleneck, particularly when handling high-resolution images and video streams.

This reliance engenders an acute system-level imbalance: as edge language backbones scale down to compact regimes such as 0.5B or 1.5B parameters, the relative latency contribution of the vision encoder expands dramatically. In a mobile-oriented model pairing a SigLIP-400M vision encoder with a Qwen2.5-1.5B language backbone, visual feature extraction alone accounts for nearly 50% of the entire end-to-end inference latency on mobile hardware when processing a \(1024 \times 1024\) image. Concurrently, prior mobile Vision Transformers (such as EdgeViT and SH-ViT) were predominantly optimized for ImageNet classification rather than dense multimodal reasoning, and typically replicated a single homogeneous spatial mixer across all network stages. Such uniform designs neglect the shifting trade-offs between dense semantic representation and hardware memory access across different network depths. Furthermore, conventional contrastive pre-training objectives (e.g., CLIP and SigLIP) yield coarse, image-level representations misaligned with autoregressive generation, while masked reconstruction approaches lack the high-level semantic abstractions required by language decoders.

This paper tackles the challenge from the ground up: replacing monolithic, homogeneous backbones and coarse contrastive objectives with a hardware-informed heterogeneous architecture and generative multimodal pre-training. Core idea: systematically construct a macro-block heterogeneous pyramidal vision encoder (UltraViT) guided by real on-device NPU latency benchmarks, and train it via a two-stage strategy featuring dense spatial distillation followed by capacity-mixed frozen LLM generative supervision, natively achieving a 16× visual token reduction and over 1.7× speedup without post-hoc token compressors.

Method

Overall Architecture

UltraViT is tailored for edge LVLM deployment and adopts a standard four-stage pyramidal topology structured around the Metaformer macro-architecture block. An input high-resolution image \(I\) is first ingested by a convolutional stem that downsamples the spatial resolution by a factor of 4. The feature representations then pass through four distinct stages that progressively reduce spatial dimensions by a factor of two while expanding the channel capacity (\(C_{1-4} = [192, 512, 768, 1536]\) with block depths \(N_{1-4} = [4, 7, 10, 2]\)). The inter-stage downsampling blocks consist of a \(1 \times 1\) convolution, a strided \(3 \times 3\) depthwise convolution, another \(1 \times 1\) convolution, and a Squeeze-and-Excitation (SE) channel attention module. To retain both fine spatial details and deep semantics, a lightweight multi-scale feature aggregator fuses multi-stage representations for the multimodal projector. During pre-training, the student network first undergoes dense spatial distillation from a high-resolution teacher and subsequently receives generative autoregressive supervision from a dynamically alternated frozen LLM.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    In["Input Image (H × W × 3)"] --> Stem["Convolutional Stem<br/>4x Downsampling"]
    Stem --> S1["Stage 1: Convolutional Mixer<br/>DWConv + 1x1 Conv Local Modeling"]
    S1 --> S2["Stage 2: Interleaved Conv & Sparse Attention<br/>Strided Projections & NN Upsampling"]
    S2 --> S3["Stage 3: Channel-Partitioned Attention<br/>CPSA: 1/4 Channels for Contiguous Memory"]
    S3 --> S4["Stage 4: Full-Channel Single-Head Attention<br/>Global Context Modeling at Lowest Resolution"]
    S4 --> Agg["Multi-scale Feature Aggregator<br/>Strided Alignment + Concat + 1x1 Conv"]
    Agg --> Distill["Dense Feature Distillation<br/>Pixel Shuffle Reshaping + Dual Cosine Loss"]
    Distill --> GenPre["Dynamic Capacity LLM Generative Pre-training<br/>Alternating 0.5B / 1.5B Frozen Decoders"]
    GenPre --> Out["On-device Vision-Language Model (UltraVLM)"]

Key Designs

1. Macro-Block Heterogeneous Spatial Mixers: Depth-Specific Operator Adaptation for Mobile NPUs

Different network stages exhibit fundamentally divergent computational dynamics in terms of sequence length \(N = H \times W\) and channel depth \(C\). Applying a single homogeneous mixer throughout leads to either severe quadratic attention latency penalties in early stages or insufficient global semantic modeling in late stages. UltraViT preserves a unified residual meta-block topology while systematically deploying four tailored spatial mixers: - Convolutional Mixer (Stage 1): At early, high-resolution stages where sequence length \(N\) is excessively large, global routing is computationally prohibitive. The mixer utilizes a \(5 \times 5\) depthwise convolution flanked by \(1 \times 1\) pointwise projections, capitalizing on NPU-native parallel execution to inject locality and translation equivariance at maximum throughput. - Hardware-Adapted Sparse Mixer (Stage 2): Addressing the high latency of conventional sparse attention on mobile NPUs, four hardware-centric modifications are introduced: (1) multi-head attention is replaced with single-head attention; (2) discrete sampling is fused directly into \(Q, K, V\) strided convolutions; (3) transposed convolutions are replaced with hardware-friendly nearest-neighbor upsampling; and (4) projection dimensions are aggressively compressed with \(C_{qk}=16\) and value channels reduced to \(C/4\): $\(Q = \mathrm{Conv}_{s=4}(X; W^Q), \quad K = \mathrm{Conv}_{s=4}(X; W^K), \quad V = \mathrm{Conv}_{s=1}(X; W^V)\)$ $\(\tilde{X}_{global} = \mathrm{Softmax}\left(\frac{Q K^T}{\sqrt{C_{qk}}}\right) V, \quad Y = \mathrm{NN\_Upsample}(\tilde{X}_{global}) W^O\)$ - Channel-Partitioned Attention Mixer (Stage 3): Standard multi-head self-attention fragments channel dimensions across independent heads, causing severe memory fragmentation and suboptimal memory access patterns on mobile NPUs. Channel-Partitioned Self-Attention (CPSA) computes single-head attention on a small channel slice (\(C_p = C / 4\)), passing the remaining channels via an identity residual: $\(X_{att}, X_{res} = \mathrm{Split}(X; [C_p, C - C_p])\)$ $\(Y = \mathrm{Concat}\left(\mathrm{SH\text{-}Attention}(X_{att} W^Q, X_{att} W^K, X_{att} W^V), X_{res}\right) W^O\)$ - Full-Channel Single-Head Attention Mixer (Stage 4): In the final stage where spatial resolution is compact, attention complexity is no longer the main bottleneck. Full single-head attention is employed across all channels. Compared to multi-head mechanisms, single-head attention maintains contiguous memory layouts and simplifies tensor operations, unlocking global context without hardware memory fragmentation.

2. Physical On-Device Latency-Informed Architecture Selection and Feature Aggregation

Rather than relying on theoretical FLOPs or proxy CPU/GPU benchmarks that frequently diverge from edge execution realities, UltraViT performs an exhaustive block selection process measured natively on mobile hardware (Samsung Galaxy S25 Ultra, compiled via Qualcomm QNN SDK with INT8 quantization). Each mixer candidate was benchmarked across a comprehensive grid of token lengths and channel dimensions. The resulting Pareto-optimal configurations were trained with SigLIP multi-caption objectives and evaluated on zero-shot recognition, retrieval, and post-LLaVA fine-tuning accuracy. This empirical profiling established the optimal heterogeneous progression: pure convolutions in Stage 1, interleaved convolution and sparse attention in Stage 2, channel-partitioned attention in Stage 3, and single-head attention in Stage 4. To support fine-grained tasks such as document OCR and chart reasoning, a multi-scale feature aggregator projects features from intermediate stages to a unified resolution via strided convolutions, concatenates them along the channel dimension, and fuses them through a \(1 \times 1\) convolution.

3. Dense Representation Distillation via Structural Spatial Re-alignment

Conventional cross-architecture distillation typically downsamples high-resolution teacher feature maps to match the student resolution, irreversibly degrading localized spatial cues necessary for dense visual question answering. UltraViT preserves fine-grained details by re-aligning representations via Pixel Shuffle. When the spatial scale ratio \(r = H_T / H_S\) is an integer, student features \(F_S \in \mathbb{R}^{H_S \times W_S \times C_S}\) are reshaped into \(\hat{F}_S \in \mathbb{R}^{H_S \cdot r \times W_S \cdot r \times \frac{C_S}{r^2}}\), matching the teacher's spatial grid \(H_T \times W_T\). A dual-objective distillation loss combines patch-level localized alignment with global semantic preservation: $\(\mathcal{L}_{dense} = 1 - \frac{1}{H_T W_T} \sum_{i=1}^{H_T} \sum_{j=1}^{W_T} \frac{\hat{F}_S(i,j) \cdot F_T(i,j)}{\|\hat{F}_S(i,j)\| \|F_T(i,j)\|}\)$ $\(\mathcal{L}_{global} = 1 - \frac{\mathcal{P}_T(\hat{F}_S) \cdot \mathcal{P}_T(F_T)}{\|\mathcal{P}_T(\hat{F}_S)\| \|\mathcal{P}_T(F_T)\|}\)$ where \(\mathcal{P}_T(\cdot)\) denotes the teacher's frozen attention pooling operation, yielding the overall objective \(\mathcal{L}_{distill} = 0.5 \mathcal{L}_{dense} + 0.5 \mathcal{L}_{global}\).

4. Generative Pre-training with Dynamically Mixed Frozen LLM Decoders

Contrastive objectives (CLIP/SigLIP) provide coarse global alignment that is inherently misaligned with token-by-token autoregressive multimodal generation. Previous generative pre-training methods often employed tiny textual decoders, inadvertently shifting the syntactic and linguistic modeling burden onto the compact vision encoder and squandering its limited capacity. UltraViT attaches the vision encoder to a strong, pre-aligned, frozen LLM, completely relieving the encoder from modeling language syntax so its parameters remain dedicated to visual feature extraction. To balance training throughput with semantic depth, a dynamic LLM mixing strategy alternates the frozen decoder between a 0.5B and a 1.5B parameter model across pre-training batches. This prevents the linguistic degradation observed when training exclusively with small models while avoiding the prohibitive training overhead of exclusively using larger decoders.

Loss & Training

The complete training pipeline comprises three stages: 1. Dense Distillation Pre-training: Trained on a 150M subset of DataComp-1B for 25 epochs using a cosine learning rate schedule (peak LR \(1 \times 10^{-4}\), weight decay 0.1, batch size 32k) under \(\mathcal{L}_{distill}\). 2. Generative Pre-training: Initialized from stage 1, trained for 1 epoch on an 85M data mixture with a learning rate of \(1 \times 10^{-5}\), zero weight decay, batch size 192, and dynamically alternated 0.5B / 1.5B frozen Qwen2 decoders pre-aligned on 4M LLaVA-OV samples. 3. Supervised Fine-Tuning (SFT): The resulting UltraViT is integrated into LLaVA-OV to build UltraVLM, adhering strictly to the official 3-stage fine-tuning schedule (Step 2 uses 4M OCR and caption pairs; Step 3 uses 3.1M instruction examples) on 32 NVIDIA H100 GPUs using Deepspeed.

Key Experimental Results

Main Results

UltraVLM was benchmarked against the leading edge encoder FastVLM and major open-source multimodal baselines across visual question answering, document reasoning, and OCR benchmarks under an identical token budget (256 visual tokens per image). Mobile NPU throughput was measured on a Samsung Galaxy S25 Ultra smartphone in INT8 precision at \(512 \times 512\) resolution.

Model	Vision Encoder	Encoder Speed (inf./s)	GQA	TextVQA	DocVQA	InfoVQA	ChartQA	MMSTAR	Average
LLaVA-OV	SigLIP-400M	7.6	58.6	69.9	76.6	46.7	64.2	40.5	60.1
FastVLM (retrained)	FastViTHD	91.4	59.9	62.9	66.7	43.5	66.0	44.1	58.4
UltraVLM (Ours)	UltraViT	148.9	60.5	68.9	71.8	47.9	72.2	49.8	63.4
QwenVL-2 (256 tokens)	QwenVL-2	3.7	59.3	66.8	64.5	30.7	49.4	43.0	54.4
QwenVL-2.5 (256 tokens)	QwenVL-2.5	6.9	60.5	66.5	61.8	35.0	71.5	55.0	60.7

Compared to post-hoc token reduction techniques applied to standard vision backbones, UltraVLM demonstrates the superior efficiency of an intrinsic pyramidal architecture:

Method	Token Reduction	Vision Enc Speedup	TextVQA	DocVQA	ChartQA	MMSTAR	Average Score
LLaVA-OV Baseline	1.0×	1.0×	69.9	76.6	64.2	40.5	63.8
VisionZip	5.7×	1.0×	62.0	55.1	50.0	38.1	56.4
PyramidDrop	4.6×	1.0×	60.3	54.5	47.9	37.5	55.4
VisPruner	5.7×	1.0×	54.8	51.2	40.0	38.4	52.8
HiRED	5.0×	1.0×	59.8	52.3	44.1	39.3	54.9
UltraVLM (Ours)	16.0×	19.1×	68.9	71.8	72.2	49.8	65.1

Ablation Study

Ablations systematically isolate the contributions of dense distillation and generative pre-training components:

Experiment Group	Model / Pre-training Variant	DocVQA	InfoVQA	MMSTAR	SQA	Note
Distillation Objective	UltraViT + Contrastive (SigLIP)	58.9	36.4	42.3	75.8	Standard contrastive baseline
Distillation Objective	UltraViT + Dense Distillation	63.9	39.5	44.6	76.9	Substantial gain on fine-grained OCR (+5.0% DocVQA)
Distillation Objective	UltraViT + Dense Distill + Large Teacher	67.5	42.8	44.4	78.5	Demonstrates strong scaling with teacher capacity
Generative Decoder	Dense Distill (+ Large Teacher)	67.5	42.8	44.4	78.5	Without Stage 2 generative pre-training
Generative Decoder	+ Frozen 0.5B LLM Supervision	70.0	44.7	47.3	81.5	Limited decoder capacity strains visual features
Generative Decoder	+ Frozen 1.5B LLM Supervision	72.0	46.8	49.1	83.1	Strongest semantic alignment but higher compute
Generative Decoder	+ Dynamic LLM Mixing (0.5B / 1.5B)	71.8	47.9	49.8	82.9	Matches 1.5B accuracy with significantly reduced cost

Key Findings

• Heterogeneous Mixers Outperform Homogeneous Designs: Under identical macro parameters, a pure convolutional baseline (C, C, C, C) achieves higher raw throughput (171 inf./s) but lags in multimodal reasoning (56.4 average score). The proposed heterogeneous configuration (C, C+S, CP, A) maintains high throughput (148.9 inf./s) while elevating average task accuracy to 58.5, particularly boosting dense perceptual grounding.
• Superior Pre-training Sample Efficiency: Compared to FastVLM's contrastive distillation requiring 2.0B unique images and 13.0B seen samples, UltraViT's two-stage pre-training processes only 235M unique images and 4.1B total samples (~10× fewer unique images, 3× fewer total samples) while securing superior downstream multimodal alignment.
• Robust Latency Scaling Across Resolutions: Across input resolutions ranging from \(256 \times 256\) to \(1536 \times 1536\), UltraViT consistently maintains a ~1.7× inference speedup over FastViT on the Samsung Galaxy S25 Ultra NPU, and achieves a 19.1× speedup over SigLIP-400M.

Highlights & Insights

• Physical On-Device NPU Profiling in Backbone Search: Bypassing misleading theoretical FLOPs, UltraViT profiles operator latency directly on physical smartphone hardware, providing the first validated heterogeneous operator design space for edge vision encoders.
• Lossless Dense Distillation via Feature Reshaping: Employing Pixel Shuffle spatial-to-channel restructuring eliminates the information loss caused by naive downsampling, allowing compact edge encoders to inherit dense localization from high-resolution teachers.
• Decoupled Generative Alignment via Frozen LLMs: Identifying small decoders as a parasitic drain on visual capacity, the dynamic frozen LLM supervision framework allows the vision encoder to dedicate 100% of its parameters to visual grounding while achieving generative alignment.

Limitations & Future Work

• Author-Acknowledged Limitations: The two-stage pre-training framework requires access to an already vision-aligned LLM as a teacher decoder, adding an extra preparation dependency relative to monolithic self-supervised regimes.
• Future Directions: Operator latency was primarily measured on Qualcomm QNN and Snapdragon NPUs; evaluating kernel fusion and INT8 quantization behaviors across Apple Neural Engine (ANE) and MediaTek NeuroPilot platforms remains an important next step. Extending the heterogeneous mixer paradigm to spatiotemporal video encoding represents another promising avenue.

Related Work & Insights

• vs FastVLM (CVPR 2025): While FastVLM pioneered latency-aware vision encoding for LVLMs using FastViT backbones and contrastive distillation, UltraViT achieves an additional 1.7× physical speedup via heterogeneous mixers and delivers superior multimodal accuracy with 10× fewer unique pre-training images.
• vs Post-hoc Token Pruning (VisionZip / PyramidDrop / HiRED): Post-hoc pruning techniques reduce LLM sequence length but leave the heavy vision encoder latency untouched; UltraViT's pyramidal design natively shrinks tokens by 16× within the encoder, delivering 19.1× encoder speedup and comprehensive end-to-end acceleration.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Establishes the first physical NPU-guided heterogeneous architecture search for LVLM vision encoders and introduces decoupled generative pre-training.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive physical smartphone NPU benchmarking across resolutions, extensive multimodal benchmark evaluations, and rigorous ablations.
• Writing Quality: ⭐⭐⭐⭐⭐ Cohesive narrative, sound theoretical justification, and transparent hardware-software co-design rationale.
• Value: ⭐⭐⭐⭐⭐ Provides an immediately deployable, state-of-the-art vision encoder architecture and training paradigm for on-device multimodal intelligence.