Activation Quantization of Vision Encoders Needs Prefixing Registers¶
Conference: ECCV2026
Official Paper: 5152
Paper: PDF
Code: https://github.com/spbob0418/RegCache
Area: VLM Efficiency / Vision Encoder Quantization
Keywords: activation quantization, register caching, attention sinks, middle-layer prefixing, token deletion
TL;DR¶
RegCache extracts reusable middle-layer register KV caches from reference images and combines prefix insertion with residual outlier token deletion to improve post-training quantization, raising CLIP-B/16 ImageNet-1k accuracy under ERQ W4A4 from 1.56% to 46.07%, without universally recovering full-precision performance.
Background & Motivation¶
Vision encoders such as CLIP and SigLIP run both independently and inside VLM image and video pipelines. Unlike token-by-token autoregressive language generation, visual encoding is more likely to be compute-bound, making weight-only compression insufficient: quantizing weights and activations enables low-precision matrix multiplication. A few exceptionally large activations complicate this process by expanding the quantization range and reducing resolution for ordinary values. The paper shows substantial degradation in some vision encoders even at 8 bit.
Language-model methods already use attention-sink prefixes to mitigate outliers because special tokens such as BOS and SEP have fixed identities in a vocabulary. Visual patches instead occupy a continuous embedding space, and a background patch's role depends on image content. There is no vocabulary entry that directly identifies a universal visual register. Layerwise quantization reveals that sensitive positions often lie in middle-block MLP projections, coinciding with the emergence of FC2 input outliers. Images with their foreground retained and background zeroed develop outliers earlier, supporting the explanation that the model first needs several layers to identify patches suitable for nonsemantic computation.
Although outlier positions differ across images, their middle-layer representations are similar. In the SigLIP-B/16 analysis using 64 images, mean cosine similarity is 0.89 for outlier tokens versus 0.26 for ordinary tokens. This suggests reusing internal states rather than training new input tokens. Core idea: extract the KV states of middle-layer outlier tokens as external registers, then delete residual internal outlier tokens from each test image, moving the computation responsible for excessive quantization ranges away from ordinary image tokens.
Method¶
Overall Architecture¶
RegCache is an outlier-mitigation module used with an existing PTQ pipeline, not a new quantizer, and it requires no register backpropagation. Offline, it identifies a quantization-sensitive layer, curates candidate tokens, obtains their blockwise KV caches with the full-precision encoder, averages those caches, and searches the prefix count. Online, early layers remain unchanged; selected middle layers use the cached prefixes, while residual high-norm patch tokens are deleted at the input of the most sensitive block. The output remains a visual representation for classification, retrieval, or a VLM.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Reference images and<br/>pretrained vision encoder"] --> B["Sensitive-layer and<br/>candidate selection"]
B --> C["Middle-layer<br/>KV prefix caching"]
T["Test image<br/>Unchanged early layers"] --> D["Residual token deletion"]
C -->|Prefix-assisted middle-layer inference| D
D --> E["Subsequent quantized encoding<br/>with selected caches"]
E --> F["Visual representations<br/>Classification, retrieval, or VLM"]
The three designs address connected questions: which states to cache, where and how to reuse them, and what to do when outliers remain. The prefix start layer need not equal the most sensitive layer. Consequently, the diagram does not mean that all cached layers run before deletion; deletion occurs specifically at the most sensitive block's entrance.
Key Designs¶
1. Sensitive-layer and candidate selection: locate register sources through quantization sensitivity
The procedure independently quantizes one layer at a time and measures the accuracy drop on a reference ImageNet-1k training-split classification task. The layer causing the largest drop is selected. If a deployment PTQ algorithm is already chosen, that algorithm is used for sensitivity analysis; otherwise, round-to-nearest (RTN) quantization is used. This avoids assuming that every encoder develops outliers at a fixed depth and makes the search specific to the intended quantizer. A sensitive "layer" often means a sublayer such as FC2, whereas deletion occurs at the input of the transformer block containing it; these are not the same location.
Reference images are then processed, and tokens at the sensitive layer's input are ranked by their \(\ell_\infty\) norm, the maximum absolute activation over their channels. The experiments sample 50,000 images from the ImageNet-1k training split and retain the 100 highest-norm candidate tokens, not 100 tokens from every image. Because sink tokens can emerge earlier, the same search is also performed in up to three preceding blocks, forming separate candidate sets. High norm does not establish semantic importance: here, cross-image similarity motivates treating these states as reusable nonsemantic computation.
2. Middle-layer KV prefix caching: reuse internal states instead of learning new patches
For each candidate, the unquantized encoder supplies key/value states from a candidate starting middle layer through subsequent blocks. Corresponding KV caches are averaged across candidates to form a reusable register. Inference provides these states directly as prefix caches, rather than concatenating a reference image with the test image or retraining the backbone. For a fixed model and quantization algorithm, the cache is reused across test images without a per-image register search or optimization.
One averaged register may not sufficiently absorb internal sink behavior. The method therefore duplicates the same cache and searches a prefix count \(\tau\in\{1,2,\ldots,15\}\), selecting the value with the highest reference classification accuracy. Early layers receive no prefixes because raw patches have not yet developed stable, reusable nonsemantic states. In the experiments, CLIP and SigLIP receive prefixes from the selected starting layer through the final layer; DINOv2 instead performs better when prefixing only the searched layer. Thus, prefixing every middle-to-final layer is not a universal rule across visual pretraining regimes.
3. Residual token deletion: remove internal outliers that prefixes fail to replace
External registers do not guarantee that every internal sink disappears. For each test image, the method selects and deletes the \(\tilde{k}\) native image patch tokens with the largest \(\ell_\infty\) norms at the input of the most sensitive block. The reference task also determines the deletion count. The deleted objects are internally emerging image tokens, not the external KV prefixes. Cache parameters can be fixed offline, but norm computation and token selection still depend on the current image, so the full method is not free of per-image processing.
Deletion and caching need to work together. Deletion alone can remove sink functionality the model still relies on; caching alone can leave the quantization range excessively wide. On SigLIP-B/16, deletion alone yields only 42.41% accuracy against a 69.71% baseline. On SigLIP2-B/16, caching alone reduces accuracy from 26.04% to 23.82%. High-norm tokens are therefore not simply disposable: the availability of a suitable external replacement matters.
A Worked Example¶
Consider SigLIP2-B/16 under W8A8. Offline, the method selects 100 high-norm candidates at each investigated position from 50,000 training images, builds averaged KV caches, selects a prefix count between 1 and 15, and tunes the deletion count. The main paper does not report this model's final selected prefix and deletion counts; the search limits must not be presented as deployment settings.
A test image passes through unchanged early layers, uses register prefixes at the selected middle layers, loses residual high-norm patches at the most sensitive block's entrance, and continues through the encoder. Main-paper Table 7 reports 26.04% accuracy for the original quantized model, 23.82% for caching alone, 69.06% for deletion alone, and 72.35% for the combination. This is not a pipeline in which adding any individual component must monotonically improve performance: the effect depends on the other operation and on the model.
Loss & Training¶
RegCache introduces no training loss, updates no model weights, and trains no register embeddings. Its cost comes from layerwise sensitivity analysis, cache construction, and reference-task hyperparameter selection. The authors report approximately 1 hour of total search using naive RTN pseudo-quantization on an RTX 4090. This is not a guarantee of identical search time for every PTQ baseline and encoder.
Training-free does not mean data-free or label-free. The main procedure uses ImageNet-1k training-split classification accuracy and a candidate pool of 50,000 images. The text mentions an unlabeled reconstruction-loss alternative in the appendix, but that appendix is absent from the available cache, so its loss definition, data budget, and results cannot be supplied here. Cached text for original equations (1) and (2) is corrupted; this note retains the prose-supported ranking, averaging, and deletion definitions without reconstructing the missing equations.
Key Experimental Results¶
Main Results¶
Standalone encoders include CLIP, OpenCLIP, SigLIP, SigLIP2, and DINOv2, evaluated at W4A4, W6A6, and W8A8, denoting the respective weight and activation bitwidths. Baselines include PTQ4ViT, RepQ-ViT, NoisyQuant, FIMA-Q, ERQ, and naive quantization. The following selection comes from main-paper Tables 2, 3, and 4. Gains are percentage-point differences between a baseline and its RegCache counterpart within the same setting, not comparable improvements across tasks.
| Model and evaluation | Quantization setting | Baseline | With RegCache | Gain |
|---|---|---|---|---|
| CLIP-B/16, ImageNet-1k accuracy | ERQ W4A4 | 1.56 | 46.07 | +44.51 |
| CLIP-B/16, ImageNet-1k accuracy | FIMA-Q W4A4 | 50.41 | 62.08 | +11.67 |
| OpenCLIP-B/16, ImageNet-1k accuracy | RepQ-ViT W6A6 | 46.51 | 68.22 | +21.71 |
| SigLIP2-B/16, ImageNet-1k accuracy | PTQ4ViT W6A6 | 41.54 | 69.02 | +27.48 |
| DINOv2-B/14, ImageNet-1k accuracy | RepQ-ViT W8A8 | 82.27 | 81.55 | -0.72 |
| CLIP-B/16, COCO image-to-text R@1 | FIMA-Q W4A4 | 40.92 | 51.82 | +10.90 |
| Qwen3-VL-2B, VQAv2 | RepQ-ViT, 4 bit | 32.03 | 42.72 | +10.69 |
| Qwen3-VL-8B, Video-MME | RepQ-ViT, 4 bit | 50.44 | 52.89 | +2.45 |
Full-precision CLIP-B/16 classification accuracy is 68.32%, so even the large ERQ W4A4 improvement leaves a substantial gap. Full-precision Qwen3-VL-2B scores 77.05% on VQAv2, also far above 42.72% with RegCache. The evidence supports mitigating low-bit degradation, not universally lossless compression.
Ablation Study¶
Main-paper Table 7 reports ImageNet-1k accuracy under naive W8A8 quantization. PC denotes prefix caching and TD denotes token deletion. The combination is best, but this does not mean that both components contribute equally on every model.
| Configuration | SigLIP-B/16 | SigLIP2-B/16 |
|---|---|---|
| Baseline | 69.71 | 26.04 |
| PC only | 74.37 | 23.82 |
| TD only | 42.41 | 69.06 |
| PC + TD | 74.38 | 72.35 |
On SigLIP, the combination exceeds caching alone by just 0.01 percentage points, so caching supplies most of the gain. On SigLIP2, deletion alone recovers most performance, and adding caching contributes another 3.29 points. This difference is more informative about model dependence than an unqualified claim of strong synergy.
Key Findings¶
Main-paper Table 6 averages the maximum input token norm at the sensitive layer over 500 images. CLIP falls from 41.38 to 11.45, and SigLIP2 from 148.20 to 15.16. Reduced outlier magnitude is directionally consistent with improved quantized accuracy, but this association is not a universal, single-cause proof.
Prefixes searched on ImageNet-1k also transfer to other datasets. In Table 5, CLIP-B/16 improves from 33.30% to 74.68% on Food-101, supporting cross-dataset reuse. This does not establish that a single prefix transfers between different models.
Deployment measurements use an NVIDIA A6000 and TensorRT. In Table 8, CLIP-B/16 at batch size 64 takes 132.13 ms at full precision, 60.64 ms with INT8, and 61.27 ms with INT8 + RegCache. The latter adds 1.04% over INT8; SigLIP-B/16 incurs 1.54% overhead. These are real INT8 measurements, not evidence of an identical W4A4 speedup. The Qwen3-VL latency evaluation likewise uses an INT8 deployment setting, distinct from the 4 bit accuracy results above.
Highlights & Insights¶
- The method connects visual quantization outliers to a specific internal computational role instead of only adjusting quantization scales after outliers appear. The reusable object is a middle-layer KV state, not a fixed input patch.
- Candidate selection is driven by quantization sensitivity and downstream quantized accuracy. This distinguishes the objective from adding registers to improve attention-map interpretability.
- Caching and deletion jointly reshape outlier behavior, allowing the method to complement multiple PTQ baselines. Benefits still depend on how much error the base quantizer already removes and whether sink tokens dominate the remaining problem.
Limitations & Future Work¶
- The authors explicitly acknowledge heuristic selection of prefix counts, deletion counts, and related hyperparameters for each encoder and base quantization algorithm. Training-free operation reduces cost but does not eliminate calibration or search.
- DINOv2 prefers a different insertion range and degrades in some 8 bit settings; low-bit performance is not universally repaired either. Differences between text-supervised and self-supervised encoders need further explanation.
- From this note's assessment, classification, retrieval, and VQA results do not fully establish the safety of high-norm patch deletion for dense prediction, spatial alignment, or fine-grained local evidence. Deleted tokens cannot be assumed useless for every downstream task.
- The cache lacks the appendix referenced by the main text. The unlabeled alternative, additional VLM results, exact hyperparameters, and output-token alignment implementation cannot be verified here and require further inspection for reproduction.
- Low-bit accuracy gains and deployment acceleration need separate validation. Low INT8 overhead on one GPU at specified batch sizes cannot be directly extrapolated to all edge devices or 4 bit kernels.
Related Work & Insights¶
- Vision Transformers Need Registers adds registers during training to improve internal representations. RegCache extracts caches from an existing model, avoids retraining, and focuses on activation quantization.
- Vision Transformers Don't Need Trained Registers moves maximum activations in frequently outlying channels into an extra token at inference. RegCache primarily reuses precomputed token-level KV states but still performs per-image token deletion.
- Language-model prefix methods supply the starting idea of mitigating outliers with attention-sink prefixes. Visual candidate discovery, middle-layer insertion, and residual deletion address the differences between continuous patch embeddings and discrete vocabularies.
- RepQ-ViT, PTQ4ViT, FIMA-Q, and ERQ address quantization through scales, quantization functions, or error compensation. RegCache changes the model's outlier behavior and complements rather than replaces these methods.
Rating¶
- Novelty: 4/5. Prefixing has precedents, but visual middle-layer candidate construction and coordinated deletion provide a targeted adaptation.
- Experimental Thoroughness: 4/5. Multiple encoders, PTQ methods, classification, retrieval, VLM tasks, ablations, and real latency are covered; dense tasks and broader hardware validation remain gaps.
- Writing Quality: 4/5. Observations, method, and ablations are clearly connected, although some broad claims exceed the configuration-level evidence in the tables.
- Value: 4/5. A practical training-free quantization plugin for outlier-dominated low-bit settings, with a continuing need for model-specific tuning.