Efficient Quantization-Aware Adaptation for Visual Foundation Models

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/leenas233/EQuA
Area: Model Compression / VLM Efficiency
Keywords: Visual Foundation Models, Quantization-Aware Fine-Tuning, Memory Optimization, Side Adapter, Zero-Overhead Deployment

TL;DR

Addressing the severe memory explosion caused by cached intermediate activations and activation quantizers when fine-tuning visual foundation models, EQuA decouples a lightweight sub-network side adapter branch from the backbone based on activation-scaled weight importance, cuts training memory by over 70%, and merges the adapter back into the backbone for zero-overhead quantized deployment.

Background & Motivation

In the large language model (LLM) domain, parameter-efficient fine-tuning (PEFT) techniques combined with low-bit quantization—such as QLoRA, QA-LoRA, and PEQA—have become the de facto standard for resource-constrained downstream adaptation. The fundamental premise of these methods is that LLM GPU memory consumption is predominantly driven by massive pre-trained floating-point weights (e.g., 138 GB for LLaMA2-70B in FP16). Consequently, compressing the static weights into 4-bit integers and optimizing compact adapters substantially alleviates training memory constraints.

However, directly transferring this paradigm to visual foundation models such as SAM-H and Vision Transformers (ViTs) fails completely. Visual foundation models maintain relatively modest weight parameters compared to LLMs (for example, SAM-H weights take merely 2.4 GB, accounting for only 3.3% of the total fine-tuning memory footprint), but they process high-resolution visual inputs and dense prediction tokens that produce massive intermediate feature activations. Empirical measurements reveal that the primary memory bottlenecks during quantization-aware adaptation stem from two distinct sources: cached intermediate activations across deep ViT blocks required for backpropagation (consuming over 65% of memory) and the backward pass through numerous activation quantizers (consuming an additional 21.8%). As a result, methods like QA-LoRA still demand nearly 70 GB of VRAM on SAM-H, rendering single-GPU training on commodity hardware (e.g., 24 GB RTX 3090) impossible. While side-tuning methods like QST isolate the backbone to curb memory, their detached adapters incur substantial inference latency at deployment and absorb unmitigated quantization errors from the frozen backbone.

The angle of attack in this paper is to restructure the computation graph topology during adaptation: eliminate both backbone and activation-quantizer activation caching entirely while ensuring that the adapted components can be natively folded back into the backbone at deployment. Core idea: dynamically split a lightweight Sub-network Side Adapter (SSA) from the backbone linear layers based on activation-scaled weight importance, freeze the memory-intensive backbone branch during adaptation with memory-free gradient recomputation and block-wise activation tuning (BAQF), and merge the adapter back into the quantized backbone at deployment for zero extra computation and parameter overhead.

Method

Overall Architecture

EQuA divides the adaptation workflow into three synchronized phases: importance-guided sub-network decomposition, side adapter quantization-aware fine-tuning, and deployment-time lossless merging. Before adaptation begins, the channel importance of linear layer weights is evaluated via input-activation-scaled magnitudes, decoupling the Multi-Head Self-Attention (MHSA) projection matrices and Multi-Layer Perceptron (MLP) weights into a Memory-Intensive Backbone (MIB) branch and a low-dimensional Sub-network Side Adapter (SSA) branch. During adaptation, the MIB branch parameters and activation quantizers are excluded from the backward computation graph, drastically cutting activation caching. A one-step analytical gradient recomputation feeds back supervisory signals from MIB into SSA, while Block-wise Activation Quantization Fine-tuning (BAQF) periodically updates activation scaling factors locally. Upon deployment, the SSA weights are re-concatenated with the MIB weights along their original channel indices, yielding a standard uniform quantized model without external adapter modules.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image & Features F"] --> B["Activation-Scaled Importance Estimation<br/>Calculate channel probability P_M & sample"]
    B --> C["Sub-network Side Adapter (SSA) Decomposition<br/>Split MHSA V/Proj & MLP linear layers"]
    C --> D["Side Adapter Quantization-Aware Fine-tuning (SAQF)<br/>Freeze MIB computation graph + attach LoRA on SSA"]
    D --> E["Backbone Supervisory Gradient Recomputation (GR)<br/>One-step analytical MIB input gradient via masks"]
    E --> F["Block-wise Activation Quantization Fine-tuning (BAQF)<br/>Minimize block MSE every K epochs to update s_a"]
    F --> G["Deployment Weight Reconstruction & Merging<br/>Inverse-concatenate SSA & MIB with zero extra overhead"]

Key Designs

1. Activation-scaled sub-network side adapter decomposition: channel-importance-guided parameter-free decoupling

To eliminate deployment latency and redundant parameters caused by external adapter structures, EQuA directly splits a sub-network from within the ViT linear layers. Exploiting block-matrix multiplicability in MHSA (\(W^V, W^{Proj}\)) and MLP (\(W^{Lin1}, W^{Lin2}\)): $\(MHSA(X) = AX W_{mib}^V W_{mib}^{Proj} + AX W_{ssa}^V W_{ssa}^{Proj} = MHSA_{mib}(X) + MHSA_{ssa}(X)\)$ $\(MLP(X) = \text{GeLU}(X W_{mib}^{Lin1}) W_{mib}^{Lin2} + \text{GeLU}(X W_{ssa}^{Lin1}) W_{ssa}^{Lin2} = MLP_{mib}(X) + MLP_{ssa}(X)\)$ The non-decomposable attention map weights (\(W^Q, W^K\)) remain shared and unsplit. Rather than relying solely on raw weight magnitudes, which overlook domain-specific activation shifts in visual downstream tasks, EQuA scales weight magnitudes by the token-wise \(L_2\) norm of input activations across batch and sequence dimensions: \(M_i = \sum_{j=1}^{D_{out}} \|X_{:, i}\|_2 \cdot |W_{i, j}|\). Applying Softmax transforms \(M_i\) into probability distribution \(P_i^M\), guided by which SSA channel indices \(\mathcal{I}_{ssa}\) are stochastically sampled. This stochastic selection provides smoother optimization than a greedy Top-\(D_s\) cut, and enables direct dimension-wise re-merging into the original matrix shape at deployment.

2. Side adapter quantization-aware fine-tuning with memory-free gradient recomputation

During adaptation, the high-dimensional MIB branch is frozen and detached from PyTorch's backward computation graph, confining activation caching exclusively to the narrow SSA branch of dimension \(D_s\) (default 64). To further curb optimizer state overhead, low-rank adaptation (LoRA) matrices are wrapped around the quantized SSA linear layers: $\(\hat{W}_{ssa} = \text{Dequant}(\text{Quant}(W_{ssa} + \alpha \cdot A_r B_r, s_w, z_w), s_w, z_w)\)$ This restricts trainable parameters to merely 1.2% of the full model. To prevent performance collapse caused by completely severing backbone feedback, EQuA introduces a one-step analytical Gradient Recomputation (GR) for the MIB input activations (\(X_{mib}^{Lin1}, X_{mib}^V\)). By caching only lightweight pre-clipping Boolean quantization masks (e.g., \(\mathds{1}^{Proj}_{mib}, \mathds{1}^{Lin2}_{mib}\)) during forward propagation, the input gradients of the frozen MIB branch are analytically synthesized without storing full activation tensors and injected into the SSA gradient flow.

3. Block-wise activation quantization fine-tuning: local error calibration without full-model backpropagation

Global fine-tuning of activation quantizers introduces severe memory overhead (21.8%) due to end-to-end caching across all transformer layers. To eliminate this overhead while preventing severe visual quantization noise, EQuA designs a localized Block-wise Activation Quantization Fine-tuning (BAQF) routine. Every \(K\) epochs of SAQF, all model weights and adapter parameters are temporarily locked. The model iterates block by block through each ViT layer, minimizing the Mean Squared Error (MSE) between the output of quantized block \(\mathcal{F}_l(\hat{Y}_{l-1})\) and its full-precision counterpart \(F_l^{fp}(Y_{l-1})\): $\(s_a \leftarrow s_a - \eta \cdot \nabla_{s_a} \mathbb{E}\left[ \|F_l^{fp}(Y_{l-1}) - \mathcal{F}_l(\hat{Y}_{l-1})\|_2^2 \right]\)$ Because this optimization is confined strictly within individual blocks (running for 100 local iterations), the additional memory overhead is virtually zero, yet it effectively stops activation clipping errors from compounding across deep layers.

Key Experimental Results

Main Results

Experiments were conducted on SAM-Huge (SAM-H, 641M backbone parameters) and ViT-Base (ViT-B, 86M parameters) across polyp segmentation (CVC-T, Kvasir, ETIS), camouflaged object detection (CAMO, COD10K), agricultural leaf disease segmentation (Leaf), and the VTAB-1K benchmark (19 classification datasets). The table below reports the downstream segmentation performance and memory footprint of SAM-H under the challenging 4-bit (W4A4) setting (data from Table 1 and Table 7 in the original paper).

Method	Bit Setting	Trainable Param	Train Memory	CVC-T (\(S_\alpha\))	Kvasir (\(S_\alpha / E_\phi\))	Leaf (mIoU / mDice)	Latency (b=1)	Speedup
Full (FP)	W32A32	641.09 M	49.6 GB	0.917	0.917 / 0.954	0.710 / 0.818	0.75 s	1.00×
LoRA (FP)	W32A32	8.03 M	45.4 GB	0.937	0.936 / 0.962	0.730 / 0.831	0.75 s	1.00×
LSQ (QAT)	W4A4	641.46 M	72.4 GB	0.876	0.895 / 0.931	0.697 / 0.807	0.54 s	1.38×
QA-LoRA	W4A4	7.13 M	68.9 GB	0.919	0.923 / 0.948	0.705 / 0.815	0.54 s	1.38×
PEQA	W4A4	4.43 M	69.4 GB	0.927	0.926 / 0.952	0.718 / 0.819	0.54 s	1.38×
QST	W4A4	8.30 M	21.6 GB	0.905	0.894 / 0.921	0.656 / 0.772	0.63 s	1.18×
EQuA (Ours)	W4A4	7.67 M	18.5 GB	0.922	0.915 / 0.947	0.709 / 0.816	0.54 s	1.38×

Ablation Study

The following table details the impact of progressively incorporating each component into W4A4 SAM-H on trainable parameters, training duration, memory consumption, and segmentation metrics on Kvasir (data from Table 3 in the original paper).

Configuration	Component Mix	Param	Train Time	Memory	Kvasir \(S_\alpha\)	Kvasir \(E_\phi\)	Mechanism & Impact
Baseline QAT	Base (LSQ)	641.46 M	0.60 h/ep	72.4 GB	0.895	0.931	Full backpropagation with massive activation caching
In-graph sub-branch	+ SSA* (MIB in graph)	29.90 M	0.57 h/ep	67.6 GB	0.918	0.941	Fine-tuning only sub-network, but MIB graph still cached
Detached MIB graph	+ SSA (MIB excluded)	29.90 M	0.32 h/ep	18.6 GB	0.881	0.913	Memory drops by 72.5%, but lacks backbone supervisory flow
Gradient recomputation	+ SSA & GR	29.90 M	0.50 h/ep	18.6 GB	0.910	0.932	Restores MIB supervision without memory cost (\(S_\alpha\) +0.029)
Low-rank adaptation	+ SSA & GR & LoRA	7.67 M	0.50 h/ep	18.5 GB	0.906	0.926	Trainable parameters compressed to 1.2%, metrics stay solid
Block activation tuning	+ SSA & GR & LoRA & BAQF	7.67 M	0.52 h/ep	18.5 GB	0.915	0.947	Mitigates activation clipping error with zero extra memory

Key Findings

• Overcoming the GPU memory barrier: On SAM-H, EQuA slashes training memory from 72.4 GB (standard QAT) and 68.9 GB (QA-LoRA) to 18.5 GB (a 73.3% reduction), successfully enabling full quantization-aware adaptation on a single commodity 24 GB GPU (RTX 3090/4090).
• Outperforming detached side-tuning baselines: Compared to QST, which also aims at memory efficiency, EQuA achieves lower memory (18.5 GB vs. 21.6 GB) while improving segmentation performance on the Leaf dataset by 0.053 mIoU and 0.044 mDice. Crucially, by re-integrating weights into the backbone at deployment, EQuA delivers a 1.38× inference speedup (native 4-bit latency) versus QST's lagged 1.18×.
• Orthogonal compatibility: Ablations confirm that \(D_s=64\) and \(r=4\) hit the sweet spot of performance and efficiency. Furthermore, combining EQuA with direct low-precision activation methods like COAT reduces memory even further from 18.5 GB to 12.7 GB.

Highlights & Insights

• Split-and-Merge topology: Unlike standard PEFT methods that force a trade-off between external adapter latency and full-model fine-tuning memory, EQuA leverages the additive nature of matrix multiplications to split out linear sub-networks, tuning them in isolation and fusing them seamlessly post-training for zero-overhead inference.
• Analytical gradient recomputation: By saving only compact 1-bit boolean clipping masks instead of full intermediate activation tensors, EQuA analytically reconstructs the input gradients of frozen layers, demonstrating an elegant compromise between arithmetic computation and memory retention.
• Decoupled block-wise quantization tuning: Separating sensitive activation scale optimization from global backpropagation and solving it via localized block-wise MSE calibration avoids both the 21.8% activation quantizer memory overhead and inter-layer error compounding.

Limitations & Future Work

• Inability to split attention query and key projections: Due to the non-linear matrix multiplication in attention map computation (\(Q K^T\)), \(W^Q\) and \(W^K\) cannot be expressed as block-matrix additions and must remain entirely frozen, potentially constraining adaptation capacity on tasks requiring major attention shifts.
• Alternating training orchestration: Incorporating BAQF requires switching between standard SSA fine-tuning and block-wise MSE updates every \(K\) epochs, adding hyper-parameters (\(K\) and local iterations) that must be configured per task.
• Generalization to video and 3D vision foundation models: Current evaluations focus on 2D vision models (SAM-H, ViT-B). Extending the SSA split-and-merge paradigm to video architectures and 3D point cloud backbones with even larger spatiotemporal activation tensors represents an impactful next step.

Related Work & Insights

• vs. QA-LoRA / PEQA: These LLM-centric approaches focus on weight compression because weights dominate LLM memory, but they fail on vision foundation models where activation caching consumes over 85% of training memory. EQuA directly resolves the activation bottleneck.
• vs. QST: While QST attaches external side adapters to cut memory, it cannot fuse them into the backbone at deployment (incurring latency penalties) and fails to optimize activation quantizers. EQuA achieves superior accuracy, lower memory, and zero deployment overhead.
• Transferable Insight: The split-and-merge strategy can be broadly adapted beyond 4-bit uniform quantization to structured pruning, continuous domain adaptation, and modular on-device model personalization.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Elegant structural decomposition of ViT linear layers coupled with zero-overhead deployment and analytical gradient recomputation.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across SAM and ViT on over 20 medical, natural, and agricultural datasets, backed by hardware-level memory and latency profiling.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear problem formulation, rigorous mathematical derivations, and coherent narrative.
• Value: ⭐⭐⭐⭐⭐ Solves the critical practical hurdle of running 4-bit quantization-aware adaptation of massive vision foundation models on single consumer GPUs.

Related Papers

• [CVPR 2026] Curvature-Aware Zeroth-Order Optimization for Memory-Efficient Test-Time Adaptation
• [ECCV 2026] VQT: Vector Quantization Tuning for Efficient Fine-tuning and Compression of Pre-trained Vision Transformers
• [ECCV 2026] MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptation
• [CVPR 2026] Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradients
• [CVPR 2026] MASQuant: Modality-Aware Smoothing Quantization for Multimodal Large Language Models