Skip to content

A Target-Centric Survey of Quantization-Aware Training

Conference: EMNLP 2026
Paper: ACL Anthology
Area: Model Compression
Keywords: Quantization-Aware Training, Target-Centric Taxonomy, Straight-Through Estimator, Large Language Models, Deployment Fidelity

TL;DR

Addressing the limitation of prior quantization surveys that overlook the distinct numerical properties and lifecycles of different tensor targets, this paper presents the first target-centric survey of Quantization-Aware Training (QAT), deconstructing weight, activation, joint W+A, KV cache, and gradient quantization to formulate cross-target transferability rules, empirical evaluation protocols, and a rigorous minimal reporting checklist for real hardware deployment.

Background & Motivation

The exponential growth in foundation model scale incurs prohibitive memory footprints and massive computational demands, hindering real-world deployment across edge platforms and commodity accelerators. Post-Training Quantization (PTQ) provides a calibration-free path to compression, yet it encounters catastrophic performance degradation in aggressive ultra-low-bit regimes (e.g., INT4 or INT2) or when processing representations with extreme structured outliers, such as the attention heads and residual streams of Transformer architectures. Quantization-Aware Training (QAT) circumvents this dilemma by inserting fake-quantization operators directly into the computational graph during training to emulate discrete low-precision inference in FP32, using the Straight-Through Estimator (STE) to guide continuous updates of full-precision master weights and successfully restoring baseline accuracy even under extreme bit constraints.

However, existing literature treats QAT merely as an auxiliary practical engineering step within broader neural network quantization surveys, or organizes methods primarily by model backbone family (CNNs, ViTs, LLMs) and nominal bit-width. This architecture-centric framing obscures the fundamental numerical, temporal, and lifecycle differences across the distinct tensors involved in computation. Model weights are persistent, stationary, and repeatedly reused across inputs; intermediate activations are dynamic, input-conditioned, and prone to extreme outliers; the KV cache accumulates errors across autoregressive decoding trajectories; and backpropagated gradients feature wide dynamic ranges and structural non-uniformity. Conflating these tensors under uniform taxonomic bins has left the field without systematic guidance for quantizer grid adaptation, surrogate-gradient stabilization, and numerical format selection (INT vs. FP).

To bridge this foundational gap, this survey introduces a target-centric taxonomy dedicated exclusively to QAT. It investigates how the unique numerical characteristics, lifetimes, and hardware constraints of different quantization targets govern algorithm design, details the mathematical underpinnings of uniform affine quantization and surrogate gradients, establishes cross-target transferability principles, and critically addresses the persistent simulation-to-deployment gap between floating-point fake-quantization and integer hardware execution. Core idea: Ground QAT analysis in the computational lifecycle and numerical properties of specific tensor targets, establishing a target-centric taxonomy, cross-target strategy transferability principles, and a standardized reporting checklist across structural diagnostics, multi-step reasoning, and four-level hardware deployment fidelity.

Method

Overall Architecture

The survey builds a cohesive conceptual and methodological pipeline spanning foundational mathematical formulations, a target-centric taxonomy, cross-target analytical synthesis, and multi-dimensional evaluation protocols. It first establishes the mechanics of Uniform Affine Quantization (UAQ), forward fake-quantization, and Straight-Through Estimators (STE). It then branches into a five-target taxonomy covering weights, activations, joint W+A, KV cache, and gradients. Next, it synthesizes cross-target failure modes, strategy transferability, and INT vs. FP numerical format allocations. Finally, it addresses the simulation-to-deployment gap by formalizing a four-level deployment hierarchy and establishing a minimal reporting checklist for reproducible research.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Theoretical Foundations<br/>Uniform Affine Quantization (UAQ) & STE Surrogates"] --> B["Target-Centric Taxonomy<br/>Weights (W) / Activations (A) / Joint (W+A) / KV Cache / Gradients (G)"]
    B --> C["Cross-Target Analytical Synthesis<br/>Transferable Numerical Strategies vs Non-Transferable Mechanics"]
    C --> D["Numerical Format Co-Design<br/>Stationary Weights (INT) vs Dynamic Activations/Gradients (FP)"]
    D --> E["Evaluation Protocols & Minimal Checklist<br/>Task Scores / Structural Diagnostics / Complex Reasoning / 4-Level Deployment"]

Key Designs

1. Target-Centric Taxonomy: Decoupling Lifecycles and Error Propagation Dynamics

Prior classification frameworks organize literature by network architectures, obscuring why algorithmic techniques succeed on certain tensors but fail on others. This taxonomy explicitly dissects five quantization targets based on their computational roles: - Weights (W): Model parameters are persistent, relatively stationary across inputs, and exhibit heterogeneous sensitivity across channels and layers. Dominant failure modes involve the distortion of salient weights, threshold grid-point oscillation (where weights flip between adjacent discrete bins across training iterations), and severe capacity collapse below 3 bits. Representative paradigms include oscillation dampening and iterative freezing, selective/salient weight preservation (e.g., PB-LLM), parameter-efficient QAT adaptation (QA-LoRA, LoftQ, L4Q, EfficientQAT), and pre-training integrated low-bit initialization (BitNet b1.58, CQPT). - Activations (A): Intermediates generated dynamically during forward propagation are token-, sample-, and layer-dependent, exhibiting severe structured outliers and non-stationary distribution drift. Primary failure modes center on clipping saturation and scale distortion. Key solutions involve learnable clipping thresholds (PACT), learnable step sizes (LSQ/LSQ+), non-uniform quantization grids (nu-LSQ), and training-inference decoupling. - Weight + Activation (W+A): Quantizing both linear operands simultaneously produces coupled distortion, mathematically expanding perturbation into \(\Delta W A + W \Delta A + \Delta W \Delta A\), which frequently triggers accumulator overflow and breaks graph-level integerization. Solutions require computational-graph redesign, including integer-compatible nonlinear and normalization operators (I-BERT, I-ViT for Softmax, GELU, LayerNorm) and heterogeneity-aware bit allocations across graph node degrees (Degree-Quant) or diffusion timesteps (BiDM, EfficientDM). - Key-Value Cache (KV): Dynamically generated in autoregressive decoding and persistently reused across future attention steps. Errors compound temporally across decoding iterations. Representative methods like LLM-QAT synthesize teacher training data to jointly simulate low-bit weights, activations, and KV states. - Gradients (G): Exist strictly within the backward pass, exhibiting high dynamic variance, extreme sensitivity to magnitude, and severe underflow/overflow vulnerabilities. Directional distortion directly destabilizes the optimization trajectory. Solutions encompass direction-sensitive clipping, distribution-adaptive channel-wise scaling (Unified INT8, GradScale, Xi et al.), and fully quantized FP4 training frameworks (Chmiel et al.).

2. Cross-Target Strategy Transferability: Differentiating Generic Numerics from Target-Specific Semantics

The survey establishes clear criteria governing whether algorithmic innovations developed for one tensor target can be transferred to another: - Transferable Strategies: Mechanisms addressing target-agnostic numerical phenomena generalize robustly across targets. Fine-grained scaling (per-channel, per-group, per-block), adaptive range control (learnable step sizes and clipping thresholds), and sensitivity-guided mixed-precision allocation are broadly effective across weights, activations, KV caches, and gradients. - Non-Transferable Strategies: Mechanisms tailored to specific tensor lifecycles, temporal reuse patterns, or graph-level execution constraints do not transfer directly. For instance, weight-oscillation suppression techniques cannot be applied to transient activations; KV-specific sequence-length compression cannot govern forward activations; gradient directional preservation is irrelevant to forward weight clipping; and integer-only operator redesign (I-BERT/I-ViT) is orthogonal to weight decomposition.

3. Numerical Format Co-Design: Deconstructing INT vs. FP Error Allocations

Addressing the persistent debate between integer and floating-point formats, the survey formalizes their trade-offs: - Error Distribution Characteristics: Uniform INT formats enforce a fixed absolute step size across a linear grid, concentrating degradation in bounded rounding and out-of-range clipping. Conversely, floating-point formats (FP8/FP4) distribute representational capacity logarithmically via exponent-mantissa splits, providing wide dynamic range with step sizes proportional to magnitude, but suffering from limited mantissa precision and relative rounding errors. - Target-Specific Format Matching: INT formats are optimally suited for bounded, stationary weight distributions, enabling maximum integer arithmetic efficiency. FP formats provide superior accommodation for highly dynamic, outlier-heavy activation and backward gradient tensors. However, QAT's explicit optimization narrows the gap between INT and FP observed in PTQ, confirming that neither format is universally superior and motivating hybrid target-specific configurations.

4. Evaluation Paradigms & Minimal Reporting Checklist: Closing the Simulation-to-Deployment Gap

To prevent academic evaluations from masking severe real-world deployment regressions, the survey formalizes an end-to-end evaluation methodology: - Structural Diagnostics: Evaluates normalized mean squared error (nMSE), cosine similarity, clipping/saturation ratios, layer-wise error curves, block-output drift, KL divergence of output logits, and accumulator overflow bounds (\(\sum_{i=1}^n |q_{w_i}| |q_{a_i}| \le n \cdot q_{w_{\max}} \cdot q_{a_{\max}}\)). - Higher-Order Reasoning Benchmarks: Demonstrates that language modeling perplexity fails to reflect multi-step reasoning capabilities. LLM evaluations must incorporate mathematical reasoning (GSM8K/MATH), multi-step knowledge reasoning (BBH/MMLU), and code generation (HumanEval/MBPP) under fixed prompt and decoding configurations. - Four Deployment Levels: Rigorously categorizes implementations into Level 1 (Fake Quantization in FP32), Level 2 (Simulated Integer Arithmetic), Level 3 (Partial Integer Deployment with floating-point non-linear fallbacks), and Level 4 (End-to-End Integer Deployment with integer-only arithmetic, non-linear operators, and accumulator-safe execution). - Minimal Reporting Checklist: Mandates the disclosure of complete quantization configurations, evaluations across at least two bit-widths with matched full-precision deltas (\(\Delta\)), at least one structural diagnostic, one measured hardware latency/throughput figure, and reasoning-intensive LLM benchmarks.

Benchmark & Evaluation Protocol

The survey conducts a comprehensive meta-analysis of empirical evaluations spanning CNNs (ImageNet), Vision Transformers (ImageNet-1K), BERT encoders (GLUE, SQuAD), LLMs (WikiText-2, C4, MMLU, GSM8K, HumanEval), diffusion models (CIFAR-10, LSUN FID), and GNNs (citation and inductive graphs). It tracks how task accuracy, structural distortion, and hardware acceleration vary across model backbones and target configurations.

Key Experimental Results

Main Results: Cross-Architecture QAT Performance and Precision Deltas

The following table synthesizes representative QAT experimental results across CNNs, ViTs, Transformer encoders, and LLMs, highlighting the degradation relative to full-precision baselines (\(\Delta\)):

Architecture / Domain Method Base Model Bits & Granularity Key Metric QAT Performance (Delta \(\Delta\))
CNN Classification Unified INT8 (Zhu et al.) ResNet-50 W8A8G8 (Layer) ImageNet Top-1 (%) 76.34 (-0.26)
CNN Classification LLSQ ResNet-18 W4A4 (Channel/Layer) ImageNet Top-1 (%) 69.84 (+0.08)
CNN Classification LLSQ ResNet-18 W3A3 (Channel/Layer) ImageNet Top-1 (%) 68.08 (-1.68)
CNN Classification LLSQ MobileNetV2 W4A4 (Channel/Layer) ImageNet Top-1 (%) 67.37 (-4.43)
CNN Classification LSQ+ ResNet-18 W2A2 (Layer) ImageNet Top-1 (%) 66.80 (-3.30)
Vision Transformer I-ViT (End-to-End Int) DeiT-S W8A8 (Integer-only) ImageNet Top-1 (%) 80.12 (+0.27)
Vision Transformer I-ViT (End-to-End Int) ViT-B W8A8 (Integer-only) ImageNet Top-1 (%) 84.76 (+0.23)
Vision Transformer Q-ViT DeiT-S W4A4 ImageNet Top-1 (%) 80.90 (+1.00)
Vision Transformer Q-ViT DeiT-S W2A2 ImageNet Top-1 (%) 72.10 (-7.80)
BERT / Encoder TernaryBERT BERT-base W2E2A8 (Layer/Row) MNLI-m / SQuAD F1 83.3 (-1.2) / 87.4 (-1.3)
BERT / Encoder BinaryBERT BERT-base W1E1A8 (Layer/Tensor) MNLI-m / SQuAD F1 84.2 (-0.4) / 88.3 (-0.2)
BERT / Encoder I-BERT (End-to-End Int) RoBERTa-Base W8A8 (Integer-only) MNLI-m (%) 87.5 (-0.3)
Large Language Model LR-QAT LLaMA-2-7B W4 g128 WikiText-2 PPL / Zero-shot Avg 5.59 (+0.12) / 69.88 (-0.59)
Large Language Model EfficientQAT LLaMA-2-7B W3 g128 WikiText-2 PPL / Zero-shot Avg 5.81 (+0.34) / 64.02 (-0.84)
Large Language Model EfficientQAT LLaMA-2-7B W2 g64 WikiText-2 PPL / Zero-shot Avg 6.86 (+1.39) / 60.14 (-4.72)
Large Language Model LLM-QAT LLaMA-1-7B W4A8KV4 (Token) WikiText-2 PPL / Zero-shot Avg 11.6 (+1.20) / 63.00 (-3.20)

Ablation Study: Instruction-Tuned LLMs on MMLU and Commonsense Reasoning

The following table analyzes how aggressive low-bit configurations and Q-PEFT strategies affect complex reasoning and zero-shot commonsense benchmarks on LLaMA-1-7B (data compiled from Table 17 and related experimental synthesis in the survey):

Method Target Bits & Settings 5-shot MMLU (%) Delta from Base \(\Delta\) Commonsense Avg (%) Delta from Base \(\Delta\)
QA-LoRA W4 (g32) 39.4 +4.8 61.8 +3.5
QA-LoRA W3 (g32) 37.4 +2.8 60.1 +1.8
QA-LoRA W2 (g32) 27.5 -7.1 53.7 -4.6
DL-QAT W4 39.9 +5.3 63.4 +5.1
DL-QAT W3 33.9 -0.7 61.1 +2.8
L4Q W4 (g128) 35.7 +0.6 62.7 +1.0
L4Q W3 (g128) 31.8 -3.3 61.2 +0.5
LoftQ (LLaMA-2-7B) W4 NF4 5.16 (PPL) +0.04 53.7 (GSM8K) -4.6
LoftQ (LLaMA-2-7B) W2 NF2 7.69 (PPL) +2.57 - -

Key Findings

  • INT8 QAT is essentially lossless, but real deployment remains a bottleneck: Across CNNs and Transformer encoders, INT8 QAT achieves near-lossless compression (ResNet-50 drops only -0.26%, while I-ViT achieves +0.27% over FP32 on DeiT-S). However, real deployment fidelity remains challenged by accumulator overflow and compiler support for integer nonlinearities.
  • W4 QAT is practical, but failure modes are architecture-dependent: While residual CNNs sustain W4A4 without accuracy degradation, compact models with depthwise convolutions (MobileNetV2) collapse by -4.43% under LLSQ due to severe sensitivity to activation clipping. ViTs degrade primarily through self-attention and GELU distortions. In LLMs, weight-only W4 is largely solved, but joint W4A8KV4 (LLM-QAT) incurs significant drops (-3.20% zero-shot accuracy).
  • Sub-3-bit and binary QAT represent a distinct failure regime: Degradation below 3 bits is non-linear and steep across architectures. Q-ViT drops by -7.80% on DeiT-S at W2A2; EfficientQAT W2g64 incurs a -4.72% drop in zero-shot ability; and QA-LoRA drops -7.1% on MMLU at W2. The primary failure causes shift from basic truncation to severe STE gradient mismatch, threshold oscillation, and salient-weight erasure.
  • Perplexity is deceptively insufficient for LLM QAT: Small variations in WikiText-2 perplexity often mask catastrophic collapses on multi-step reasoning benchmarks like GSM8K and HumanEval. Furthermore, Q-PEFT methods can inflate MMLU or zero-shot scores through downstream adaptation, masking underlying quantization damage.
  • Nominal compression does not imply realized hardware acceleration: End-to-end integer deployment (Level 4) is required to unlock real-world speedups. I-ViT achieves 3.72-4.11x wall-clock inference speedups over FP32 by redesigning the entire computational graph (Softmax, GELU, LayerNorm) for integer arithmetic, whereas pipelines retaining floating-point nonlinearities suffer severe latency penalties due to frequent cast operations.

Highlights & Insights

  • Target-Centric Conceptual Taxonomy: Replaces conventional architecture- or bit-width-centric taxonomies with an analytical framework rooted in tensor lifecycles, dynamic ranges, and error accumulation patterns, clarifying the exact root causes of quantization errors.
  • Bifurcated Transferability Framework: Formulates a rigorous boundary between generic numerical mechanisms (fine-grained scaling, learnable step sizes) that transfer seamlessly across targets, and lifecycle-dependent mechanisms (oscillation suppression, KV channel alignment, integer non-linear graph redesign) that cannot be directly transferred.
  • Four-Level Hardware Deployment Hierarchy: Formulates four unambiguous levels of deployment fidelity (from FP32 fake-quantization to end-to-end pure integer execution), directly confronting the systemic failure of academic literature to validate real-world hardware compatibility and accumulator safety.

Limitations & Future Work

  • Author-Acknowledged Limitations: The survey focuses predominantly on QAT algorithmic mechanisms and evaluation methodology, giving less coverage to low-level compiler optimizations and custom accelerator kernel designs. Direct QAT literature for emerging targets like long-context KV caches remains sparse, and heterogeneous experimental setups across papers preclude a unified absolute leaderboard.
  • Extended Limitations: The survey does not fully address the emerging challenges of Mixture-of-Experts (MoE) dynamic routing quantization, where activation sparsity and expert load balancing interact unfavorably with QAT, nor does it explore the emerging microscaling (MXFP4/MXFP6) format standards in multi-node training clusters.
  • Future Research Directions: (1) Designing "quantization-native" architectures that naturally eliminate activation outliers and bypass STE gradient mismatch; (2) Developing ultra-efficient, single-GPU QAT techniques that eliminate the need for expensive full-precision master weights and heavy teacher distillation; (3) Establishing automated format-target co-design that coordinates mixed INT/FP/MX allocations across W, A, KV, and G.
  • vs Nagel et al. (2021): Prior work focuses on general deployment pipelines where QAT is merely an operational component; this paper centers entirely on QAT, deeply exploring its optimization stability, surrogate-gradient dynamics, and cross-target transferability.
  • vs Wei et al. (2024): Prior work offers a broad macro-level overview of quantization; this survey provides granular mathematical formalizations, detailed piecewise gradient derivations, and target-specific error analysis.
  • vs Yang et al. (2025): Prior work classifies literature along architecture lines (CNNs, ViTs, LLMs); this survey's target-centric framework cuts across model architectures to expose shared tensor-level properties and failure modes.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ [Pioneers the target-centric taxonomy, breaking free from traditional model-centric and bit-width categorization]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Synthesizes dozens of representative works across 7 model families with comprehensive meta-analytical rigor]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Exemplary mathematical derivations, fluent prose, and lucid visual taxonomies]
  • Value: ⭐⭐⭐⭐⭐ [Crucial benchmark standardization and reporting checklist that directly bridges academic simulation and hardware execution]