InfoQuant: Shaping Activation Distributions for Low-Bit Weight-Activation Quantization in LLMs¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/LLIKKE/InfoQuant
Area: Model Compression
Keywords: LLM Quantization, Post-Training Quantization (PTQ), Weight-Activation Quantization, Orthogonal Transformation, Distribution Shaping
TL;DR¶
InfoQuant recasts activation transformation as quantizer-facing distribution design from an information-theoretic perspective, combining train-free Peak Suppression Orthogonal Transformation (PSOT), adaptive outlier-token selection (ASOT), and learnable activation clipping (LAC) to preserve 97% of floating-point accuracy under W4A4KV4.
Background & Motivation¶
As large language models (LLMs) scale up in parameter volume and context length, post-training quantization (PTQ) has become essential for reducing inference latency and memory footprints on commodity hardware. While weight-only quantization typically results in minor accuracy loss due to the flat and uniform nature of weight parameters, low-bit activation quantization remains the dominant performance bottleneck. Activations across Transformer layers systematically exhibit extreme, dominant outlier coordinates that drastically inflate the dynamic numerical range. Under aggressive 4-bit or 3-bit regimes with only 16 discrete bins, standard round-to-nearest (RTN) uniform quantizers are forced to stretch their intervals over wide ranges, causing dense, informative activation values centered around the mean to collapse into the same few discrete centroids.
Existing transformation-based PTQ methods attempt to alleviate this mismatch by pre-transforming activations before quantization. SmoothQuant migrates dynamic difficulty from activations to weights through per-channel diagonal scaling, QuaRot and SpinQuant exploit orthogonal rotations (fixed Hadamard or task-loss optimized matrices) to redistribute activation energy across channels, and FlatQuant adopts fine-grained Kronecker-based affine transformations. Despite their differing formulations, these methods are primarily driven by heuristic objectives: suppressing conspicuous outlier peaks, equalizing channel energy, or minimizing local mean squared error (MSE). However, they rarely define what transformed activation distribution is actually quantization-friendly for a low-bit uniform quantizer, frequently producing activations that look smoother while still collapsing distributional resolution.
This paper addresses this fundamental gap by reframing activation transformation as quantizer-facing distribution design. By analyzing the smoothed Kullback-Leibler (KL) divergence and normalized clamped quantization error bounds, the authors theoretically demonstrate that an easily quantizable distribution must jointly satisfy two complementary properties: a smaller numerical range and sufficient dispersion within that range. Core idea: adaptively reshape activation geometry into quantization-friendly distributions via train-free Peak Suppression Orthogonal Transformation (PSOT) guided by an information-theoretic bound, reinforced by adaptive outlier-token selection (ASOT) and learnable activation clipping (LAC) to spread activation mass evenly across discrete low-bit centroids.
Method¶
Overall Architecture¶
InfoQuant is a train-free PTQ framework designed for low-bit weight, activation, and KV-cache quantization in LLMs. Instead of relying on end-to-end backpropagation or heuristic peak smoothing, it directly optimizes the discretizability of the activation streams. The framework proceeds through three progressive stages: 1. Peak Suppression Orthogonal Transformation (PSOT): Initializes block-diagonal orthogonal matrices from Hadamard transforms and optimizes them via Cayley SGD on calibration activations, penalizing dominant coordinates to suppress peak magnitude while diffusing energy into orthogonal dimensions; 2. Adaptive Outlier-Token Selection (ASOT): Evaluates the cross-sample positional inconsistency of outlier tokens to filter out spurious outliers tied to fixed prompt formatting, adaptively emphasizing semantically driven outlier tokens during rotation optimization; 3. Learnable Activation Clipping (LAC): Introduces bounded dual scaling parameters on the reshaped activation bounds, fine-tuning the quantization step size via local block-level reconstruction before performing standard GPTQ weight compensation.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Calibration Activation Stream<br/>Bell-shaped with dominant outlier coordinates"] --> B["Peak Suppression Orthogonal Transformation<br/>PSOT: Block-diagonal rotation shaping distribution"]
B --> C["Adaptive Outlier-Token Selection<br/>ASOT: Reweighting based on positional inconsistency"]
C --> D["Learnable Activation Clipping<br/>LAC: Block-level reconstruction of step size"]
D --> E["W4A4/W4A4KV4 Low-Bit Deployment<br/>Compatible with lightweight inference kernels"]
Key Designs¶
1. Peak Suppression Orthogonal Transformation (PSOT): Joint Optimization of Narrow Range and High Normalized Dispersion
To address the inability of conventional MSE-based smoothing to capture discrete quantization resolution, the paper derives a normalized clamped quantization error bound using a smoothed continuous relaxation with Laplace kernels \(\delta(x; \theta)\):
where \(\lambda = s/\sigma\) is the normalized step size, \(\kappa = c/\sigma = 1/b_n\) is the normalized clipping ratio, and \(b_n = \sigma/c\) represents the max-normalized dispersion. For bell-shaped distributions (such as Gaussian and Laplace), the bound \(F_{P,B}(\kappa)\) decreases monotonically as \(\kappa\) decreases in the tail-controlled regime, which directly translates to shrinking the numerical range \(c\) while increasing dispersion \(b_n\). PSOT realizes this objective by parameterizing a block-diagonal orthogonal rotation matrix \(R\) (\(R^\top R = I\)) initialized from Hadamard matrices, removing the coordinate mean via centering projector \(P_\perp = I - \frac{1}{d}\mathbf{1}\mathbf{1}^\top\), and optimizing a soft peak-suppression loss:
where \(\pi_T(y) = \text{softmax}(|y|/T)\) denotes temperature-controlled softmax weights. Because orthogonal transformations strictly preserve the Euclidean norm (\(\|tA\|_2 = \|t\|_2\)), penalizing the maximum coordinates forces the preserved token energy to spread across remaining dimensions. This suppresses the peak coordinate \(\|t'\|_\infty\) and increases the max-normalized dispersion \(b_n(t') = \frac{1}{\sqrt{d}} \frac{\|t'\|_2}{\|t'\|_\infty}\), directly aligning the continuous activation density with the discrete capacity of low-bit uniform quantizers.
2. Adaptive Outlier-Token Selection (ASOT): Disentangling True Semantic Peaks from Positional Formatting Artifacts
Optimizing over all calibration tokens equally dilutes gradient signals on genuine outlier coordinates, while naive thresholding risks overfitting to fixed prompt-formatting positions. ASOT introduces an automated statistical selection mechanism. Given a threshold coefficient \(k\), outlier token indices for calibration sample \(r\) are identified by \(T^{(r)}(k) = \{i \mid o_i^{(r)} > k\}\), where \(o_i^{(r)} = \|(x_i^{(r)} - \mu)/\sigma\|_\infty\). To identify whether outlier positions vary across distinct sequences rather than being tied to static positional artifacts, ASOT measures cross-sample positional inconsistency:
As \(k\) increases, \(\eta_m(k)\) exhibits a characteristic rise before plateauing when stable sample-dependent outliers are isolated. ASOT selects the optimal threshold \(k^\star\) via:
Selected tokens in \(T^{(r)}(k^\star)\) receive an amplified weight \(\gamma > 1\) (e.g., \(\gamma=30\)), while normal tokens retain unit weight. This concentrates the rotational optimization budget specifically on coordinates that govern the clipping boundary.
3. Learnable Activation Clipping (LAC): Refining Quantization Boundaries via Local Block Reconstruction
After PSOT reshapes the activation distribution into a compact shape, residual variance across distinct channels can still degrade precision if naive min-max scaling is used. LAC optimizes two bounded clipping scaling parameters \(\alpha, \beta \in [0.5, 1.0]\) to dynamically calibrate the quantization step size:
These parameters are optimized by minimizing the output Frobenius norm error between the full-precision Transformer block \(\mathcal{F}_l\) and the quantized block \(\widehat{\mathcal{F}}_l\):
By fine-tuning only the boundary scaling parameters through lightweight block-level reconstruction without modifying model weights, LAC strikes an optimal balance between in-range rounding error and clipping tail truncation error.
Loss & Training¶
InfoQuant is entirely train-free, requiring no full-model backpropagation or global task loss. During the PSOT phase, block-diagonal orthogonal matrices are optimized via Cayley SGD on the Stiefel manifold over 15 epochs with an initial learning rate of 2.0 (linearly decayed) and temperature \(T=2\), using 128 sequences of length 2048 from WikiText-2. ASOT sets sample count \(m=10\), stabilization tolerance \(\delta=0.02\), sparsity ratio \(\tau=0.4\), and reweighting multiplier \(\gamma=30\). LAC optimizes clipping bounds with AdamW over 5 epochs with cosine annealing (initial lr 0.05, initial clipping ratio 0.95). Finally, weights are quantized using asymmetric per-channel GPTQ (group size 128 for KV cache). Block-diagonal matrix partitioning (2 blocks for 7B/13B, 4 blocks for 70B) allows the entire 70B quantization process to complete in minutes on an NVIDIA RTX 4090 within 24GB of GPU memory.
Key Experimental Results¶
Main Results¶
The authors evaluate InfoQuant across LLaMA-2 (7B, 13B, 70B), LLaMA-3 (8B, 70B), and Qwen2.5 (14B, 32B), reporting WikiText2 perplexity (PPL) and average zero-shot accuracy across nine benchmarks from lm-evaluation-harness (ARC-Easy, ARC-Challenge, BoolQ, HellaSwag, LAMBADA, OpenBookQA, PIQA, SIQA, WinoGrande). Representative results under the W4A4KV4 and W4A4 configurations are summarized below:
| Model | Bit-width (W-A-KV) | Method | WikiText2 PPL (↓) | 9-Task Zero-Shot Avg (↑) | FP16 Retention (%) |
|---|---|---|---|---|---|
| LLaMA-3 8B | 16-16-16 | FloatingPoint | 6.14 | 68.09 | 100.0% |
| 4-4-4 | QuaRot | 8.18 | 61.38 | 90.1% | |
| 4-4-4 | SpinQuant | 7.35 | 64.10 | 94.1% | |
| 4-4-4 | Kurtail | 7.20 | - | - | |
| 4-4-4 | BASE-Q | 7.17 | 65.39 | 96.0% | |
| 4-4-4 | INFOQUANT | 7.16 | 65.57 | 96.3% | |
| LLaMA-3 70B | 16-16-16 | FloatingPoint | 2.86 | 73.81 | 100.0% |
| 4-4-4 | QuaRot | 6.60 | 65.33 | 88.5% | |
| 4-4-4 | SpinQuant | 6.24 | 66.31 | 89.8% | |
| 4-4-4 | Kurtail | 7.20 | - | - | |
| 4-4-4 | BASE-Q | OOM | OOM | - | |
| 4-4-4 | INFOQUANT | 5.39 | 70.21 | 95.1% | |
| LLaMA-2 7B | 16-16-16 | FloatingPoint | 5.47 | 65.21 | 100.0% |
| 4-4-4 | QuaRot | 6.11 | 61.48 | 94.3% | |
| 4-4-4 | SpinQuant | 5.96 | 62.01 | 95.1% | |
| 4-4-4 | BASE-Q | 5.85 | 62.50 | 95.8% | |
| 4-4-4 | INFOQUANT | 5.89 | 63.16 | 96.9% | |
| LLaMA-2 13B | 16-16-16 | FloatingPoint | 4.88 | 67.61 | 100.0% |
| 4-4-4 | QuaRot | 5.39 | 65.16 | 96.4% | |
| 4-4-4 | SpinQuant | 5.74 | 64.13 | 94.9% | |
| 4-4-4 | BASE-Q | 5.21 | 65.48 | 96.9% | |
| 4-4-4 | INFOQUANT | 5.18 | 66.33 | 98.1% | |
| LLaMA-2 70B | 16-16-16 | FloatingPoint | 3.32 | 71.59 | 100.0% |
| 4-4-4 | QuaRot | 3.80 | 70.30 | 98.2% | |
| 4-4-4 | SpinQuant | 3.61 | 70.57 | 98.6% | |
| 4-4-4 | BASE-Q | OOM | OOM | - | |
| 4-4-4 | INFOQUANT | 3.64 | 70.35 | 98.3% | |
| Qwen2.5 14B | 4-4-4 | QuaRot | 6.77 | 67.23 | 94.8% |
| 4-4-4 | SpinQuant | 6.55 | 67.29 | 94.8% | |
| 4-4-4 | INFOQUANT | 6.30 | 67.65 | 95.3% | |
| Qwen2.5 32B | 4-4-4 | QuaRot | 6.04 | 68.14 | 95.8% |
| 4-4-4 | SpinQuant | 5.88 | 68.51 | 96.3% | |
| 4-4-4 | INFOQUANT | 5.69 | 69.90 | 98.3% |
Ablation Study¶
The module-wise ablation in Table 4 isolates the progressive perplexity improvements under W4A4KV4:
| Configuration | Hadamard Baseline | PSOT Rotation | GPTQ Weight Rec. | ASOT Token Selection | LAC Boundary Clipping | LLaMA-3 8B PPL (↓) | LLaMA-2 7B PPL (↓) |
|---|---|---|---|---|---|---|---|
| Fixed Hadamard Baseline | ✓ | - | - | - | - | 10.90 | 8.99 |
| + PSOT Transformation | - | ✓ | - | - | - | 8.84 | 7.03 |
| + GPTQ Reconstruction | - | ✓ | ✓ | - | - | 7.53 | 6.01 |
| + ASOT Reweighting | - | ✓ | ✓ | ✓ | - | 7.42 | 5.97 |
| Full INFOQUANT | - | ✓ | ✓ | ✓ | ✓ | 7.16 | 5.89 |
Key Findings¶
- PSOT is the dominant driver of quantization gains: Swapping the fixed Hadamard transform with the learned PSOT rotation drives WikiText2 perplexity down from 10.90 to 8.84 on LLaMA-3 8B and from 8.99 to 7.03 on LLaMA-2 7B, accounting for over 50% of the total improvement.
- Orthogonal compatibility with weight reconstruction: Adding GPTQ after activation distribution reshaping yields an additional drop of ~1.0 PPL point, verifying that quantizer-facing activation transformation works synergistically with second-order weight compensation.
- High calibration and initialization robustness: PSOT converges to identical performance whether initialized from Hadamard matrices or random QR orthogonal matrices (both yielding 7.18 on LLaMA-3 8B and 5.18 on LLaMA-2 13B). Cross-calibration evaluations across C4 and WikiText2 across 128 to 512 samples exhibit standard deviations below 0.17.
- Superior hardware efficiency at scale: On LLaMA-3 70B, baseline methods like BASE-Q encounter out-of-memory (OOM) failures, whereas InfoQuant remains strictly under 24GB GPU memory during calibration, achieving 2.46× prefill speedup and 2.91× memory savings during deployment on an RTX 4090.
Highlights & Insights¶
- Theoretical grounding for activation PTQ: Moves beyond empirical peak smoothing by deriving closed-form bounds relating low-bit quantization error to numerical range compression and normalized dispersion \(b_n\).
- Hardware-friendly block-diagonal rotation: Demonstrates that full global rotation is not strictly necessary; block-diagonal orthogonal matrices significantly reduce residual matrix multiplication overhead while maintaining the required geometric flexibility.
- Robust outlier filtering via positional inconsistency: The \(\eta_m(k)\) formulation provides a principled metric to filter out pseudo-outliers caused by static prompt layouts, ensuring rotation matrices generalize across diverse input distributions.
Limitations & Future Work¶
- Reliance on bell-shaped distribution assumptions: The theoretical closed-form bounds assume Gaussian or Laplace densities; complex heavy-tailed or multi-modal activation distributions (e.g., in Mixture-of-Experts architectures) may require broader distribution models.
- Independent layer-wise optimization: Rotations are solved independently per layer, leaving potential cross-layer error compounding unconstrained across deep Transformers.
- Extreme sub-4-bit regimes: While near-lossless at W4A4, scaling down to ultra-low W3A3 or W2A2 bit-widths remains bounded by the representational limit of uniform quantizers.
Related Work & Insights¶
- vs QuaRot & SpinQuant: QuaRot relies on fixed, data-agnostic Hadamard matrices. SpinQuant optimizes rotations end-to-end on task losses via full-model backpropagation, requiring massive compute. InfoQuant provides a train-free alternative directly optimizing distribution shape with minimal compute and memory.
- vs FlatQuant: FlatQuant utilizes dynamic affine transformations with high inference FLOPs and custom kernels. InfoQuant maintains lightweight rotation folding into weights, introducing minimal inference overhead.
- vs OSTQuant: OSTQuant relies on end-to-end distillation from full-precision teachers. InfoQuant demonstrates that unsupervised distribution shaping matches or outperforms distillation-based methods on multiple model scales without expensive supervision.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ Reframes activation transformation from an information-theoretic distribution design perspective, providing rigorous theoretical justification.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across multiple model families (7B to 70B), 9 zero-shot benchmarks, inference profiling, and ablation studies.
- Writing Quality: ⭐⭐⭐⭐⭐ Well-structured, mathematically rigorous with closed-form proofs in the appendix and informative figures.
- Value: ⭐⭐⭐⭐⭐ Enables high-fidelity W4A4KV4 deployment of 70B models on consumer hardware (24GB GPU), offering high industrial and practical utility.