CoQuant: Joint Weight-Activation Subspace Projection for Mixed-Precision LLMs¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Zachary5895/CoQuant
Area: Model Compression
Keywords: Mixed-Precision Quantization, Post-Training Quantization, Subspace Projection, Joint Covariance Optimization, KV Cache Quantization
TL;DR¶
Addressing the flaw in prior subspace-based mixed-precision quantization that relies solely on activation statistics while overlooking weight noise, CoQuant derives a first-order joint error expectation model that yields a closed-form weighted PCA solution balancing activation and weight covariances, substantially reducing layer-wise perturbation and achieving state-of-the-art perplexity and zero-shot reasoning accuracy on Llama-3.2 and Qwen2.5.
Background & Motivation¶
The broad deployment of Large Language Models (LLMs) across critical applications is bottlenecked by massive GPU memory capacity and memory bandwidth constraints. Post-training quantization (PTQ) has emerged as an essential approach because it bypasses costly retraining while maintaining low deployment overhead. While early PTQ techniques like GPTQ and AWQ focused heavily on weight-only compression, achieving end-to-end hardware acceleration requires simultaneously quantizing activations. However, the presence of systemic extreme activation outliers in modern LLMs causes direct low-bit uniform quantization to suffer catastrophic performance degradation. Although rotation-based methods such as QuaRot and SpinQuant apply equivalent orthogonal transforms (e.g., Hadamard rotations) to disperse outliers across dimensions, uniform quantization at ultra-low bit-widths (e.g., 4-bit) still exhibits an undeniable accuracy gap relative to full-precision baselines.
To recover lost accuracy under tight memory budgets, mixed-precision quantization (MPQ) has become a prominent research direction. Its core principle is to isolate a tiny fraction of critical, highly sensitive elements in high precision (e.g., 8-bit) while quantizing the vast majority of channels into low precision (e.g., 4-bit). Recently, ResQ advanced this paradigm to a subspace perspective, using Principal Component Analysis (PCA) to identify a high-variance, low-rank subspace to be preserved at higher precision. Nevertheless, existing subspace-based methods suffer from a fundamental theoretical blind spot: their mathematical formulations rely exclusively on upper-bounding activation quantization errors, meaning the selected high-precision subspace is determined entirely by activation covariance statistics.
The core tension lies in the fundamental nature of matrix multiplications \(Y = XW\) in linear layers: output perturbations are intrinsically driven by the joint interaction of both activation and weight quantization errors. The singular value distribution and principal directions of the weight matrix inherently possess strong directional sensitivity. By ignoring weight statistics and selecting the projection basis solely from input activations, previous methods fail to capture the holistic error landscape, yielding suboptimal subspace partitioning. Core idea: Starting from the expected Frobenius norm of linear layer output error under mixed precision, CoQuant jointly models both activation and weight perturbations under an isotropic noise formulation to derive a closed-form weighted PCA objective that optimally fuses activation and weight covariances.
Method¶
Overall Architecture¶
CoQuant aims to construct an orthogonal transformation matrix \(U = [U_l, U_h]\) that partitions the feature space into an \(r\)-dimensional high-precision subspace and a complementary \((d-r)\)-dimensional low-precision subspace, minimizing the expected output error of \(\hat{Y} = Q_l(X U_l) Q_l(U_l^\top W) + Q_h(X U_h) Q_h(U_h^\top W)\). For each functional block sharing identical input activations, CoQuant accumulates input activation covariance and fused weight covariance, builds a weighted composite covariance matrix based on bit-width error coefficients, solves for the optimal projection basis via eigendecomposition, and symmetrically extends this joint projection to autoregressive KV cache compression.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Calibration Inputs and Model Weights"] --> B["Joint Error Modeling and Closed-Form Weighted PCA<br/>AQNM expected error, derive M = λ_X Σ_X + λ_W Σ_W"]
B --> C["Fused Projections for Shared-Input Linear Layers<br/>Fuse Q/K/V & Gate/Up weight covariances, eigendecomposition for P_h"]
C --> D["Generalized Bidirectional KV Cache Co-Quantization<br/>V cache paired with W_O, K cache adaptively aligned with Q covariance"]
D --> E["Mixed-Precision Weights, Activations & KV Cache<br/>1/8 channels in 8-bit, 7/8 in 4-bit (avg. 4.5-bit)"]
Key Designs¶
1. Joint Error Modeling and Closed-Form Weighted PCA: Eliminating Loose Upper Bounds via Isotropic White Noise Modeling
To overcome the suboptimality of selecting subspaces purely via activation PCA, CoQuant rigorously formulates the total expected output perturbation. Given the decomposed quantized components \(\hat{X}_k = X_k + E_{X_k}\) and \(\hat{W}_k = W_k + E_{W_k}\) for \(k \in \{l, h\}\), internal random orthogonal rotations \(R_h\) Gaussianize the representations and disperse outliers. This structural property validates the Additive Quantization Noise Model (AQNM) where noise is zero-mean and uncorrelated with signals. Neglecting the second-order perturbation \(E_{X_k} E_{W_k}\), the expected squared Frobenius norm simplifies to cross-interaction terms: \(\mathbb{E}[\|\hat{Y}-Y\|_F^2] \approx \sum_{k} (\mathbb{E}[\|E_{X_k}W_k\|_F^2] + \mathbb{E}[\|X_k E_{W_k}\|_F^2])\).
Rather than relying on loose sub-multiplicative norm bounds, CoQuant introduces an isotropic white noise assumption, modeling expected noise energy as directly proportional to signal energy: \(\mathbb{E}[\|E_{X_k}\|_F^2] \approx \alpha_k^2 \|X_k\|_F^2\), where relative error coefficients are governed strictly by bit-width \(N_k\): \(\alpha_k^2 = \beta_k^2 = (2^{N_k-1}-1)^{-2}\). Applying the matrix trace identity, the cross-error evaluates exactly to \(\mathbb{E}[\|E_{X_k}W_k\|_F^2] \approx \frac{\alpha_k^2}{d_k} \|X_k\|_F^2 \|W_k\|_F^2\). Leveraging energy conservation across complementary orthogonal subspaces \(\|X_l\|_F^2 = \|X\|_F^2 - \|X_h\|_F^2\) and discarding the weaker \(O(r^2)\) cross-penalty term, the joint error minimization transforms into a clean weighted PCA surrogate objective: $\(\max_{P_h^\top P_h = I} \mathrm{Tr}\left( P_h^\top (\lambda_X \Sigma_X + \lambda_W \Sigma_W) P_h \right)\)$ where \(\Sigma_X = X^\top X\) and \(\Sigma_W = W W^\top\) denote uncentered second-moment matrices, with weights \(\lambda_X = \gamma_l \|W\|_F^2\), \(\lambda_W = \gamma_l \|X\|_F^2\), and combined error coefficient \(\gamma_k = (\alpha_k^2 + \beta_k^2)/d_k\). By the Rayleigh-Ritz theorem, the optimal high-precision basis \(P_h\) consists of the top-\(r\) eigenvectors of \(M = \lambda_X \Sigma_X + \lambda_W \Sigma_W\), providing a mathematically rigorous balance between activation and weight energy distributions.
2. Fused Projections for Shared-Input Linear Layers: Preserving Kernel Fusion and Minimizing Overhead
Applying individual subspace projections to every separate linear layer would disrupt runtime operator fusion and introduce substantial memory and compute overhead during calibration and inference. To ensure seamless deployment, CoQuant proposes a fused projection strategy for layers sharing identical input activations.
In modern multi-head and grouped-query attention blocks, the query (\(W_Q\)), key (\(W_K\)), and value (\(W_V\)) projection layers all consume the identical input \(X_{\text{attn}}\). CoQuant constructs a unified attention weight covariance by linearly superimposing their outer-product matrices: $\(\Sigma_W^{\text{attn}} = W_Q W_Q^\top + W_K W_K^\top + W_V W_V^\top\)$ Similarly, in SwiGLU MLP blocks, \(W_{\text{gate}}\) and \(W_{\text{up}}\) share the input activation \(X_{\text{mlp}}\), yielding a fused covariance \(\Sigma_W^{\text{mlp}} = W_{\text{gate}} W_{\text{gate}}^\top + W_{\text{up}} W_{\text{up}}^\top\). By substituting these fused covariances into the joint objective, a single eigendecomposition yields the unified projection basis \(P_{\text{attn}}\) or \(P_{\text{mlp}}\). This design preserves fused GEMM execution while globally minimizing the collective quantization error across parallel linear branches.
3. Generalized Bidirectional KV Cache Co-Quantization: Adaptively Aligning Dynamic Decoding Dynamics
During autoregressive generation, memory bandwidth saturation from expanding KV caches constitutes the primary inference bottleneck. CoQuant extends its matrix multiplication error formulation to dynamic caches by treating any interacting pair of tensors as generalized "activations" and "weights". For the Value cache, stored tokens \(V\) are subsequently multiplied by the static output projection matrix \(W_O\). CoQuant sets \(\Sigma_X\) as the empirical covariance of \(V\) and \(\Sigma_W\) as the static covariance of \(W_O\) (computed per KV head to align with Grouped-Query Attention), precisely safeguarding the dimensions most sensitive to output projection.
For the Key cache, the physical interaction is the attention dot-product \(Q K^\top\) following Rotary Position Embedding (RoPE). In this operation, Keys act as the operands while Query states \(Q\) function as dynamic transformation "weights". CoQuant treats \(Q\) as dynamic weights, setting \(\Sigma_X\) to the empirical covariance of \(K\) and \(\Sigma_W\) to the empirical covariance of \(Q\). This formulation adaptively steers the Key high-precision subspace to protect the structural manifolds most critical to the dynamic query distributions, preserving fine-grained attention accuracy even under aggressive quantization.
Loss & Training¶
CoQuant is a strictly training-free PTQ algorithm that requires no gradient-based fine-tuning. Per-token asymmetric quantization is applied to activations, per-channel symmetric quantization to weights, and per-head asymmetric quantization to the KV cache, with GPTQ utilized for second-order weight rounding. The default setup allocates \(r = d/8\) of channels to 8-bit high precision and the remaining \(7/8\) to 4-bit low precision, yielding an average bit-width of 4.5-bit. Calibration statistics are extracted from 512 randomly sampled WikiText sequences (maximum length 2048) on a single NVIDIA A100 GPU, completing covariance computation and eigendecomposition in minutes.
Key Experimental Results¶
Main Results¶
WikiText perplexity (PPL ↓) and average zero-shot common-sense reasoning accuracy (Avg 0-shot ↑ across ARC-c, ARC-e, BoolQ, PIQA, SIQA, and WinoGrande) on Llama-3.2 (1B, 3B) and Qwen2.5 (0.5B, 1.5B, 7B, 14B) models are detailed below. For mixed-precision approaches (QUIK, ResQ, and CoQuant), 1/8 of the channels are preserved in 8-bit while the rest are in 4-bit (average 4.5-bit W/A/KV).
| Model Family | Method | W/A/KV Bits | Wiki PPL (↓) | Avg 0-shot (↑) |
|---|---|---|---|---|
| Llama-3.2-1B | FP16 | 16/16/16 | 9.75 | 56.48% |
| RTN | 4/4/4 | 329.65 | 40.77% | |
| GPTQ | 4/4/4 | 213.48 | 41.86% | |
| QuaRot | 4/4/4 | 14.39 | 51.03% | |
| QUIK | 4.5/4.5/4.5 | 16.55 | 49.92% | |
| ResQ | 4.5/4.5/4.5 | 12.03 | 52.30% | |
| CoQuant (Ours) | 4.5/4.5/4.5 | 11.60 | 52.75% | |
| Llama-3.2-3B | FP16 | 16/16/16 | 7.81 | 64.18% |
| QuaRot | 4/4/4 | 10.06 | 56.13% | |
| QUIK | 4.5/4.5/4.5 | 10.78 | 55.56% | |
| ResQ | 4.5/4.5/4.5 | 9.18 | 60.03% | |
| CoQuant (Ours) | 4.5/4.5/4.5 | 8.93 | 60.23% | |
| Qwen2.5-0.5B | FP16 | 16/16/16 | 13.07 | 54.02% |
| QuaRot | 4/4/4 | 204.10 | 41.50% | |
| QUIK | 4.5/4.5/4.5 | 301.06 | 39.51% | |
| ResQ | 4.5/4.5/4.5 | 18.19 | 48.24% | |
| CoQuant (Ours) | 4.5/4.5/4.5 | 17.76 | 49.12% | |
| Qwen2.5-1.5B | FP16 | 16/16/16 | 9.26 | 62.97% |
| QuaRot | 4/4/4 | 6169.32 | 42.03% | |
| ResQ | 4.5/4.5/4.5 | 11.75 | 55.66% | |
| CoQuant (Ours) | 4.5/4.5/4.5 | 11.67 | 57.59% | |
| Qwen2.5-7B | FP16 | 16/16/16 | 6.85 | 70.13% |
| QuaRot | 4/4/4 | 3736.39 | 40.66% | |
| ResQ | 4.5/4.5/4.5 | 8.95 | 66.46% | |
| CoQuant (Ours) | 4.5/4.5/4.5 | 8.70 | 67.44% | |
| Qwen2.5-14B | FP16 | 16/16/16 | 5.29 | 72.60% |
| QuaRot | 4/4/4 | 6.80 | 68.68% | |
| ResQ | 4.5/4.5/4.5 | 6.50 | 69.64% | |
| CoQuant (Ours) | 4.5/4.5/4.5 | 6.48 | 70.01% |
Ablation Study¶
1. Impact of Covariance Components on Subspace Optimization: Decoupling the CoQuant objective on Llama-3.2 models reveals the comparative performance of activation-only (ResQ), weight-only, and joint optimization.
| Model | Optimization Objective | Wiki PPL (↓) | Avg 0-shot (↑) |
|---|---|---|---|
| Llama-3.2-1B | Activation-only (ResQ) | 12.03 | 52.30% |
| Weight-only | 13.23 | 50.68% | |
| Joint (CoQuant) | 11.60 | 52.75% | |
| Llama-3.2-3B | Activation-only (ResQ) | 9.18 | 60.03% |
| Weight-only | 10.23 | 57.61% | |
| Joint (CoQuant) | 8.93 | 60.23% |
2. Resilience Under Ultra-Low Bit Quantization Stress (3.375-bit Average): Pushing models to extreme limits by compressing the low-precision majority to 3-bit while keeping 1/8 channels in 6-bit (average 3.375-bit) demonstrates the robust stability of CoQuant against baseline collapse.
| Model | Method | Average Bits | Wiki PPL (↓) | Avg 0-shot (↑) |
|---|---|---|---|---|
| Llama-3.2-1B | FP16 | 16 | 9.75 | 56.48% |
| QuaRot | 3.0 | 950.62 | 39.90% | |
| ResQ | 3.375 | 39.81 | 42.81% | |
| CoQuant | 3.375 | 28.71 | 43.75% | |
| Llama-3.2-3B | FP16 | 16 | 7.81 | 64.18% |
| QuaRot | 3.0 | 193.60 | 38.26% | |
| ResQ | 3.375 | 16.77 | 46.77% | |
| CoQuant | 3.375 | 15.76 | 48.39% |
Key Findings¶
- Symmetric Joint Covariance is Essential: Ablation confirms that weight-only PCA performs worst (PPL degrades to 13.23 on 1B) because it fails to protect activation outliers. While activation-only (ResQ) suppresses outliers, it ignores structural weight sensitivity. CoQuant cleanly synthesizes both, reducing PPL to 11.60. An isolated layer-wise error check on the
kprojmodule shows CoQuant achieves a 30%–60% reduction in actual output perturbation across all transformer layers. - Superior Robustness on Severe Outlier Architectures: On Qwen2.5 models notorious for extreme activation anomalies, uniform baselines (RTN, GPTQ, and QUIK) completely diverge (PPL exceeding 20,000 on 7B). CoQuant maintains strong stability, attaining 57.59% average zero-shot reasoning on Qwen2.5-1.5B—outperforming ResQ by 1.93 percentage points.
- Robustness Under Ultra-Low Bit Regimes: When pushed to 3.375 bits, rotation baseline QuaRot collapses (1B PPL explodes to 950.62), whereas CoQuant reduces ResQ's perplexity from 39.81 down to 28.71, indicating that joint modeling becomes increasingly critical as quantization noise intensifies.
- Minimal Sample Calibration Overhead: Experiments across calibration set sizes (16 to 512 samples) indicate that just 16 sequences achieve 11.70 PPL, while 512 samples provide marginal refinement to 11.67, confirming high calibration efficiency without risk of overfitting.
Highlights & Insights¶
- Mathematical Elegance of Closed-Form Weighted PCA: Leveraging the isotropic nature of rotated representations, CoQuant bypasses loose sub-multiplicative bounds and derives an exact trace formulation, giving joint quantization a clean, closed-form spectral interpretation.
- Unifying Matrix Multiplications Across Weights and KV Caches: By abstracting static projection layers and dynamic attention dot-products into a unified linear interaction framework, CoQuant seamlessly protects decoding-stage Key and Value states without ad-hoc heuristics.
- Hardware-Friendly Fused Covariance Formulations: Grouping shared-input operators (QKV in attention, Gate/Up in MLP) into fused covariance matrices solves the subspace decomposition in a single eigendecomposition, ensuring compatibility with standard kernel fusion optimizations.
Limitations & Future Work¶
- First-Order Error Approximation: The theoretical formulation drops the second-order noise interaction term \(E_X E_W\). While valid for small perturbations, this assumption may degrade under extreme sub-2-bit quantization.
- Uniform Subspace Ratios Across Layers: The method assigns a fixed high-precision ratio (\(r = 1/8\)) across all layers, leaving layer-wise or block-wise adaptive rank allocation unexplored.
- Lack of Custom High-Throughput Kernels: The study primarily validates mathematical formulation and numerical accuracy; dedicated CUDA/Triton kernels for mixed-precision GEMM are required to realize maximum end-to-end latency speedups in production.
Related Work & Insights¶
- vs ResQ (Saxena et al., 2025): ResQ introduced subspace partitioning for mixed precision but relied solely on activation covariance; CoQuant demonstrates that output perturbation is jointly governed by weights, incorporating weight covariance to achieve a 30%–60% layer-wise error reduction.
- vs QuaRot (Ashkboos et al., 2024b): QuaRot uses random rotations to enable uniform 4-bit quantization; CoQuant integrates rotations with low-rank mixed-precision subspaces, preventing accuracy degradation at ultra-low bit-widths.
- vs QUIK (Ashkboos et al., 2024a): QUIK retains sensitive outlier channels based on coordinate-aligned heuristics; CoQuant rotates and projects representations into an optimal spectral subspace, yielding higher compression fidelity on models with dispersed outliers.
Rating¶
- Novelty: ⭐⭐⭐⭐ [Derives an exact closed-form weighted PCA objective for joint weight-activation subspace projection, moving beyond activation-only heuristics]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across Llama-3.2 and Qwen2.5 models covering WikiText, six reasoning benchmarks, long-context generation, and stress tests]
- Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical progression from isotropic noise assumptions through trace identities to the final Rayleigh-Ritz solution]
- Value: ⭐⭐⭐⭐⭐ [Provides a practical, robust, and mathematically grounded foundation for sub-4-bit LLM mixed-precision deployment]