PN-Mix: Pair-Norm Guided Mixed-Precision Key Cache Quantization for Efficient Long-Output LLM Inference¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Summer-Summer/Kitty (Extended upon Kitty baseline)
Area: LLM Reasoning
Keywords: KV Cache Quantization, Mixed Precision, RoPE Channel Pairs, Long Chain-of-Thought, Attention Logit Perturbation
TL;DR¶
Addressing the severe KV cache memory bloat and reasoning accuracy collapse under uniform 2-bit quantization in long-output LLMs, PN-Mix demonstrates that query-key pair-norm products in RoPE-coupled channels dictate attention logit perturbation, proposing a mixed-precision key-cache quantization framework (2.26-bit Key) with one-time prefill salient-pair selection that maintains reasoning accuracy and unlocks up to 2.37x serving throughput.
Background & Motivation¶
As modern large language models (LLMs) transition from short-form instruction following to complex long chain-of-thought (Long-CoT) reasoning, leading open-source models such as DeepSeek-R1 and the Qwen series generate tens of thousands of intermediate reasoning tokens when tackling competition mathematics (e.g., AIME) and complex software synthesis. However, autoregressive decoding incurs a key-value (KV) cache that scales linearly with both batch size and sequence length (\(\mathcal{O}(B \cdot L \cdot H_{kv} \cdot T \cdot d)\)), rapidly becoming the primary hardware bottleneck that constrains batch capacity and generation horizon. For example, Qwen3-8B in FP16 triggers an out-of-memory (OOM) fault on a 40GB NVIDIA A100 GPU when the generation length exceeds 4K at a batch size of 32, or when the batch size exceeds 20 at an 8K generation length. Although aggressive KV cache quantization offers an intuitive remedy, conventional uniform 2-bit quantization schemes (such as KIVI and PolarQuant) incur severe error accumulation over long reasoning trajectories, precipitating a catastrophic breakdown in logical consistency.
Diagnostic sensitivity analyses reveal that under stringent low-bit constraints, the key cache is substantially more vulnerable to quantization noise than the value cache, with error sensitivity distributed heterogeneously across feature channels. To retain generation fidelity within tight bit budgets, recent efforts have shifted toward channel-level mixed-precision key quantization. For instance, ChanMix utilizes an offline calibration dataset to identify retrieval-heavy heads and allocate 4-bit precision, while assigning residual channels via range clustering; Kitty dynamically re-evaluates channel saliency per quantization group during decoding based on mean absolute key magnitudes, allocating 4-bit and 2-bit budgets. Nonetheless, all existing channel-level mixed-precision methods determine importance purely from isolated key statistics (such as peak range \(\max_t K - \min_t K\) or magnitude \(\mathbb{E}_t[|K|]\)), entirely ignoring the fundamental query-key inner-product interaction that governs attention logits. Furthermore, they partition precision along isolated scalar channels, breaking the two-dimensional rotary coupling naturally established across adjacent channel pairs by Rotary Position Embedding (RoPE). Moreover, dynamic group-wise re-estimation in schemes like Kitty introduces non-trivial index storage overhead and memory traffic during decoding.
Recognizing that attention distributions emerge from joint query-key interactions and that RoPE couples feature channels into structural pairs, the central thrust of this work is to decompose attention logit perturbations to uncover the precise channel structures that amplify quantization error. Core idea: by conducting attention-logit perturbation analysis at the RoPE channel-pair granularity, the query-key pair-norm product is proven to be the primary sensitivity amplifier; utilizing a separable prefill-stage formulation enables one-time input-adaptive identification of salient RoPE pairs assigned 3-bit precision, paired with 1-bit midpoint correction for residual channels, realizing lightweight 2.26-bit key-cache quantization without decoding-time recomputation or dynamic index overhead.
Method¶
Overall Architecture¶
PN-Mix is tailored for RoPE-based decoder-only LLM inference, structured into three primary phases: prefill-stage salient channel-pair identification, group-wise mixed-precision cache layout structuring, and Triton-accelerated fused dequantization-attention execution. During prompt prefill, the framework tracks post-RoPE query and key states across layers and KV heads, evaluating the query-key pair-norm product to score channel-pair saliency \(\sigma_{h,i}\). The top-\(r\%\) (e.g., 50%) pairs are selected and encoded into a static binary mask. In the subsequent autoregressive decoding phase, the initial \(N_s = 32\) attention sink tokens and a rolling residual buffer of size \(N = 128\) remain uncompressed in FP16 to guarantee numerical stability. Once full, buffered keys are committed to mixed-precision pages (salient pairs stored in 3-bit, non-salient pairs compressed to 1-bit with bin midpoint correction), while values are quantized using per-token 2-bit asymmetric groups. Custom Triton kernels perform on-chip register-level dequantization and streaming dot-product operations, eliminating intermediate global memory roundtrips.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Prompt & Prefill States<br/>Q, K, V Tensors (Post-RoPE)"] --> B["Query-Key Channel-Pair Norm Saliency Criterion<br/>Decoupled estimation of attention perturbation sensitivity"]
B --> C["Prefill-Stage One-Time Salient Pair Prediction & Reuse<br/>Generate static binary mask without decoding-time recomputation"]
C --> D["RoPE Block Mixed-Precision Layout & 1-bit Midpoint Correction<br/>Salient pairs assigned 3-bit / residual 1-bit + per-token 2-bit V"]
D --> E["Fused Triton Kernels & Autoregressive Decoding<br/>Ultra-long generation with 2.37x serving throughput scaling"]
Key Designs¶
1. Query-Key Channel-Pair Norm Saliency Criterion: Key-only scalar statistics neglect attention interaction and RoPE rotation pairing -> Decompose attention-logit perturbation into pair-norm sensitivity scaling -> Accurately isolate dominant logit error contributors
Prior key-cache mixed-precision techniques quantify channel importance exclusively via univariate key magnitude or range, detached from actual query interactions. Under RoPE, adjacent channel dimensions are rotated in coordinate pairs, structuring the head dimension \(d\) into \(d/2\) discrete two-dimensional subspaces: \(q_{ti} := (\tilde{Q}_{h,t,2i-1}, \tilde{Q}_{h,t,2i})\) and \(k_{si} := (\tilde{K}_{h,s,2i-1}, \tilde{K}_{h,s,2i})\). Dropping the constant scale factor \(1/\sqrt{d}\), the unquantized attention logit decomposes into vector norms and angular cosine components: $\(\tilde{Q}_{h,t} \tilde{K}_{h,s}^\top = \sum_{i=1}^{d/2} \|q_{ti}\|_2 \|k_{si}\|_2 \cos(\phi_{t,s}^i)\)$ When key tensors are quantized and reconstructed as \(k'_{si}\), norm and angle perturbations manifest as \(\|k'_{si}\|_2 = \|k_{si}\|_2 + \epsilon_{si}\) and \(\cos(\phi'^i_{t,s}) = \cos(\phi^i_{t,s}) + \delta^i_{t,s}\). Defining the relative norm distortion as \(\rho_{si} = \epsilon_{si}/\|k_{si}\|_2\), the exact logit discrepancy is formulated as: $\(\tilde{Q}_{h,t} \tilde{K}_{h,s}^\top - \tilde{Q}_{h,t} \tilde{K'}_{h,s}^\top = -\sum_{i=1}^{d/2} \|q_{ti}\|_2 \|k_{si}\|_2 \left( \delta^i_{t,s} + \cos(\phi^i_{t,s})\rho_{si} + \rho_{si}\delta^i_{t,s} \right)\)$ While the parenthesized terms represent quantization-induced geometric distortions, the pre-quantization query-key pair-norm product \(\|q_{ti}\|_2 \|k_{si}\|_2\) serves as the exact scaling factor dictating how severely local quantization error translates into logit degradation. To avoid enumerating all causal query-key coordinate pairs \((t, s)\) with \(s \le t\), PN-Mix introduces a separable formulation: $\(\sigma_{h,i} = \bar{q}_{h,i} \cdot \bar{k}_{h,i} = \mathbb{E}_t\left[\|q_{ti}\|_2\right] \cdot \mathbb{E}_s\left[\|k_{si}\|_2\right]\)$ Empirical benchmarks confirm that this separable metric achieves a Spearman rank correlation of 0.990–0.996 and an overlap exceeding 95% with the exact causal-joint oracle ranking across layers and heads, eliminating computational complexity while capturing true attention sensitivity. For Grouped Query Attention (GQA), query scores are pooled across all query heads sharing KV head \(h\): \(\sigma_{h,i} = \frac{1}{|g^{-1}(h)|} \sum_{h' \in g^{-1}(h)} \bar{q}_{h',i} \bar{k}_{h,i}\).
2. Prefill-Stage One-Time Salient Pair Prediction & Reuse: Decoding-time dynamic channel re-selection incurs heavy indexing overhead -> Exploit high temporal stability and concentration of pair saliency -> Eliminate online recomputation and memory traffic
Dynamic mixed-precision frameworks like Kitty periodically recalibrate channel saliency for each quantized group, requiring dedicated index buffers to track channel permutations across sequence pages. PN-Mix investigates the temporal persistence of channel-pair importance and discovers that attention saliency is heavily concentrated and remarkably stable across extensive generation trajectories: across Qwen3 and DeepSeek-R1, the top 50% of channel pairs consistently capture 97%–99% of total importance mass. More crucially, the salient channel pairs identified solely from prompt prefill tokens exhibit sustained overlap with full-sequence decoding ground truth (Jaccard similarity remains at ~0.72 even along an extreme 32,672-step decoding run from a 96-token prefill prompt, with step-wise temporal drift under 0.007). Capitalizing on this stability, PN-Mix evaluates \(\sigma_{h,i}\) strictly during prefill, generating a compact binary mask of size \(d/2\) bits per KV head that remains static throughout decoding. Across batch size \(B\) and \(L\) layers, total metadata overhead is strictly bounded to \(B \times L \times H_{kv} \times (d/2)\) bits, amortizing to an asymptotic cost of nearly zero (\(1 / (2(N_s + T_q))\) bits/token).
3. RoPE Block Mixed-Precision Layout & 1-bit Midpoint Correction: Uniform 2-bit quantization suffers severe CoT degradation while raw 1-bit exhibits truncation bias -> Assign 3-bit to salient pairs / 1-bit with midpoint correction to residual pairs + per-token 2-bit V -> Deliver high fidelity at 2.26-bit budget
PN-Mix implements an asymmetric bit-allocation layout to balance compression and fidelity. In the key cache, the top-\(r\%\) salient RoPE pairs are allocated \(b_H = 3\) bits (integers in \([0, 7]\)), whereas the remaining non-salient pairs are compressed to \(b_L = 1\) bit (integers in \([0, 1]\)). Because standard min-max 1-bit quantization snaps values exclusively to extreme interval boundaries, introducing severe reconstruction bias, PN-Mix incorporates an asymmetric midpoint correction:
$\(s_X = \frac{\max X - \min X}{2}, \quad z_X = \frac{3 \min X + \max X}{4}\)$
This shifts reconstructed levels into bin centers (\(\hat{X} = s_X X_q + z_X\)), slashing mean squared error and suppressing attention KL divergence (\(\mathcal{D}_{\text{AKL}}\) drops from 0.273 to 0.080 in simulation). For value tensors, per-token 2-bit asymmetric grouping (\(b_V = 2\)) is deployed to align with weighted-sum reductions. At the hardware level, Triton kernels pack least significant bits (LSB 1-bit) across all channels into a uniform packed_low tensor (8 tokens per byte), while the upper 2 bits of salient pairs are packed into a selective packed_hi tensor (4 tokens per byte). Fused attention kernels load packed chunks into registers, dequantize on the fly, and perform dot-product arithmetic without materializing dequantized caches into global memory.
Benchmark & Evaluation Protocol¶
The empirical evaluation is organized around three rigorous validation dimensions: 1. Standard Reasoning and Code Generation Benchmarks: Tested on general-purpose models including Llama-3.1-8B-Instruct, Qwen3-8B, Qwen2.5-14B-Instruct, and Qwen3-32B across GSM8K (8-shot), MATH-Algebra (4-shot), HumanEval (0-shot Pass@1), and GPQA-Diamond (5-shot) with a 4096-token generation horizon. 2. Long Chain-of-Thought Mathematics: Evaluated on AIME 2024 and AIME 2025 using reasoning-specialized models (DeepSeek-R1-Distill-Llama-8B, DeepSeek-R1-Distill-Qwen-7B, and Qwen3-8B) with sequence limits up to 32,768 tokens (\(T=0.6, p=0.95, k=20\)), reporting average accuracy over 10 responses per problem. 3. Hardware Serving Efficiency: Measured on a single NVIDIA A100-SXM4-40GB GPU under a static cache pre-allocation protocol, tracking peak GPU memory footprint, time-to-first-token (TTFT), marginal decoding latency, and throughput scaling against FP16, uniform baselines (KIVI, PolarQuant), and channel-level baselines (ChanMix, Kitty). All compressed baselines operate in a matched 2.25–2.39 average key bit-width range.
Key Experimental Results¶
Main Results¶
The table below summarizes performance on standard reasoning and code synthesis benchmarks across four model scales, keeping the value cache in FP16 to isolate key quantization effects (mean ± standard deviation across 5 runs):
| Model | Method | Key Bits | GSM8K | MATH-Alg. | HumanEval | GPQA-Dia | Overall Avg. |
|---|---|---|---|---|---|---|---|
| Llama-3.1-8B-Instruct | FP16 | 16 | 76.83±0.90 | 48.17±0.89 | 64.51±1.70 | 26.63±1.22 | 54.04 |
| KIVI | 2.25 | 66.97±1.07 | 35.27±0.80 | 55.49±3.18 | 23.67±2.06 | 45.35 | |
| PolarQuant | 2.25 | 26.85±0.49 | 10.38±0.85 | 34.27±1.30 | 20.41±2.35 | 22.98 | |
| ChanMix | 2.32 | 70.51±0.75 | 37.57±0.70 | 59.63±2.23 | 23.37±1.38 | 47.77 | |
| Kitty | 2.39 | 74.51±0.62 | 46.25±0.72 | 62.68±1.78 | 23.98±2.14 | 51.86 | |
| PN-Mix (1) | 2.26 | 75.66±0.94 | 46.57±1.43 | 64.39±2.10 | 25.71±2.59 | 53.08 | |
| PN-Mix (2) | 2.29 | 75.65±0.61 | 46.28±1.24 | 63.78±2.30 | 26.33±3.02 | 53.01 | |
| Qwen3-8B | FP16 | 16 | 94.27±0.64 | 88.41±0.90 | 85.85±0.46 | 42.86±0.56 | 77.85 |
| KIVI | 2.25 | 88.20±0.74 | 69.47±0.23 | 79.88±2.18 | 35.10±0.59 | 68.16 | |
| PolarQuant | 2.25 | 1.15±0.09 | 0.00±0.00 | 2.07±0.30 | 0.82±0.69 | 1.01 | |
| ChanMix † | 2.32 | 90.34±5.86 | 86.57±4.01 | 83.65±1.30 | 38.97±0.76 | 74.88 | |
| Kitty | 2.39 | 91.93±0.44 | 82.29±0.68 | 83.90±1.62 | 39.59±2.53 | 74.43 | |
| PN-Mix (1) | 2.26 | 93.31±0.31 | 88.76±0.38 | 86.34±0.83 | 40.41±2.20 | 77.21 | |
| PN-Mix (2) | 2.29 | 93.89±0.25 | 87.40±0.46 | 84.87±2.26 | 41.84±2.21 | 76.66 | |
| Qwen2.5-14B-Instruct | FP16 | 16 | 90.83±0.29 | 78.16±0.29 | 77.07±2.46 | 40.00±3.27 | 71.52 |
| KIVI | 2.25 | 85.66±1.07 | 72.00±1.26 | 71.71±2.43 | 36.73±0.72 | 66.52 | |
| PolarQuant | 2.25 | 61.47±1.18 | 44.53±1.04 | 58.17±1.71 | 27.76±2.77 | 47.98 | |
| ChanMix | 2.32 | 88.79±0.53 | 73.95±0.66 | 74.88±2.95 | 41.02±3.19 | 69.66 | |
| Kitty | 2.39 | 90.46±0.79 | 80.39±0.93 | 77.20±3.38 | 40.00±2.38 | 72.01 | |
| PN-Mix (1) | 2.26 | 90.07±0.55 | 79.58±0.87 | 78.17±2.13 | 40.61±2.51 | 72.11 | |
| PN-Mix (2) | 2.29 | 89.01±0.46 | 78.15±1.54 | 77.44±1.02 | 40.71±1.81 | 71.33 | |
| Qwen3-32B | FP16 | 16 | 90.80±0.42 | 84.62±1.00 | 85.61±1.13 | 53.27±2.20 | 78.58 |
| KIVI | 2.25 | 87.47±0.62 | 59.87±1.15 | 84.51±0.62 | 46.73±1.53 | 69.65 | |
| PolarQuant | 2.25 | 63.21±0.64 | 1.31±0.13 | 46.71±2.27 | 7.55±1.63 | 29.70 | |
| ChanMix † | 2.32 | 90.08±0.84 | 82.73±0.78 | 86.46±2.36 | 48.78±2.15 | 77.01 | |
| Kitty | 2.39 | 91.10±0.54 | 76.73±0.98 | 87.68±2.36 | 47.86±1.53 | 75.84 | |
| PN-Mix (1) | 2.26 | 90.33±0.30 | 82.81±0.60 | 88.29±1.36 | 50.82±1.39 | 78.06 | |
| PN-Mix (2) | 2.29 | 90.61±0.41 | 83.91±0.71 | 85.37±1.77 | 48.88±0.59 | 77.19 |
On the rigorous AIME competition benchmarks under full KV cache compression (\(b_V = 2\)), PN-Mix displays superior resilience against long-CoT drift. For Qwen3-8B on AIME24, uniform KIVI drops to 56.33% and ChanMix achieves 60.66%, whereas PN-Mix (1) secures 74.67% (matching Kitty's 71.33% despite requiring fewer bits). On AIME25 (\(b_V = 2\)), PN-Mix (1) scores 64.00%, outpacing both KIVI (51.33%) and Kitty (62.00%).
Ablation Study¶
An ablation on Qwen3-8B investigates the role of the importance score, 1-bit midpoint adjustment, attention sink preservation, and online re-estimation (PN-Mix (2) base, \(b_V = 16\)):
| Importance Metric | 1-bit Midpoint Corr. | Sink Token Preserv. | AIME24 Acc. (%) | HumanEval (Pass@1 %) | Configuration Notes |
|---|---|---|---|---|---|
| FP16 Baseline | – | – | 76.33±4.82 | 85.85±0.46 | Uncompressed upper bound |
| \(\tilde{Q}\tilde{K}\) Pair Norm (PN-Mix) | ✓ | ✓ | 76.00±4.67 | 84.87±2.26 | Full proposed system |
| \(\tilde{Q}\tilde{K}\) Pair Norm | ✓ | ✗ | 72.00±5.42 | 84.02±2.54 | Omit initial 32 sink tokens |
| \(\tilde{Q}\tilde{K}\) Pair Norm | ✗ | ✓ | 69.67±6.90 | 80.36±2.48 | Omit 1-bit correction (major drop) |
| \(\tilde{K}\) Range (ChanMix-style) | ✓ | ✓ | 70.67±3.59 | 83.78±2.48 | Univariate key range |
| \(\tilde{K}\) Magnitude (Kitty-style) | ✓ | ✓ | 73.00±5.04 | 84.39±1.95 | Univariate mean absolute key |
| \(\tilde{K}\) Pair Norm | ✓ | ✓ | 74.67±4.00 | 84.63±2.01 | Key-only RoPE pair norm |
| \(\tilde{Q}\tilde{K}\) Pair Norm* (Online Dynamic) | ✓ | ✓ | 72.33±5.39 | 82.31±0.01 | Decoding-time group re-estimation |
In hardware profiling on an A100-40GB GPU running Qwen3-8B, at \(B = 32, P = 141, L_{\text{gen}} = 32\text{K}\), both FP16 and Kitty crash due to OOM, while PN-Mix executes successfully utilizing 37.12 GB. At \(L_{\text{gen}} = 8\text{K}\), FP16 caps at \(B = 20\) (469.8 tok/s peak), while PN-Mix scales comfortably to \(B = 110\) (38.21 GB), achieving 1114.0 tok/s—a 2.37x throughput speedup over the maximum feasible FP16 throughput.
Key Findings¶
- 1-bit midpoint correction is indispensable for extreme compression: Omitting midpoint correction triggers a 6.33% accuracy collapse on AIME24 (76.00% down to 69.67%), proving that eliminating residual endpoint clipping bias is mandatory when non-salient pairs are squeezed to 1-bit.
- Incorporating Query information provides a decisive edge: Progressing from \(\tilde{K}\) Magnitude (73.00%) to \(\tilde{K}\) Pair Norm (74.67%) and finally to the joint \(\tilde{Q}\tilde{K}\) Pair Norm (76.00%) demonstrates that scaling key error by query norm accurately targets actual attention logit sensitivity.
- Prefill-only selection outclasses dynamic online updating: The online variant \(\tilde{Q}\tilde{K}\) Pair Norm* degrades accuracy to 72.33% while inflating memory traffic with page-level index tables. The full prefill prompt establishes a reliable global semantic context that outperforms noisy step-wise local adaptations.
Highlights & Insights¶
- Theoretically grounded unification of RoPE geometry and logit perturbations: Moving beyond scalar channel heuristics, PN-Mix derives an exact perturbation formulation showing that 2D RoPE rotation blocks scale quantization distortion by the query-key pair norm product, placing KV cache compression on solid mathematical foundations.
- Zero-overhead metadata via static prefill prediction: Bypassing both fragile offline dataset profiling (ChanMix) and expensive online group-wise index tracking (Kitty), PN-Mix identifies salient channel pairs once during prefill, translating algorithmic insights into minimal runtime overhead.
- Graceful degradation on sub-2.25-bit Pareto frontiers: Across Pareto sweeps down to 1.5 bits, competing baselines suffer total accuracy failure, whereas PN-Mix sustains 67.66% on AIME24 (Qwen3-8B), demonstrating remarkable robustness under extreme memory pressure.
Limitations & Future Work¶
- Architectural specificity to RoPE: Saliency pair-norm tracking is explicitly formulated for Rotary Position Embedding where adjacent channels form 2D orthogonal rotation pairs. Applying PN-Mix to architectures utilizing ALiBi, NoPE, or absolute position embeddings will require generalizing the formulation to arbitrary channel covariance structures.
- Kernel dispatch overhead at low concurrency: Triton fused dequantization kernels introduce minor computational overhead. In memory-abundant regimes with small batch sizes (\(B=1\)) and moderate context lengths, FP16 remains slightly faster per step due to native matrix multiplication efficiency.
Related Work & Insights¶
- vs KIVI / PolarQuant: KIVI enforces uniform 2-bit per-channel quantization, while PolarQuant quantizes in polar coordinates; neither accounts for channel sensitivity heterogeneity, resulting in severe degradation or complete failure on long-CoT tasks. PN-Mix's mixed-precision RoPE pair allocation completely overcomes this accuracy cliff.
- vs ChanMix: ChanMix relies on static offline dataset calibration, failing to adapt to dynamic user prompts. PN-Mix computes \(\sigma_{h,i}\) directly from the target prompt during prefill, achieving input-adaptive saliency without offline profiling.
- vs Kitty: Kitty uses key magnitudes for group-wise dynamic re-estimation, missing query-side scaling and incurring recurring page index I/O overhead. PN-Mix leverages query-key pair norms with one-time prefill selection, delivering superior accuracy at a lower average bit-width (2.26 vs 2.39 bits) with faster decoding throughput.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Grounding mixed-precision key allocation in RoPE 2D geometric coupling and attention logit perturbation analysis is mathematically elegant and novel.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 4 standard models, 3 long-CoT models, diverse benchmarks (math, code, general reasoning), and end-to-end hardware benchmarks.
- Writing Quality: ⭐⭐⭐⭐⭐ Highly structured, rigorous mathematical derivations, and transparent hardware profiling.
- Value: ⭐⭐⭐⭐⭐ Provides immediate, tangible throughput improvements and memory reduction for production serving of long-context reasoning LLMs.