Skip to content

HyQuant: Hybrid-Precision Quantization for LLM Attention

Conference: EMNLP 2026
Paper: https://aclanthology.org/2026.emnlp-main.1705/
Code: https://github.com/jerrysfls/HyQuant
Area: LLM Reasoning
Keywords: Hybrid-Precision Quantization, Attention Mechanism, KV-Cache Compression, Long-Context Inference, Vertical Attention Pattern

TL;DR

To tackle severe performance degradation in aggressive low-bit attention during long-context Chain-of-Thought (CoT) reasoning, HyQuant introduces a vertical-line-aware hybrid-precision quantization framework that preserves a tiny subset (<5%) of persistent vertical-line anchor tokens and a recent sliding window in full precision while compressing the vast majority of historical keys/values to 4-bit with fused FlashAttention-style operators, achieving near-lossless accuracy alongside 1.32ร—โ€“3.58ร— decode-kernel speedups.

Background & Motivation

With recent frontier models such as OpenAI o1, Gemini, and DeepSeek bringing long Chain-of-Thought (CoT) reasoning into mainstream applications, LLM generation traces frequently scale to tens of thousands or even hundreds of thousands of tokens. In such massive context regimes, system bottlenecks diverge sharply across inference stages: the Prefill stage is dominated by large-scale matrix multiplications over the entire prompt prefix and is fundamentally compute-bound, whereas the autoregressive Decode stage repeatedly accesses and updates historical Key-Value (KV) states token by token, rendering memory capacity and memory bandwidth the primary runtime constraints. While low-bit quantization has been widely explored to mitigate these hardware pressures, aggressively pushing attention modules and KV caches to uniform low bit-widths consistently leads to catastrophic degradations in reasoning fidelity.

This failure stems from the severe non-uniformity and structural imbalance of attention allocations in modern language models. Quantization errors are not uniformly distributed in their downstream impact: they are exponentially amplified at high-scoring key positions that serve as global attention anchors, where minor numerical perturbations in \(K\) and \(V\) get repeatedly propagated across thousands of subsequent query decoding steps. Existing sparse attention methods (such as block-sparse skipping or selective KV cache eviction) tend to be fragile in multi-step reasoning where token importance is non-stationary and prematurely evicted tokens may become vital later. Concurrently, existing KV cache quantization methods treat token positions homogeneously, failing to provide token-heterogeneous precision allocations that protect sensitive attention anchors from lossy compression.

The authors ground their strategy in a pervasive empirical observation across model architectures (including Qwen3, Llama-3.1, and Gemma4): attention heatmaps consistently manifest pronounced vertical bright stripes. These vertical lines represent a minute subset of persistent attention anchors (typically under 5% of all keys) that absorb a disproportionately large share of global attention mass over long generation spans, and combining them with a modest local window accounts for over 80% of total attention mass. Core idea: instead of uniform low-bit quantization or lossy cache eviction, preserve the tiny subset (<5%) of vertical-line anchor tokens and a local sliding window in full precision (FP16/BF16) while compressing the vast majority of context to low bit-widths (4-bit), unified through tightly fused operators across both Prefill and Decode stages.

Method

Overall Architecture

HyQuant is an integrated hybrid-precision attention system spanning both compute-intensive prefill and memory-bandwidth-bound decode phases. Built upon the observation of column-wise attention concentration, HyQuant partitions historical keys into three disjoint sets: persistent vertical-line anchors \(K_{\text{VL}}\), a local sliding window \(K_{\text{Win}}\), and the quantizable long-tail majority \(K_Q\). During the Prefill phase, it executes a single segmented FlashAttention-style operator fusing low-bit matrix multiplication over \(K_Q\) with full-precision computation over \(K_{\text{VL}} \cup K_{\text{Win}}\). During the Decode phase, historical states are organized into compact 4-bit storage alongside full-precision anchor buffers, where an online dequantization-in-attention kernel unpacks quantized blocks on the fly without intermediate global memory roundtrips.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Query Q and Historical KV Sequences"] --> B["Vertical-Line Awareness & Token Set Partitioning<br/>Identify Top-ฯ Column Mass and Local Window W"]
    B --> C["Prefill Segmented Hybrid-Precision Attention<br/>Fused Online Softmax over Low-Bit Bulk and FP Anchors"]
    C --> D["Prefill-to-Decode Cache Reorganization<br/>Freeze Vertical Set and Pack Non-Window KV into 4-Bit"]
    D --> E["Decode Fused Dequantization-in-Attention Kernel<br/>On-the-Fly 4-Bit Dequantization and FP Accumulation"]
    E --> F["Output Autoregressive Hidden State ot"]

Key Designs

1. Vertical-Line Awareness and Token Set Partitioning: Lightweight Identification of Global Anchors

Uniform low-bit quantization severely corrupts high-attention anchors whose perturbation corrupts all subsequent queries. To isolate these sensitive tokens with minimal overhead, HyQuant separates keys into a recent sliding window \(K_{\text{Win}}(t) = \{k \mid k \in [\max(0, t-W+1), t]\}\) of width \(W\) (default \(W=128\)) and a non-window prefix \(K_{\text{pre}}(t) = K \setminus K_{\text{Win}}(t)\). To identify persistent vertical columns, HyQuant tracks running column-wise attention mass:

\[S(k) = \sum_{t \in \mathcal{T}} a_{t,k}\]

where \(\mathcal{T}\) represents an evaluation query set (e.g., tail queries within a running buffer). Selecting the highest-scoring fraction \(\rho\) (typically \(\rho=5\%\)) yields the vertical-line set \(K_{\text{VL}} = \text{TopK}(S|_{K_{\text{pre}}}, \lceil\rho \cdot |K_{\text{pre}}|\rceil)\), while remaining non-window tokens constitute \(K_Q\). Because \(S(k)\) is computed natively over Grouped Query Attention (GQA) heads without materializing full expansion and evaluated only every 64 steps, vertical-line awareness incurs an auxiliary compute latency of merely 3% to 5%, avoiding expensive pre-scoring overheads present in sparse retrieval methods.

2. Prefill-Stage Segmented Fused Hybrid-Precision Attention: Balancing High Throughput with Numerical Precision

Executing full-precision anchors and low-bit bulk context via disjoint kernel launches incurs prohibitive kernel dispatch and global memory I/O overhead. HyQuant designs a unified Triton-based segmented online softmax operator that processes both low-bit tiles and full-precision tiles in a single pass:

\[O = \text{Softmax}\left(\left[\frac{Q \hat{K}_Q^\top}{\sqrt{d}} \;\middle\|\; \frac{Q K_{\text{FP}}^\top}{\sqrt{d}}\right]\right) \begin{bmatrix} \hat{V}_Q \\ V_{\text{FP}} \end{bmatrix}\]

Tile processing executes in three sequential stages within the blockwise loop: it first computes low-bit (INT8 or FP8) GEMM over \(K_Q\), maintaining running row-max \(m\), partition sum \(\ell\), and accumulator \(\text{acc}\); it then seamlessly streams tiles from \(K_{\text{VL}}\) in FP16, followed by the local causal window \(K_{\text{Win}}\) in FP16, continuously updating \((m, \ell, \text{acc})\) before normalizing via \(O_b = \text{acc} / \ell\). This architecture retains the high computational throughput of low-bit tensor cores while driving layer-wise mean squared error (MSE) down to the level of 8-bit baselines.

3. Decode-Stage Hybrid Low-Bit KV Cache with Fused Attention: Eliminating Memory Bandwidth Bottlenecks

Autoregressive decoding is heavily bottlenecked by memory bandwidth during past KV fetches. Conventional quantization schemes that decouple dequantization from attention computation (dequantize-then-attend) inadvertently inflate memory traffic by materializing full-precision tensors back to DRAM. HyQuant reorganizes the KV cache at the prefill boundary (FREEZE): \(K_{\text{VL}}\) and \(K_{\text{Win}}\) remain stored in native FP16, a small staging buffer \(K_S\) accepts new tokens, and the vast non-window prefix is packed into Key-4bit and Value-4bit representations. During decoding, the kernel decomposes the sequence into parallel split chunks, unpacks 4-bit tiles directly into on-chip registers on the fly (dequantize-in-attention), computes online softmax attention, and merges results with the FP16 vertical-line and local-window buffers. By completely circumventing global memory roundtrips, it cuts KV memory footprint by up to 75% and achieves dramatic speedups on bandwidth-bound workloads.

Benchmark & Evaluation Protocol

Evaluations are conducted on mathematical reasoning benchmarks (GSM8K and MATH500) and the bilingual long-context benchmark LongBench v1 (spanning 11 tasks across Single-Document QA, Multi-Document QA, Summarization, Few-shot Learning, and Code/Semantic Retrieval). Experiments evaluate four backbone models: Qwen3-8B (with long-CoT thinking mode enabled), Qwen3-32B, Llama-3.1-8B-Instruct, and GLM-4-9B-0414. Testing is deployed on an NVIDIA H100 (80GB) GPU under standard configurations: 4-bit KV quantization, top-5% vertical token retention (\(\rho=0.05\)), and sliding window \(W=128\), measuring end-to-end task accuracy, layer-wise attention MSE against FlashAttention-2 (FA2), decode latency across prefix lengths from 1K to 32K tokens, and high-concurrency throughput.

Key Experimental Results

Main Results

On the comprehensive LongBench v1 benchmark, HyQuant with 4-bit KV cache and top-5% full-precision anchors matches or slightly surpasses full-precision FlashAttention-2 (FA2), drastically outperforming strict low-bit baselines (KIVI, KVTuner, and SageAttention):

Model / Method Single-Doc QA (Narr/Qasp/MF) Multi-Doc QA (Hotp/2Wiki/Mus) Summarization (QMS/MNews) Few-shot & Retrieval (Triv/SAM/Pre) LongBench Avg
Qwen3-8B (Thinking Mode)
FA2 (Full Precision) 30.37 / 41.68 / 50.34 50.42 / 39.69 / 33.88 13.11 / 10.85 97.50 / 33.80 / 88.81 44.59
KIVI (K4V4) 24.76 / 35.13 / 45.51 42.55 / 31.04 / 18.92 10.51 / 8.19 88.87 / 27.56 / 81.41 37.68 (-6.91)
SageAttention 19.45 / 34.54 / 41.95 46.59 / 44.33 / 24.31 18.67 / 23.23 74.93 / 32.62 / 58.86 38.13 (-6.46)
KVTuner (4bit) 13.74 / 32.45 / 45.32 37.68 / 54.18 / 31.92 20.20 / 10.67 94.70 / 29.32 / 74.81 40.45 (-4.14)
HyQuant (K4V4, top-5%) 32.28 / 41.82 / 49.74 49.75 / 43.31 / 35.24 15.39 / 11.94 96.65 / 33.05 / 86.32 45.04 (+0.45)
Llama-3.1-8B-Instruct
FA2 (Full Precision) 30.96 / 45.16 / 54.60 55.63 / 45.70 / 31.69 24.26 / 26.17 88.36 / 12.48 / 97.90 46.63
SageAttention 31.16 / 43.29 / 55.01 51.81 / 46.97 / 30.99 24.96 / 26.64 86.67 / 6.87 / 98.15 45.68 (-0.95)
KVTuner (4bit) 31.29 / 42.06 / 53.97 50.11 / 45.57 / 32.49 24.91 / 27.08 88.42 / 7.68 / 98.70 45.66 (-0.97)
HyQuant (K4V4, top-5%) 30.75 / 46.51 / 55.84 51.22 / 45.30 / 31.90 24.84 / 27.21 87.67 / 13.75 / 99.12 46.73 (+0.10)
Qwen3-32B (Thinking Mode)
FA2 (Full Precision) 34.93 / 46.38 / 43.79 54.18 / 50.57 / 45.59 17.11 / 14.74 90.60 / 36.80 / 100.00 48.61
KIVI (4bit) 19.89 / 42.21 / 37.87 47.42 / 46.16 / 41.30 12.67 / 11.16 80.37 / 28.68 / 94.00 41.98 (-6.63)
KVTuner (4bit) 32.97 / 45.64 / 44.89 52.13 / 50.47 / 45.05 16.79 / 14.82 90.60 / 37.10 / 100.00 48.22 (-0.39)
HyQuant (K4V4, top-5%) 45.56 / 41.05 / 46.17 44.44 / 53.12 / 43.76 17.15 / 14.89 90.60 / 36.35 / 100.00 48.46 (-0.15)

In multi-step mathematical reasoning, Qwen3-8B with FA2 achieves 95.88% on GSM8K and 80.14% on MATH500. While uniform 4-bit quantization degrades performance severely (KIVI drops to 92.48% and 72.89%; KVTuner drops to 93.69% and 75.83%), HyQuant retains 96.52% on GSM8K and 78.73% on MATH500, confirming that preserving vertical anchor tokens prevents cascading logical errors during complex derivation steps.

Ablation Study

1. Decode Kernel Latency and End-to-End Speedup across Context Lengths (NVIDIA H100)

Sequence Length FA2 Latency (ms/token) HyQuant Latency (ms/token) Kernel Speedup (vs FA2) End-to-End Decode Speedup (vs FA2) KIVI Relative Speed (vs FA2) KVTuner Relative Speed (vs FA2)
1,024 0.224 0.170 1.32ร— 1.04ร— 0.72ร— 0.78ร—
2,048 0.416 0.173 2.40ร— 1.07ร— 0.72ร— 0.79ร—
4,096 0.805 0.263 3.06ร— 1.09ร— 0.73ร— 0.80ร—
8,192 1.604 0.478 3.36ร— 1.12ร— 0.73ร— 0.80ร—
16,384 3.181 0.903 3.52ร— 1.13ร— 0.71ร— 0.74ร—
32,768 6.354 1.775 3.58ร— 1.17ร— 0.69ร— 0.72ร—

2. Quantized Long-Tail vs. Vertical-Only Dynamic Sparsity (MInference Style)

To evaluate the marginal utility of retaining non-vertical context in low-bit precision rather than dropping it via sparse attention masks (e.g., MInference), an ablation on Qwen3-8B across 100 test samples yields:

Configuration NarrativeQA (F1) HotpotQA (F1) MultiNews (ROUGE-L) TriviaQA (F1) Average Delta vs FA2
FA2 (Full Attention Reference) 31.49 62.83 14.45 89.86 49.66 -
MInference-Vertical (Vertical Sparsity Only) 14.69 38.55 13.87 79.20 36.58 -13.08
HyQuant (FP16 Vertical + 4-Bit Long Tail) 31.49 62.83 14.52 91.25 49.97 +0.31

Key Findings

  • The Long Tail Retains Critical Context: Discarding non-vertical tokens outright (pure sparsity) causes a catastrophic 24.28-point drop on HotpotQA and a 13.08-point drop overall. Compressing these tokens into 4-bit format completely restores performance, proving that low-bit background representations prevent severe contextual starvation.
  • Dequantize-in-Attention Is Mandatory for Real Speedups: Traditional low-bit KV baselines (KIVI, KVTuner) suffer from a "dequantize-then-attend" penalty, running 20% to 30% slower than unquantized FA2 due to global memory roundtrips. HyQuant's on-the-fly register unpacking enables monotonic speedup scaling, achieving up to 3.58ร— kernel acceleration and 1.17ร— end-to-end decode speedup at 32K context.
  • Superior Scalability under High Concurrency: In high-parallel batch scaling on Qwen3-8B (32K prefix on a single H100), FA2, KIVI, and KVTuner trigger Out-Of-Memory (OOM) at Batch Size = 16. In contrast, HyQuant comfortably scales to Batch Size = 16 at a throughput of 231.6 tokens/s.

Highlights & Insights

  • Reimagining Attention Geometry for Quantization: While earlier literature treated vertical attention stripes as cues for sparse pruning, HyQuant re-envisions them as sensitivity boundaries for mixed-precision arithmetic, establishing that protecting <5% of anchor tokens is sufficient to stabilize 4-bit attention.
  • Integrated Full-Lifecycle Optimization: Unlike prior techniques isolated to prefill matrix operations or decode KV cache storage, HyQuant introduces an end-to-end framework that couples segmented online softmax during prefill with on-the-fly register dequantization during decode.
  • Seamless Plug-and-Play Adaptability: The method requires zero post-training weight tuning or offline calibration, making it immediately deployable across long-context retrieval, document processing, and mathematical derivations.

Limitations & Future Work

  • Diminished Returns at Short Contexts: On sequence lengths under 1,024 tokens, inference time is heavily dominated by linear projection weights rather than KV cache memory traffic; thus, end-to-end decode speedup remains marginal (~4%).
  • Hardware Saturation Boundaries: On cutting-edge accelerators like the NVIDIA H100, single-batch decoding is partially compute-latency bound at intermediate lengths, limiting the visibility of memory bandwidth savings until sequence lengths surpass 8K or batch sizes scale up.
  • Unverified Dynamics in Interactive Tool-Use Environments: Current validations focus on extended CoT derivation and document aggregation; behavior under highly non-stationary, multi-turn agentic environments with intermittent tool outputs remains open for exploration.
  • vs MInference: MInference relies on vertical-slash dynamic masks to drop unselected tokens entirely; HyQuant demonstrates that complete omission severely compromises multi-step reasoning, whereas keeping the long tail in 4-bit precision achieves lossless fidelity with comparable efficiency.
  • vs KVTuner / KIVI: KIVI and KVTuner focus on per-channel or layer-wise quantization policies but treat tokens within a layer uniformly and execute decoupled dequantization; HyQuant exploits spatial token-level attention imbalance and avoids memory traffic through kernel-fused online dequantization.

Rating

  • Novelty: โญโญโญโญโ˜† Translating visual vertical-line attention patterns into token-wise mixed-precision boundaries is intuitive, elegant, and impactful.
  • Experimental Thoroughness: โญโญโญโญโญ Comprehensive coverage across 4 modern LLM families, LongBench tasks, mathematical reasoning, kernel-level profiling, and multi-batch scaling.
  • Writing Quality: โญโญโญโญโญ Clear theoretical rationale, structured mathematical exposition, and rigorous algorithmic pseudocode.
  • Value: โญโญโญโญโญ Extremely high engineering and practical value for low-cost, lossless long-context and extended-reasoning model serving.