Don't Read Everything: A Curvature-Conditioned Query for Linear Attention¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/ledong0110/ccq
Area: LLM Reasoning
Keywords: Linear Attention, Local Curvature, Read-Side Correction, Covariance Contraction, Long-Context Retrieval
TL;DR¶
Addressing the failure of linear attention models in long-context retrieval caused by the lack of Softmax competitive normalization, this paper proposes Curvature-Conditioned Query (CCQ), an anisotropic query contraction based on running key covariance that modifies only the read step while preserving constant \(O(1)\) inference cache.
Background & Motivation¶
Linear attention and recurrent state-space models reduce the quadratic computational and memory footprint of standard Softmax attention to linear training time and constant inference cache by maintaining recurrent fast-weight matrix states. However, this efficiency gain comes at the cost of sacrificing the intrinsic competitive normalization mechanism of Softmax. In standard attention, the key-query inner product is coupled with a log-partition function that enforces a fixed probability budget across all past matches. Without this normalizer, the additive accumulation in linear attention allows historical key vectors along dominant covariance directions to build up as an overwhelming background, systematically diluting critical isolated targets and creating a severe performance gap on in-context retrieval and long-sequence reasoning.
Prior attempts to narrow this gap have almost exclusively intervened on the "write side" of memory. Data-dependent gating architectures such as GLA, RetNet, and Mamba selectively decay stale memory entries; delta-rule networks such as DeltaNet and Gated DeltaNet substitute additive writes with soft overwrites of similar keys; and kernel approximation approaches like Hedgehog warp feature maps to mimic sharp exponential distributions. Nonetheless, regardless of how carefully the state updates decide what enters memory, the read operation remains an unmodified, single matrix-vector product. Once the query arrives, it reads blindly from the accumulated state without adapting to the geometric distribution of the observed keys.
This paper addresses this underexplored read-side limitation by leveraging the geometric curvature of the missing normalizer. Once historical keys are observed, the competitive log-partition term is purely a function of the incoming query; a second-order Taylor expansion around the isotropic-attention reference point reveals that its Hessian curvature exactly matches the running sample covariance of the keys. Core idea: maintain running key covariance statistics via recurrent mechanisms identical to the linear attention backbone, and apply an anisotropic contraction to the query along dominant covariance directions prior to reading the fast-weight memory (CCQ), dynamically suppressing background clutter and surfacing isolated targets.
Method¶
Overall Architecture¶
The CCQ framework cleanly decouples the key-side geometric tracking, query-side curvature contraction, and the underlying backbone memory read. At each step, running first-order key statistics \(\mu_t\) and uncentred second-moment matrices \(\bar{C}_t\) are updated through chunkwise scans during training (or recursive updates during inference) to track the centred covariance \(\Sigma_t = \bar{C}_t - \mu_t \mu_t^\top\). Given the current query \(q_t\), a data-dependent gating scalar \(\lambda_t\) dynamically calibrates the contraction strength, producing a cleaned query \(q_t^{\text{clean}}\) that attenuates high-variance memory directions. Finally, the cleaned query reads from the black-box linear attention state \(S_t\) without modifying the underlying recurrent write rule.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input sequence embeddings x_t<br/>Project and L2-normalize to q_t, k_t, v_t"] --> B["Dynamic Key Statistics Tracking<br/>Maintain running mean μ_t and second-moment C_t"]
B --> C["Curvature-Conditioned Query Contraction<br/>Compute Σ_t and adaptive gate λ_t to clean query"]
C --> D["Decoupled Read and Backbone Memory Update<br/>Read fast-weight state S_t with cleaned query"]
D --> E["Output representation o_t<br/>Preserve O(1) inference cache without background dilution"]
Key Designs¶
1. Dynamic Key Statistics Tracking: Capturing Softmax local curvature via isomorphic recurrence To bypass the quadratic cost of materializing the full nonlinear log-partition function \(A_t(q) = \log \sum_{m \le t} \exp(k_m^\top q)\), CCQ proves that its Hessian at the isotropic reference point \(q=0\) coincides precisely with the running sample covariance of stored keys: $\(\nabla_q^2 A_t(0) = \Sigma_t \triangleq \bar{C}_t - \mu_t \mu_t^\top = \frac{1}{t}\sum_{j \le t} k_j k_j^\top - \left(\frac{1}{t}\sum_{j \le t} k_j\right)\left(\frac{1}{t}\sum_{j \le t} k_j\right)^\top\)$ To maintain this curvature at constant computational complexity, CCQ carries a \(d_k \times d_k\) running second-moment matrix \(\bar{C}_t\) alongside a \(d_k\)-dimensional prefix-sum key vector \(\mu_t\) per head. During training, these quantities are computed via chunkwise matrix multiplications and prefix sums fully fused into the chunk kernel, preserving high hardware throughput.
2. Curvature-Conditioned Query Contraction: Anisotropic suppression along dominant background axes To prevent keys aligned with high-frequency filler text from swamping rare target keys, CCQ establishes a linear query cleaning operator. Interpreting the quadratic curvature penalty \(R_t(q) = \frac{1}{2} q^\top \Sigma_t q\) as a measure of background exposure, the contraction performs a single gradient descent step on this penalty in query space: $\(q_t^{\text{clean}} = (I - \lambda_t \Sigma_t) q_t = q_t - \lambda_t \Sigma_t q_t, \quad \lambda_t = \sigma(W_\lambda q_t + b_\lambda)\)$ where \(\lambda_t \in (0, 1)\) is an input-conditioned scalar gate generated by a lightweight projection. Under unit-norm key normalization, \(\|\Sigma_t\|_2 \le \|\bar{C}_t\|_2 \le 1\), which strictly bounds the operator spectrum to \([1 - \lambda_t, 1] \subset (0, 1]\). Consequently, query directions orthogonal to memory variance pass through unaltered, while components overlapping dominant key directions are smoothly contracted without risk of numerical divergence or query annihilation.
3. Decoupled Read and Backbone Memory Update: Agnostic read-side enhancement To ensure universal composability with arbitrary linear attention mechanisms, CCQ completely decouples the state write rule from the state read. The backbone recurrence \(S_t = \text{Update}(S_{t-1}; k_t, v_t, \dots)\) remains a modular black box, while the read output is computed as: $\(o_t = S_t q_t^{\text{clean}} = S_t q_t - \lambda_t S_t \Sigma_t q_t\)$ In score space, this induces an implicit correction \(s_{tj}^{\text{CCQ}} = k_j^\top q_t - \lambda_t \text{Cov}_{\le t}[k_j^\top k, k^\top q_t]\), which penalizes a candidate key only when its alignment with the stored memory correlates with the query's read direction. This expands the retrieval margin for off-covariance targets (such as passkeys or UUIDs) against background distractors.
Loss & Training¶
All models are pretrained from scratch on identical subsets of FineWeb-Edu using standard next-token cross-entropy loss. Optimization uses AdamW with \(\beta_1 = 0.9, \beta_2 = 0.95\), weight decay of 0.1, a peak learning rate of \(10^{-3}\), 1024-step linear warmup, and cosine decay to zero in bfloat16 mixed precision. The architecture employs SwiGLU MLPs, RMSNorm, and untied input-output embeddings. The gating projection weights \(W_\lambda\) are initialized to zero with biases set such that the initial gate output is \(\approx 0.01\), ensuring training begins stably from standard linear attention dynamics.
Key Experimental Results¶
Main Results¶
Evaluations span both 500M (15B tokens) and 1.3B (40B tokens) parameter scales across language modeling perplexity (WikiText, LAMBADA) and seven zero-shot common-sense reasoning benchmarks, comparing Softmax Transformers, Mamba2, GLA, GLA-Hedgehog, Gated DeltaNet, and their CCQ-equipped counterparts.
| Scale / Model | WikiText PPL↓ | LAMBADA PPL↓ | LAMBADA Acc↑ | HellaSwag acc_n | BoolQ acc | 7-Task Avg Acc |
|---|---|---|---|---|---|---|
| 500M Scale | ||||||
| Transformer | 331.69 | 30.47 | 34.04 | 43.12 | 58.01 | 49.44 |
| Mamba2 | 26.22 | 29.37 | 34.62 | 44.15 | 55.84 | 49.17 |
| GLA | 27.27 | 32.11 | 34.02 | 41.93 | 59.66 | 48.54 |
| GLA-Hedgehog | 32.47 | 43.74 | 29.50 | 40.48 | 61.71 | 48.77 |
| Gated DeltaNet | 26.33 | 29.10 | 33.88 | 43.48 | 54.19 | 48.98 |
| CCQ-GLA | 26.17 | 25.77 | 36.74 | 43.45 | 56.85 | 49.73 |
| CCQ-Gated DeltaNet | 25.59 | 22.93 | 38.19 | 44.61 | 55.96 | 49.87 |
| 1.3B Scale | ||||||
| Transformer | 41.55 | 14.31 | 45.02 | 53.44 | 61.16 | 54.60 |
| Mamba2 | 18.17 | 13.55 | 45.55 | 57.00 | 56.97 | 55.17 |
| GLA | 18.80 | 12.31 | 47.02 | 55.44 | 58.01 | 55.75 |
| GLA-Hedgehog | 19.07 | 15.17 | 43.08 | 55.90 | 59.17 | 55.93 |
| Gated DeltaNet | 17.98 | 13.11 | 45.57 | 56.66 | 60.58 | 55.64 |
| CCQ-GLA | 16.92 | 11.39 | 47.82 | 59.14 | 59.69 | 56.10 |
| CCQ-Gated DeltaNet | 18.01 | 12.52 | 46.13 | 56.82 | 60.37 | 56.24 |
On synthetic needle-in-a-haystack retrieval (S-NIAH-1 passkey, S-NIAH-2 number, S-NIAH-3 uuid), CCQ prevents catastrophic degradation at longer contexts. At 1.3B, CCQ-GLA boosts S-NIAH-1 accuracy at 4K context from 47.8% to 96.8% (+49.0 points) and reaches 42.6% at 8K extrapolation where vanilla GLA collapses to 13.0%. On S-NIAH-3 (4K), CCQ-Gated DeltaNet achieves 46.6% compared to 6.0% for Gated DeltaNet, while CCQ-GLA achieves 64.8%. On the 14-task LongBench benchmark at 1.3B, CCQ-GLA achieves the highest average score of 15.77% among all recurrent baselines (outperforming GLA's 14.07% and Mamba2's 13.13%).
Ablation Study¶
Ablations conducted on 1.3B Gated DeltaNet checkpoints isolate the effect of covariance centring and the dynamic gating mechanism:
| Variant | WikiText PPL↓ | LAMBADA PPL↓ | 7-Task Avg Acc↑ | S1-4K Acc | S1-8K Acc | S2-4K Acc | S2-8K Acc | S3-4K Acc |
|---|---|---|---|---|---|---|---|---|
| Gated DeltaNet (no CCQ) | 17.98 | 13.11 | 55.64 | 100.0 | 99.8 | 83.4 | 56.6 | 6.0 |
| CCQ, centred \(\Sigma_t\) (learned gate) | 18.01 | 12.52 | 56.24 | 100.0 | 98.2 | 86.0 | 58.4 | 46.6 |
| CCQ, uncentred \(\bar{C}_t\) (learned gate) | 17.30 | 12.55 | 56.20 | 100.0 | 97.6 | 84.8 | 56.2 | 55.6 |
| CCQ, centred \(\Sigma_t\) (fixed \(\lambda=0.1\)) | 17.99 | 12.91 | 55.85 | 100.0 | 99.2 | 84.3 | 57.2 | 20.0 |
Key Findings¶
- Centring is beneficial but not strictly necessary: The uncentred second-moment variant \(\bar{C}_t\) achieves comparable downstream accuracy (56.20%) and even superior WikiText PPL (17.30) and S3-4K retrieval (55.6%), indicating that suppressing raw uncentred key energy directions captures the primary benefit of curvature cleaning.
- Learned dynamic gating is indispensable for hard retrieval: Fixing the gate to \(\lambda=0.1\) causes S-NIAH-3 (4K) accuracy to drop from 46.6% down to 20.0%. Empirical tracking reveals that the learned gate increases from an average of 0.29 on generic text to 0.40 on targeted retrieval queries, confirming its functional adaptivity.
- Empirical validation of geometric separation: Layer-wise probing on pretrained models (Qwen3-4B and Gated DeltaNet 500M) confirms that unique needle tokens consistently project less onto the top-16 covariance subspace than distractor tokens across 100% of layers, with a depth-averaged gap of \(\bar{\Delta}\mu = -0.111\) for Qwen3 and \(-0.209\) for Gated DeltaNet.
Highlights & Insights¶
- Reconnecting Softmax geometry with linear attention: Identifying the Hessian of the Softmax log-partition function as the running key covariance provides a principled, theoretically grounded explanation for why linear attention struggles with retrieval and how a second-order read operator resolves it.
- Orthogonal read-side modularity: Modifying exclusively the query transformation without altering the fast-weight memory tensor shapes or write updates allows CCQ to compose transparently with diverse linear attention and state-space architectures.
- Guaranteed spectral stability: The design ensures the cleaning operator's eigenvalues remain bounded strictly within \((0, 1]\), preventing vector explosion or signal cancellation across arbitrarily long sequences.
Limitations & Future Work¶
- Risk of suppressing frequent target patterns: The core premise assumes target information resides away from dominant covariance directions; if the target key itself is a common token or part of a repetitive template, CCQ might attenuate valid signal.
- Undecayed covariance mismatch: The running covariance \(\Sigma_t\) accumulates keys indefinitely without exponential decay, whereas underlying backbones like GLA or Gated DeltaNet actively forget or overwrite information. Developing decay-aligned covariance tracking remains an open avenue.
- Decoding kernel overhead: The unoptimized inference implementation introduces additional GPU kernel launches per layer, reducing token decode throughput by approximately 35% and increasing recurrent memory state by 50%.
Related Work & Insights¶
- vs Data-Dependent Gating (GLA, Mamba2): Gated architectures selectively decay memory entries along time or channel dimensions to regulate write accumulation, but their read remains an unconditioned dot product; CCQ directly complements gating by refining how memory is queried.
- vs Delta Rule Overwrites (DeltaNet, Gated DeltaNet): The delta rule uses least-squares online updates to overwrite specific keys, expanding storage capacity; CCQ operates downstream at read time to suppress residual background noise that survives overwrite updates.
- vs Kernel Feature Maps (Performer, Hedgehog): Feature map approaches warp key-query projections to mimic Softmax sharpness prior to writing, which often degrades stability at long contexts; CCQ maintains simple linear projections while applying exact second-order curvature corrections at retrieval.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates the first curvature-conditioned read-side correction derived directly from the Hessian of the Softmax normalizer.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across 500M and 1.3B scales, multiple backbones, synthetic S-NIAH, length extrapolation up to 20K, LongBench, and layer-wise geometric diagnostics.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptional mathematical rigor, lucid geometric intuitions, and transparent discussion of theoretical approximations.
- Value: ⭐⭐⭐⭐⭐ Offers a transformative and practical paradigm for resolving the retrieval bottlenecks of recurrent and linear-attention architectures.