It's Not RoPE that Creates Sinks: The Role of Self-Concentration and Value-Non-Mixing in Attention¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/kiya-raito/value-non-mixing
Area: LLM Reasoning
Keywords: Attention Sink, Massive Activations, Self-Attention Mechanism, Value-Non-Mixing, Causal Masking
TL;DR¶
By systematically decoupling BOS tokens, RoPE positional encodings, and causal attention constraints, this paper demonstrates that forced self-concentration from causal masking and the resulting "Value-non-mixing" state in early layers drive Attention Sinks and Massive Activations, decisively refuting the prevailing belief that RoPE is the culprit.
Background & Motivation¶
In large language models (LLMs), attention mechanisms frequently exhibit the "Attention Sink" (AS) phenomenon, where attention weights across numerous heads concentrate disproportionately on specific positions—most notably the sequence-initial token—even when that token carries little task-relevant semantic information. Simultaneously, these very positions in intermediate layers exhibit "Massive Activations" (MAs), characterized by hidden-state activation norms that reach hundreds or thousands of times the baseline magnitude of typical context tokens. This acute activation outlier problem severely destabilizes tensor distributions, posing a premier barrier to post-training low-bit quantization (such as INT8/INT4 PTQ) and dictating rigid eviction heuristics for streaming KV cache management.
Prior research has proposed multiple disconnected narratives to explain the emergence of these twin phenomena. Prominent accounts frame AS as an artifact of Softmax normalization constraints where attention heads require a "no-op" delimiter token to leave residual streams intact, interpret the first token as an emergent global key bias, attribute the directional pull to the structural asymmetry of causal attention graphs, or suspect Rotary Position Embedding (RoPE) due to its unique base-frequency rotation at coordinate zero. However, the sequence-initial position inherently multiplexes three distinct physical properties: the presence of a specialized token embedding (typically the BOS token), the initial positional coordinate under the positional encoding scheme, and the forced mathematical constraint of causal lower-triangular masking where the first token can only attend to itself (\(\alpha_{1,1}=1.0\)). Disentangling these confounded properties has remained an unresolved challenge in mechanistic interpretability.
This paper tackles this foundational question by establishing a rigorous intervention protocol that isolates token identity, RoPE positional encoding, and self-concentration into independent causal variables. Across diverse open-weight LLMs, the authors demonstrate that altering the first token's Key-side RoPE rotation index leaves Attention Sinks virtually undiminished, whereas forcing self-concentration at arbitrary non-initial positions reliably triggers both AS and MAs. Furthermore, synthesizing repeated-token sequences establishes that this numerical collapse occurs precisely when attention outputs pass unmixed Value vectors through the projection pipeline. Core idea: Attention Sinks and Massive Activations are not produced by RoPE or specific token embeddings, but are fundamentally driven by causal-mask-induced self-concentration and the resulting "Value-non-mixing" state in early layers, where attention outputs pass through unmixed single Value vectors instead of statistical convex combinations.
Method¶
Overall Architecture¶
The paper formulates an empirical mechanistic analysis framework structured around counterfactual interventions and unmixed state reproductions. In standard multi-head self-attention, token \(i\)'s output is computed as \(y_i = \sum_{j=1}^i \alpha_{i,j} v_j W_O\). To dissect the factors specific to position \(i=1\), the research design executes four progressive investigation phases: first, swapping the BOS token into an intermediate position (\(t=16\)) to evaluate token-semantic dependence versus positional dependence; second, intervening on the Key-side RoPE rotation matrix of the first token in BOS-free sequences to verify whether positional coordinate geometry drives attention attraction; third, enforcing artificial self-concentration (\(\alpha_{t,t}=1.0\)) at non-initial positions across layers to observe the de novo induction of AS and MAs; and fourth, formalizing the Value-non-mixing condition (\(v_j = v_i, \forall j \le i\)) and instantiating it naturally via repeated-token sequences to prove that the single-vector pass-through state alone is sufficient to generate activation outliers.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Causal Mask Sequence Input<br/>(w/o BOS Token)"] --> B["BOS Relocation Decoupling Protocol<br/>(Verifying token-type independence)"]
B --> C["RoPE Key-Index Intervention<br/>(Decoupling positional encoding geometry)"]
C --> D["Forced Self-Concentration Intervention<br/>(α_t,t = 1.0 induces AS and MAs)"]
D --> E["Value-Non-Mixing State Formalization<br/>(Single Value pass-through y_i = v_i W_O)"]
E --> F["Layer-Wise Propagation Profiling<br/>(Early layer locus and downstream suppression)"]
Key Designs¶
1. BOS Relocation Decoupling Protocol: Verifying Attention Sink Independence from Token Semantics
Most commercial and open-weight tokenizers automatically prepend a BOS token to every sequence, conflating its semantic role with the sequence-initial position. To orthogonally decouple token type from positional dynamics, the protocol swaps the BOS token with a standard context token at index \(t=16\). Attention concentration is quantitatively tracked using the metric \(\text{Sink}^\epsilon_j\), measuring the proportion of layers and heads whose average attention weight to position \(j\) exceeds threshold \(\epsilon\):
$$
\bar{\alpha}{l,h,j} = \frac{1}{T - j + 1} \sum > \epsilon)
$$
Across all evaluated models with }^T \alpha_{l,h,i,j}, \quad \text{Sink}^\epsilon_j = \frac{1}{L \cdot H} \sum_{l=1}^L \sum_{h=1}^H \mathbb{I}(\bar{\alpha}_{l,h,j\(\epsilon=0.3\), relocating BOS to position 16 induces an Attention Sink at position 16, yet an equally robust Attention Sink and Massive Activation signature spontaneously persists at position 1 occupied by an ordinary word (e.g., in Llama-2-7b, \(\text{Sink}^{0.3}_1\) remains at 0.9178). This confirms that the sequence-initial position inherently manufactures AS and MAs independently of BOS token features.
2. RoPE Key-Index Intervention: Disproving Positional Encoding Geometry as the Sink Driver
A prominent hypothesis posits that Rotary Position Embedding (RoPE) inherently biases attention toward the initial position due to the zero-angle coordinate rotation matrix \(R_{\Theta,1}\). To test this, the authors construct sequences without a BOS token and perform an intervention specifically on the Key vector of the first token: the rotation matrix \(R_{\Theta,1}\) is replaced with \(R_{\Theta,r}\), where \(r\) is uniformly sampled from \(\{2, \dots, T\}\):
$$
k_1^{\text{interv}} = x_1 W_K R_{\Theta,r}^\top
$$
If RoPE coordinate properties were the primary cause of Attention Sinks, randomizing its Key rotation would precipitously collapse \(\text{Sink}^\epsilon_1\). Counterfactually, across all five evaluated model architectures, the RoPE intervention produces almost negligible changes in \(\text{Sink}^\epsilon_1\) (e.g., Llama-2-7b shifts marginally from 0.9305 to 0.9273; Llama-3.2-3B shifts from 0.9218 to 0.9133). This directly refutes the conjecture that RoPE creates attention sinks.
3. Forced Self-Concentration Intervention: Inducing Sinks and Massive Activations at Arbitrary Positions
Under causal masking, the initial token is mathematically constrained to allocate its entire attention budget to itself (\(\alpha_{1,1} = 1.0\)). To establish whether this forced self-concentration is the causal engine of AS and MAs, the protocol intervenes on the attention weight matrix at a non-initial token position \(t=16\) in BOS-free sequences, overriding the computed softmax distribution with complete self-concentration: \(\alpha_{t,t} = 1.0\) and \(\alpha_{t,j \neq t} = 0.0\). Across all models, this localized intervention at position 16 induces an intense emergent Attention Sink from subsequent tokens, causing \(\text{Sink}^{0.3}_{16}\) to surge from near zero (0.0000–0.0073) up to 0.4703–0.8397, accompanied by the eruption of layer-wise hidden-state norm spikes mirroring the natural first-token MAs.
4. Value-Non-Mixing State Formalization: Uncovering the Latent Dynamical Mechanism
The authors hypothesize that self-concentration does not operate via attention routing alone, but because it forces the attention output to bypass convex value averaging. For typical context tokens (\(i > 1\)), attention outputs represent a smooth convex combination of diverse Value vectors. Under self-concentration, the output reduces to an unmixed single-vector pass-through:
$$
v_j = v_i, \quad \forall j \le i \text{ s.t. } \alpha_{i,j} > 0 \implies y_i = v_i W_O
$$
To validate this hypothesis without manually overwriting attention weights, the experiment constructs synthetic sequences of repeated identical tokens (e.g., "city city city...") devoid of BOS tokens. Because every context token shares an identical Value representation, attention outputs remain unmixed pass-throughs even when attention weights spread diffusely across multiple keys. By inserting a distinct token (e.g., "table") at index 1, 2, or 3, the range of Value-non-mixing is precisely modulated. Results confirm that in uniform sequences, MAs erupt across all token positions despite distributed attention; when a distinct token is inserted, MAs persist exclusively up to the transition boundary and sharply diminish once value mixing commences.
Benchmark & Evaluation Protocol¶
The empirical evaluation investigates five prominent open-weight LLMs spanning distinct architectural designs: Llama-2-7b-hf (standard MHA, full RoPE, RMSNorm), Llama-3.2-3B (GQA, full RoPE, RMSNorm), Mistral-7B-v0.3 (GQA, full RoPE, RMSNorm), Qwen2-7B (GQA, full RoPE, RMSNorm with QKV biases), and pythia-1b (MHA, 25% partial RoPE, standard LayerNorm with QKV biases). Primary evaluations use WikiText-2 (\(N=100\) sequences, \(T=64\)). Robustness sweeps assess sequence lengths \(T \in \{32, 64, 128, 256\}\), datasets including GSM8K and SlimPajama, and thresholds \(\epsilon \in \{0.2, 0.3, 0.4, 0.5\}\). Experiments were conducted on an NVIDIA RTX 6000 Ada Generation GPU utilizing PyTorch 2.9.0 and Hugging Face Transformers 4.57.1.
Key Experimental Results¶
Main Results¶
The primary benchmark contrasts the vanilla baseline against the Key-side RoPE intervention at position 1 and the forced self-concentration intervention at position 16 on WikiText (\(T=64\), BOS-free sequences, \(\epsilon=0.3\)).
| Model | Architecture Specifics | Vanilla Pos 1 \(\text{Sink}^{0.3}_1\) | RoPE Interv Pos 1 \(\text{Sink}^{0.3}_1\) | Vanilla Pos 16 \(\text{Sink}^{0.3}_{16}\) | Self-Conc Interv Pos 16 \(\text{Sink}^{0.3}_{16}\) |
|---|---|---|---|---|---|
| Llama-2-7b-hf | MHA / RMSNorm / 100% RoPE | 0.9305 | 0.9273 | 0.0073 | 0.8397 |
| Llama-3.2-3B | GQA / RMSNorm / 100% RoPE | 0.9218 | 0.9133 | 0.0012 | 0.7424 |
| Mistral-7B-v0.3 | GQA / RMSNorm / 100% RoPE | 0.0343 | 0.1525 | 0.0373 | 0.1509 |
| Qwen2-7B | GQA / RMSNorm / 100% RoPE + Bias | 0.8173 | 0.7931 | 0.0000 | 0.5085 |
| pythia-1b | MHA / LayerNorm / 25% RoPE + Bias | 0.7337 | 0.7547 | 0.0000 | 0.4703 |
Ablation Study¶
To isolate which depth coordinates mediate this phenomenon, the self-concentration intervention was restricted to 3-layer subsets (Early: Layers 1–3, Middle: network midpoint, Late: preceding final layer). In parallel, the repeated-token regime was evaluated across distinct token insertion patterns (Pattern 0: Uniform, Pattern 1: distinct at index 1, Pattern 2: distinct at index 2) with \(T=64, \epsilon=0.3\).
| Model | Self-Conc: All Layers | Self-Conc: Early (L1–3) | Self-Conc: Middle | Self-Conc: Late | Repeated: Pattern 0 (Uniform) | Repeated: Pattern 1 (Pos 1 Distinct) | Repeated: Pattern 2 (Pos 2 Distinct) |
|---|---|---|---|---|---|---|---|
| Llama-2-7b-hf | 0.8397 | 0.7724 | 0.0074 | 0.0074 | 0.0000 | 0.7754 | 0.7955 |
| Llama-3.2-3B | 0.7424 | 0.6670 | 0.0013 | 0.0012 | 0.0000 | 0.7740 | 0.8042 |
| Mistral-7B-v0.3 | 0.1509 | 0.1396 | 0.0374 | 0.0374 | 0.0000 | 0.1548 | 0.1568 |
| Qwen2-7B | 0.5085 | 0.4737 | 0.0000 | 0.0000 | 0.0000 | 0.5482 | 0.5404 |
| pythia-1b | 0.4703 | 0.4296 | 0.0000 | 0.0000 | 0.0000 | 0.0055 | 0.0000 |
Key Findings¶
- RoPE invariance across families: Randomizing Key-side RoPE phase angles on the initial token produces negligible perturbation on \(\text{Sink}^{0.3}_1\) across all architectures, establishing that RoPE does not dictate sink formation.
- Early-layer dominance: Restricting self-concentration to Layers 1–3 reproduces 90%–93% of the all-layer intervention strength across all models, whereas intervening solely in middle or late layers leaves \(\text{Sink}^{0.3}_{16}\) at baseline levels. MAs and AS originate strictly as early-layer bottom-up cascades.
- Value-non-mixing without attention intervention: In uniform repeated sequences where attention weights disperse across identical keys (\(\text{Sink}^{0.3}_1 = 0.0000\) in Pattern 0), MAs still erupt everywhere. Once a distinct token breaks Value homogeneity, AS and MAs align strictly with the unmixed prefix positions.
Highlights & Insights¶
- Decisive orthogonal experimental isolation: The paper elegantly untangles the conflated trilogy of BOS token semantics, RoPE phase coordinate zero, and causal mask self-concentration through clean counterfactual interventions.
- Microscopic dynamical explanation via Value-non-mixing: Instead of treating Attention Sinks as an abstract "no-op trash can" or static Key bias, the work identifies the statistical mechanics of attention output: unmixed single Value vectors passing through \(W_O\) accumulate destructive residual norms.
- Actionable implications for outlier-free architectures: Highlights that enforcing value mixing in early layers or modifying causal initialization could prevent Massive Activations natively during pretraining, unlocking seamless INT4/INT8 quantization.
Limitations & Future Work¶
- Architectural boundaries constrained to dense causal models: All evaluated models belong to dense decoder-only Transformers; verifying whether Value-non-mixing manifests similarly in Mixture-of-Experts (MoE) or non-autoregressive bidirectional architectures remains open.
- Input sensitivity in weak-responder models: Mistral-7B-v0.3 and pythia-1b exhibit significantly lower baseline sink metrics and higher sensitivity to semantic token pairings, requiring further work to trace head-specific projection circuits.
- Metric sensitivity to length: The thresholded \(\text{Sink}^\epsilon_j\) metric remains sensitive to sequence length \(T\) due to average weight scaling (\(1/i\)); developing a continuous, geometry-aware "Value Mixing Index" is an important future direction.
Related Work & Insights¶
- vs Gu et al. (ICLR 2025) & Sun et al. (COLM 2024): Gu et al. established the Key-bias interpretation and Sun et al. documented MAs; this paper links the two into a single causal chain, demonstrating that both are downstream consequences of early-layer Value-non-mixing initiated by causal self-concentration.
- vs Bondarenko et al. (NeurIPS 2023) "Quantizable Transformers": Bondarenko et al. argued that attention heads require extreme softmax logits to execute no-ops on delimiter tokens; this paper shows that causal masking inherently forces a no-op on the first token, triggering activation explosions without needing explicit delimiter tokens.
- vs Barbero et al. (COLM 2025) "Anti-Overmixing Theory": Barbero et al. frame AS as a protective mechanism preventing contextual over-mixing across layers; this work examines the dual problem from the intra-layer Value composition perspective, providing a mechanistic counterpart to their representation-level theory.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Systematically refutes the RoPE sink hypothesis and introduces the Value-non-mixing mechanistic framework.]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Exemplary counterfactual intervention design across 5 model families, multiple sequence lengths, datasets, and layer ablations.]
- Writing Quality: ⭐⭐⭐⭐⭐ [Crystal-clear logical progression, immaculate empirical rigor, and precise formalization.]
- Value: ⭐⭐⭐⭐⭐ [Crucial foundational insights for transformer interpretability, LLM efficiency, and outlier-free quantization design.]