Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPE¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/aflah02/Partial_RoPE_Analysis
Area: LLM Reasoning
Keywords: Rotary Positional Embedding (RoPE), Partial RoPE, Training Convergence, Memory Optimization, Long-Context Reasoning
TL;DR¶
This paper delivers the first systematic empirical pretraining study of partial Rotary Position Embeddings (RoPE) across 1B and 8B scales over 100B+ tokens, revealing that rotating merely 10% of per-head hidden dimensions matches the convergence speed, final loss, and downstream reasoning accuracy of 100% full RoPE while cutting RoPE cache memory footprint by up to 10× in extreme long-context regimes.
Background & Motivation¶
Rotary Positional Embedding (RoPE) has become the de facto positional encoding standard across modern autoregressive causal language models, favored for its elegant mathematical derivation that injects relative distance directly into query-key dot products, its superior length extrapolation behavior, and its seamless interoperability with diverse attention variants. However, looking across the landscape of both commercial and open-weight model architectures reveals massive inconsistency and arbitrary rule-of-thumb heuristics regarding the proportion of hidden dimensions rotated. Early models such as GPT-NeoX, GPT-J, and Pythia adopted a 25% partial application, whereas the original LLaMA and Qwen series enforced 100% full rotation across all head channels. Interestingly, the latest Qwen3-Next retreated to a 25% partial configuration aiming to enhance extrapolation, while Nemotron-4-340B and GLM-4.5 adopted 50%, and Phi-2 chose 40%. This divergent design space exposes a long-standing blind spot in the literature: what is the minimum fraction of hidden dimensions that actually requires rotation to fully capture relative positional reasoning without wasting memory and compute?
This fundamental architectural choice escalates into a critical memory bottleneck as frontier language models push their context windows toward million-token and even 100-million-token frontiers. At conventional context lengths of 2K or 4K tokens, the sine and cosine cache precomputed and held in GPU VRAM represents a negligible overhead. But because RoPE cache scales strictly linearly with context length, maximum sequence windows in extreme contexts demand tens to hundreds of gigabytes of VRAM per GPU. In multi-GPU distributed data-parallel paradigms, either replicating the tables across devices squanders precious memory bandwidth or sharding them induces prohibitive cross-node communication overhead. Prior literature offered no systematic pretraining guidance on this dilemma: the only documented empirical basis for partial RoPE stemmed from an informal 124M-parameter GPT2-Small model trained on a trivial budget of 82M tokens.
To resolve this empirical gap, the authors perform extensive pretraining from scratch across varying model sizes (1B and 8B parameters), network topologies (sequential versus parallel transformer blocks), sequence lengths (1024 to 8192 tokens), and pretraining corpora qualities (FineWeb versus FineWeb-Edu) over 100B tokens each. Core idea: by systematically examining the minimum positional subspace within attention heads across diverse training scales, the study demonstrates that applying RoPE to only ~10% of hidden dimensions fully matches the convergence dynamics and downstream reasoning capability of full RoPE, while exposing and remedying the fatal loss spikes inherent to NoPE.
Method¶
Overall Architecture¶
The framework systematically investigates the parameter subspace of Rotary Positional Embeddings by varying the fraction of rotated channels in the query and key representations during transformer pretraining from scratch. The methodology couples head-dimension fraction scheduling with a quantitative linear memory scaling formulation and an attention normalization stability protocol to guarantee rock-solid convergence across architectures and extreme context lengths.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Tokens and Attention Query/Key Projections"] --> B["Hidden Dimension Allocation<br/>0% NoPE to 100% Full RoPE"]
B --> C["RoPE Cache Memory Scaling Model<br/>Linear VRAM Footprint vs Sequence Length"]
C --> D["QK-Norm Attention Stabilization<br/>Eliminating Gradient and Loss Spikes"]
D --> E["Downstream Evaluation and Convergence Analysis<br/>Validating 10% Optimal Trade-off Frontier"]
Key Designs¶
1. Hidden Dimension Allocation: Establishing the Minimal Rotated Subspace Boundary
To rigorously answer how many dimensions within each attention head require rotation, the authors systematically formulate a dimension spectrum comprising 0% (No Positional Encoding, or NoPE), minimal feasibility (strictly 2 channels per head, corresponding to ~1% in Pythia-1B with head dimension 256 and ~4% in LLaMA-1B with head dimension 64), 10% (or 10.2%), 25%, 50%, 75%, and 100% (full RoPE). Because RoPE pairs adjacent channels along a 2D rotation plane, the number of rotated dimensions must be an even integer; percentage targets are thus systematically rounded to the nearest valid even number. For instance, in Pythia-1B (head dimension 256), a nominal 10% produces 25.6 dimensions, which is adjusted to 26 dimensions (10.2%) to preserve pairwise rotation. In LLaMA-1B (head dimension 64), 10% maps to 6 channels, while in LLaMA-8B (head dimension 128), it corresponds to 12 channels. The unrotated dimensions bypass rotation entirely and participate in standard dot-product attention without positional modulation, thereby cleanly decoupling the representation space into a compact relative-position coordinate subspace and an invariant semantic content subspace.
2. RoPE Cache Memory Scaling Model: Decoupling VRAM Footprint in Long-Context Windows
To address the memory explosion incurred by tabular positional lookup tables in extreme-context regimes, the authors model and quantify the exact VRAM consumption of the precomputed rotary sine and cosine cache. Because computing trigonometric projections on the fly inside kernel fusions degrades latency, frameworks precompute and retain the entire positional grid upon model initialization. The total memory footprint in bytes adheres to the linear equation:
where \(L_{\text{max}}\) denotes the configured maximum sequence length, \(D_{\text{rot}}\) represents the number of active rotated channels per attention head, and \(P_{\text{bytes}}\) represents the precision storage size (mandating 4 bytes for FP32, as bfloat16 causes catastrophic underflow and divergence in long contexts). As context lengths push from standard 2K to 1M, 10M, or 100M tokens, standard 100% RoPE with \(D_{\text{rot}} = D_{\text{head}} = 256\) causes the cache footprint to scale linearly from a few megabytes to 128 GB of VRAM per device, easily triggering out-of-memory crashes. By dialing \(D_{\text{rot}}\) down to 10% of the head dimension, the required cache memory scales down by an order of magnitude (achieving a 10× memory reduction), drastically lowering memory pressure on multi-GPU nodes and memory-constrained edge deployment.
3. QK-Norm Attention Stabilization: Eliminating Fatal Loss Spikes and Divergence
To address the fatal, unrecoverable loss spikes and numerical instability that plague NoPE (0% RoPE) and minimal-channel setups at large training scales, long sequences, and parallel transformer architectures, the authors integrate Query-Key Normalization (QK-Norm). Unlike conventional framework implementations that compute normalization across both attention heads and head dimensions simultaneously, this approach follows the canonical per-head formulation by normalizing query and key states strictly across the hidden dimension of each attention head. Empirical diagnosis proves that when models scale past tens of billions of tokens, attention matrices lacking strong positional constraints suffer from severe logit drift and exploding gradient norms. Per-head QK-Norm establishes a rigid magnitude clamp on query-key dot products, completely suppressing loss spikes across disparate random seeds and learning rate perturbations, and enabling stable convergence even under pure NoPE configurations.
Loss & Training¶
All pretraining runs optimize standard causal language modeling cross-entropy loss:
Optimization utilizes AdamW with an initial learning rate of \(4 \times 10^{-4}\), a 5% linear warmup schedule, and cosine learning rate decay down to 10% of the initial value, without dropout. Mixed precision training is executed with bfloat16 for forward and backward passes, maintaining FP32 for gradient accumulation and inter-GPU all-reduce primitives. Effective global batch size is held constant at 4096 sequences (totaling ~8.38M tokens per optimizer step at 2048 sequence length) across 32 NVIDIA H200 GPUs partitioned into 4 nodes. Micro-batch sizes are adjusted proportionally when sweeping sequence lengths from 1024 (micro-batch 128) to 4096 and 8192 (micro-batch 16) to preserve identical optimization trajectories. The 8B models are trained across 36 transformer layers for approximately 120 hours per run within the GPT-NeoX distributed framework.
Key Experimental Results¶
Main Results¶
Models are comprehensively evaluated under zero-shot settings using EleutherAI's LM Evaluation Harness, spanning reasoning and common-sense comprehension benchmarks including ARC-Easy, ARC-Challenge, LogiQA, PIQA, SciQ, PubMedQA, WinoGrande, WSC, and LAMBADA.
| Model & Corpus Setup | RoPE Config | ARC-c (Acc) | ARC-e (Acc) | PIQA (Acc) | SciQ (Acc) | WinoGrande (Acc) | LAMBADA Std (PPL ↓) |
|---|---|---|---|---|---|---|---|
| Llama-3.2-1B x FW x 2048 | 0% (NoPE) | 0.27 (0.01) | 0.47 (0.01) | 0.71 (0.01) | 0.71 (0.01) | 0.52 (0.01) | 37.15 (1.38) |
| Llama-3.2-1B x FW x 2048 | 4% (2 channels) | 0.28 (0.01) | 0.46 (0.01) | 0.71 (0.01) | 0.69 (0.01) | 0.51 (0.01) | 36.95 (1.36) |
| Llama-3.2-1B x FW x 2048 | 10% (6 dims) | 0.26 (0.01) | 0.48 (0.01) | 0.73 (0.01) | 0.68 (0.01) | 0.52 (0.01) | 26.91 (0.94) |
| Llama-3.2-1B x FW x 2048 | 25% (16 dims) | 0.27 (0.01) | 0.47 (0.01) | 0.72 (0.01) | 0.68 (0.01) | 0.54 (0.01) | 29.79 (1.09) |
| Llama-3.2-1B x FW x 2048 | 50% (32 dims) | 0.27 (0.01) | 0.48 (0.01) | 0.72 (0.01) | 0.71 (0.01) | 0.53 (0.01) | 27.11 (0.97) |
| Llama-3.2-1B x FW x 2048 | 100% (Full RoPE) | 0.26 (0.01) | 0.49 (0.01) | 0.73 (0.01) | 0.69 (0.01) | 0.53 (0.01) | 27.73 (0.98) |
| Llama-3.1-8B x FW x 2048 | 0% (NoPE) | 0.30 (0.01) | 0.57 (0.01) | 0.77 (0.01) | 0.80 (0.01) | 0.57 (0.01) | 11.01 (0.32) |
| Llama-3.1-8B x FW x 2048 | 10% (12 dims) | 0.32 (0.01) | 0.60 (0.01) | 0.77 (0.01) | 0.81 (0.01) | 0.59 (0.01) | 10.08 (0.29) |
| Llama-3.1-8B x FW x 2048 | 25% (32 dims) | 0.33 (0.01) | 0.58 (0.01) | 0.77 (0.01) | 0.82 (0.01) | 0.59 (0.01) | 9.66 (0.27) |
| Llama-3.1-8B x FW x 2048 | 100% (Full RoPE) | 0.33 (0.01) | 0.59 (0.01) | 0.77 (0.01) | 0.81 (0.01) | 0.59 (0.01) | 9.03 (0.25) |
Ablation Study¶
The ablation investigates the interaction of positional encoding fractions, long sequence lengths, parallel attention topology, and QK-Norm stabilization:
| Architecture & Context Setting | Experimental Config | LAMBADA OpenAI PPL ↓ | LAMBADA Std PPL ↓ | Training Outcome / Empirical Behavior |
|---|---|---|---|---|
| Llama-3.2-1B (Seq Len 8192) | 0% RoPE (NoPE) | 79.69 (3.37) | 396.00 (20.03) | Severe unrecoverable loss spike, acute PPL collapse |
| Llama-3.2-1B (Seq Len 8192) | 0% RoPE + QK-Norm | 19.09 (0.62) | 47.90 (1.85) | Loss spike eliminated; PPL improves by 87.9% |
| Llama-3.2-1B (Seq Len 8192) | 4% RoPE (2 channels) | 18.83 (0.61) | 49.06 (1.91) | Converges stably but stagnates in higher loss band |
| Llama-3.2-1B (Seq Len 8192) | 10% RoPE | 14.39 (0.45) | 32.69 (1.20) | Achieves primary convergence band, matches 100% |
| Llama-3.2-1B (Seq Len 8192) | 100% RoPE | 13.55 (0.42) | 34.64 (1.31) | Upper performance benchmark |
| Pythia-1B (Parallel Block) | 0% RoPE (NoPE) | 340,933.20 | 3,717,492.05 | Catastrophic divergence; model completely explodes |
| Pythia-1B (Parallel Block) | 0% RoPE + QK-Norm | 51.74 (2.04) | 136.88 (5.85) | Rescues training from collapse; stable convergence |
| Pythia-1B (Parallel Block) | 1% RoPE (2 channels) | 43.13 (1.69) | 144.46 (6.33) | Converges smoothly but in elevated loss band |
| Pythia-1B (Parallel Block) | 10% RoPE (26 dims) | 33.49 (1.27) | 115.78 (5.06) | Matches or outperforms 100% RoPE (PPL 122.90) |
| Pythia-1B (Parallel Block) | 100% RoPE (256 dims) | 32.88 (1.24) | 122.90 (5.35) | Full rotation baseline |
Key Findings¶
- Two Distinct Convergence Bands: Across all tested architectures (sequential vs. parallel), model sizes (1B vs. 8B), sequence lengths (1024 to 8192), and corpora (FineWeb vs. FineWeb-Edu), training loss trajectories cluster cleanly into two discrete groups. Models with NoPE or minimal 2-channel RoPE converge to an elevated loss plateau. In contrast, models with 10% or more RoPE immediately transition into a lower, optimal convergence band, overlapping indistinguishably with 25%, 50%, 75%, and 100% RoPE.
- Perplexity Is Significantly More Sensitive Than MCQ: Standard multiple-choice reasoning benchmarks exhibit minimal variance across RoPE fractions (even NoPE achieves competitive scores on ARC and PIQA). However, LAMBADA language perplexity drops precipitously when crossing the threshold from <10% to \(\ge 10\%\) RoPE (e.g., from 37.15 down to 26.91 on Llama-1B-2048), indicating that 10% partial RoPE fully unlocks nuanced token-level autoregressive predictive capacity.
- Vulnerability of NoPE at Scale and the Stabilizing Role of QK-Norm: While prior synthetic studies claimed NoPE generalized well, this large-scale study reveals that NoPE suffers from catastrophic training divergence on parallel transformer architectures and experiences sudden loss spikes at long sequence lengths (8192). Incorporating 10% RoPE or per-head QK-Norm entirely prevents these instabilities.
Highlights & Insights¶
- The 10% Golden Frontier: The paper dismantles the prevailing assumption that all hidden dimensions must undergo rotation, proving that rotating just 10% of hidden channels provides sufficient relative positional signal, freeing up 90% of RoPE cache VRAM in ultra-long contexts.
- Architectural Stability Insights: Parallel transformer blocks (where MLP and attention execute concurrently) exhibit severe numerical sensitivity in the absence of positional encodings. QK-Norm serves as a universal stabilizer, decoupling architectural choices from gradient explosion.
- Empirical Grounding for Next-Generation LLMs: Frontier designs such as Qwen3-Next have shifted toward 25% partial RoPE for inference efficiency; this paper supplies the definitive empirical justification, demonstrating that model architects can push this reduction down to 10% without performance penalty.
Limitations & Future Work¶
- Compute Budget Bounds Exploration on Ultra-Large Scales: Due to the immense cost of pretraining from scratch, the study explores up to 8B parameters and does not directly pretrain 70B+ or MoE architectures, leaving scaling laws at hundreds of billions of parameters to future work.
- Long-Context Extrapolation Benchmarks: Evaluations primarily assess in-distribution contexts up to 8K tokens and standard zero-shot reasoning; extensive needle-in-a-haystack or multi-hop retrieval over 1M+ extrapolated contexts remains to be deeply explored.
- Heterogeneous Dimension Allocation: Future investigations could explore allocating different RoPE fractions across different layers (e.g., shallower vs. deeper blocks) or individual attention heads.
Related Work & Insights¶
- vs GPT-NeoX (Black et al., 2022) & Pythia (Biderman et al., 2023): Prior work adopted 25% partial RoPE purely based on informal tests on a 124M-parameter toy model trained over 82M tokens. This paper establishes the first rigorous empirical foundation across 1B/8B models and 100B+ tokens, proving that 10% is the true minimal boundary.
- vs NoPE (Kazemnejad et al., 2023): Whereas previous literature suggested NoPE suffices for length generalization on synthetic tasks, this work proves that NoPE exhibits fatal loss spikes and training instability under large-scale, long-context pretraining unless paired with QK-Norm.
- vs Qwen3-Next (Qwen Team, 2025): Validates industry moves toward fractional rotation and demonstrates that the 25% fraction used in production models can be further compressed to 10% with zero loss degradation.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ First systematic and large-scale pretraining investigation into the minimal subspace requirements of RoPE.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous coverage across 1B/8B scales, two block topologies, four context lengths, and two datasets.
- Writing Quality: ⭐⭐⭐⭐⭐ Highly coherent, lucid exposition, self-consistent data, and immaculate structure.
- Value: ⭐⭐⭐⭐⭐ Outstanding engineering and architectural guidance for long-context LLMs, edge inference, and cache optimization.