Muon^2: Boosting Muon via Adaptive Second-Moment Preconditioning¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: Information Retrieval
Keywords: Optimization Algorithms, Foundation Model Pre-training, Orthogonalized Updates, Second-Moment Preconditioning, Newton-Schulz Iteration
TL;DR¶
Muon^2 introduces Adam-style adaptive second-moment element-wise preconditioning prior to the Newton-Schulz polar orthogonalization step in Muon (together with a memory-efficient factorized variant Muon^2-F), eliminating singular value dead zones to reduce Newton-Schulz iterations by 40% while speeding up pre-training and boosting downstream task performance.
Background & Motivation¶
The relentless scaling of modern language models in both parameter capacity and pre-training corpora has imposed enormous computational burdens and distributed communication bottlenecks on pre-training infrastructure. Coordinate-wise adaptive first-order optimizers such as Adam and AdamW remain the de facto industry standard due to their empirical robustness and ease of tuning; nevertheless, they flatten multi-dimensional parameter tensors into unstructured coordinate arrays, overlooking the underlying two-dimensional matrix geometry of neural network weights. To surpass coordinate-wise diagonal scaling without paying the prohibitive cost of full Kronecker-factored second-order curvature estimates, matrix-structured optimizers such as Muon have emerged. Muon directly approximates the polar decomposition of the momentum matrix via iterative orthogonalization, effectively mitigating gradient rank collapse and accelerating training dynamics in large architectures.
However, the orthogonalization quality of Muon strictly hinges on the number of Newton-Schulz (NS) polynomial iterations performed per update. Standard Muon requires five iterations to approximate the polar factor, with each step executing high-order matrix polynomial multiplications that induce non-trivial local matrix multiplication (GEMM) workloads and distributed latency bottlenecks. Forcing the iteration count down to three leads to severe degradation in update direction alignment and catastrophic increases in validation perplexity. Prior variants either modify update step-size adaptation (e.g., NorMuon, AdaMuon) or explore alternative polynomial coefficients (e.g., PolarExpress, Turbo-Muon), yet none address the fundamental root cause: the severely ill-conditioned singular value spectrum of the raw input momentum matrix.
A rigorous empirical inspection reveals that during the early and intermediate training regimes, the momentum matrix's singular values span several orders of magnitude, with almost half of them concentrated in an near-zero "dead zone" that fails to converge under low-order polynomial mappings. If the momentum spectrum can be element-wise conditioned prior to orthogonalization, the convergence burden on the Newton-Schulz iteration can be drastically alleviated. Drawing on random matrix theory and empirical spectral analysis, this paper proposes integrating adaptive second-moment scaling as an upfront preconditioner for polar orthogonalization. Core idea: applying coordinate-wise second-moment preconditioning prior to Newton-Schulz orthogonalization reshapes the momentum singular value spectrum and eliminates the dead zone, enabling 40% fewer iteration steps to achieve higher directional fidelity and substantially faster pre-training convergence.
Method¶
Overall Architecture¶
Muon^2 retains the structural benefits of matrix orthogonal updates while systematically repairing the ill-conditioned input spectrum. The complete execution pipeline consists of four sequential stages: first, first-order momentum and uncentered second-moment statistics are accumulated from current gradients; second, the momentum is preconditioned via element-wise division by the square root of the second moment, shifting the singular values away from the dead zone into a compact, convergent spectrum; third, the preconditioned matrix enters a streamlined Newton-Schulz polynomial loop (requiring only 3 iterations) to compute a high-fidelity polar approximation; fourth, parameters are updated along the orthogonal direction scaled by an aspect-ratio-aware learning rate. To alleviate the GPU memory overhead of storing a full second-moment tensor, the architecture introduces a low-rank outer-product factorization variant termed Muon^2-F.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Gradient and Momentum Accumulation<br/>Compute Gt and first-order Mt"] --> B["Adaptive Second-Moment Preconditioning<br/>Element-wise scaling to reshape singular spectrum"]
B --> C["Directional Alignment and Spectral Guarantees<br/>Eliminate dead zone and ensure cosine similarity"]
C --> D["Accelerated Newton-Schulz Iteration<br/>K=3 polynomial matrix polar approximation"]
D --> E["Factorized Memory-Efficient Variant Muon^2-F<br/>Row-column outer product optimizer states"]
E --> F["Orthogonal Parameter Matrix Update<br/>Aspect-ratio-scaled update step to Wt+1"]
Key Designs¶
1. Adaptive Second-Moment Preconditioning: Eliminating Singular Spectrum Collapse Standard Muon directly feeds the first-order momentum matrix \(M_t = \beta_1 M_{t-1} + (1 - \beta_1) G_t\) into the Newton-Schulz loop. Because gradient variances across parameter coordinates are highly heterogeneous, the unconditioned momentum exhibits a large condition number, causing a significant fraction of singular values to fall into the dead zone (\(10^{-4} \sim 10^{-3}\)). Muon^2 maintains an exponential moving average of squared gradients \(V_t = \beta_2 V_{t-1} + (1 - \beta_2)(G_t \odot G_t)\) and applies element-wise preconditioning: $\(\widetilde{M}_t = M_t \oslash (\sqrt{V_t} + \epsilon I)\)$ This operation standardizes coordinate energy scales across the matrix. Under random matrix theory (the Marchenko-Pastur law), once coordinate-wise scales \(s_{ij}\) are divided out, the empirical singular value distribution of \(\widetilde{M}_t\) converges almost surely to the compact support \([1 - \sqrt{\lambda}, 1 + \sqrt{\lambda}]\) (where \(\lambda = m/n < 1\) is the layer aspect ratio), completely erasing the lower dead zone and providing a pristine spectrum for polynomial mapping.
2. Directional Alignment and Spectral Guarantees: Reconciling True Update Directions Prior studies evaluate polar approximation using exact orthogonality error \(\|Q^\top Q - I\|\). However, this metric is overly sensitive to any global scalar scaling factor \(c \in (0, 1]\): even when \(Q = c U V^\top\) is perfectly aligned with the exact polar factor \(Q^\star = U V^\top\), the orthogonality error evaluates to \(|c^2 - 1| \cdot \|I\|\). In deep learning optimization, scalar constants are absorbed into the tunable learning rate \(\eta\). Muon^2 therefore formalizes directional cosine similarity as the proper evaluation criterion: $\(\cos(Q_{\text{NS}}, Q^\star) = \frac{\sum_i \phi_K(\sigma_i)}{\sqrt{n} \sqrt{\sum_i \phi_K(\sigma_i)^2}}\)$ Under this scale-invariant formulation, the convergence of polynomial mapping \(\phi(x) = ax + bx^3 + cx^5\) (\(a=3.4445, b=-4.7750, c=2.0315\)) is governed purely by relative spectral concentration. Because preconditioning raises the effective rank from 48.9 to 142.5 and cuts the dead-zone fraction from 26% to 6%, Muon^2 with only 3 steps attains a cosine similarity of 0.916 (exceeding standard Muon's 5-step score of 0.902~0.931), reaching 0.975 at 5 steps.
3. Accelerated Newton-Schulz Iteration: Superior Optimization at Reduced Cost With a well-conditioned input spectrum guaranteed, the preconditioned matrix \(\widetilde{M}_t\) is processed through \(K=3\) Newton-Schulz iterations: $\(O_t = \text{NewtonSchulz}(\widetilde{M}_t, K=3)\)$ Reducing \(K\) from 5 to 3 eliminates 40% of the costly matrix-matrix multiplication FLOPs and latency-inducing synchronization barriers per step. The parameters are then updated using the dimension-aware scaling factor: $\(W_{t+1} = W_t - \eta \sqrt{\frac{m}{n}} O_t\)$ Empirically, 3-step Muon^2 yields lower pre-training perplexity than 5-step Muon across all scales, proving that preconditioning the spectrum is fundamentally superior to executing extra iterations on an ill-conditioned input.
4. Factorized Memory-Efficient Variant Muon^2-F: Outer-Product State Compression To eliminate the GPU memory overhead of maintaining the full \(n \times m\) matrix \(V_t\), Muon^2-F adopts an Adafactor-style row- and column-wise factorized second-moment accumulator: $\(r_t = \beta_2 r_{t-1} + (1 - \beta_2) \text{sum}_{\text{col}}(G_t \odot G_t), \quad c_t = \beta_2 c_{t-1} + (1 - \beta_2) \text{sum}_{\text{row}}(G_t \odot G_t)\)$ $\(\widehat{V}_t = \frac{r_t c_t^\top}{\text{sum}(r_t)}\)$ This reduces the auxiliary optimizer state memory footprint from \(\mathcal{O}(nm)\) to \(\mathcal{O}(n + m)\). Random matrix analysis demonstrates that as long as coordinate variances admit row-column separable structure, the effective rank and non-zero singular value margins are fully preserved. On a 7B model, Muon^2-F adds only 0.24 GB over standard Muon while retaining over 95% of the perplexity gains and the entire 40% NS reduction of full Muon^2.
Loss & Training¶
The optimizer is evaluated using standard autoregressive causal language modeling cross-entropy loss: $\(\mathcal{L}(W) = -\frac{1}{T} \sum_{t=1}^T \log P(w_t \mid w_{<t})\)$ Decoupled weight decay is applied directly to weights. Hyperparameters default to second-moment decay \(\beta_2 = 0.95\), numerical stabilizer \(\epsilon = 10^{-8}\), and first-order momentum \(\beta_1 = 0.95\). Non-2D parameters (embedding tables and 1D normalization vectors) are updated via standard AdamW.
Key Experimental Results¶
Main Results¶
Muon^2 was evaluated across GPT models (Small 124M, Base 362M, Large 774M), LLaMA models (60M, 350M, 1B, 13B), and a sparse Mixture-of-Experts architecture (MoE-7B-A1B).
| Model & Scale | Tokens | Metric | Muon (Ns=5) | Muon^2 (Ns=3) | Muon^2 (Ns=5) | Gain (Ns=5 vs Muon) |
|---|---|---|---|---|---|---|
| GPT-Small (124M) | 3.0B | Val PPL (โ) | 29.51 | 28.12 | 27.95 | -1.56 |
| GPT-Base (362M) | 7.2B | Val PPL (โ) | 21.47 | 20.39 | 19.96 | -1.51 |
| GPT-Large (774M) | 15.5B | Val PPL (โ) | 17.56 | 16.99 | 16.52 | -1.04 |
| LLaMA-60M | 1.0B | Val PPL (โ) | 24.98 | 24.59 | 24.60 | -0.38 |
| LLaMA-350M | 7.3B | Val PPL (โ) | 14.03 | 13.44 | 13.44 | -0.59 |
| LLaMA-1B | 20.1B | Val PPL (โ) | 10.62 | 10.42 | 10.21 | -0.41 |
| LLaMA-13B | 26.8B | Val PPL (โ) | 9.24 | 9.11 | 8.71 | -0.53 |
| MoE-7B-A1B | 20.1B | Val PPL (โ) | 10.61 | 9.85 | 9.66 | -0.95 |
Downstream zero-shot and fine-tuned benchmark evaluation on the pre-trained LLaMA-1B models reinforces these gains across diverse language understanding tasks:
| Downstream Benchmark (LLaMA-1B) | Muon (Ns=3) | Muon (Ns=5) | Muon^2 (Ns=3) | Muon^2 (Ns=5) |
|---|---|---|---|---|
| ARC-Challengeโ (Acc %) | 27.08 | 30.01 | 31.91 | 33.48 |
| ARC-Easy (Acc %) | 42.80 | 46.09 | 47.81 | 49.03 |
| OpenBookQA (Acc %) | 32.00 | 33.20 | 31.80 | 35.20 |
| HellaSwag (Acc %) | 44.37 | 49.75 | 50.23 | 52.61 |
| PIQA (Acc %) | 71.22 | 72.25 | 72.20 | 74.05 |
| WinoGrandeโ (Acc %) | 56.75 | 59.91 | 59.88 | 61.90 |
| LAMBADA (Acc %) | 36.52 | 40.99 | 42.50 | 45.08 |
| MMLUโ (Acc %) | 29.75 | 30.71 | 30.79 | 31.31 |
| Unweighted Average (Acc %) | 42.56 | 45.36 | 45.89 | 47.83 |
Ablation Study¶
The paper conducts ablations examining orthogonalization necessity, memory footprint, hyperparameter sensitivity, and alternative Muon variants.
| Configuration | Key Metric (GPT-Small PPL @ Ns=3 / Ns=5) | Memory (LLaMA-7B) | Mechanism Note |
|---|---|---|---|
| Muon^2 (Full Model) | 28.12 / 27.95 | 60.66 GB | Full second-moment preconditioning + polar orthogonalization |
| Muon^2-F (Factorized) | 28.30 / 27.93 | 47.53 GB | Outer-product factorization; minimal memory overhead vs Muon |
| Standard Muon Baseline | 32.70 / 29.51 | 47.29 GB | No preconditioning; suffers severe degradation at Ns=3 (+4.58) |
| Precondition Only (No NS) | 27.14 (LLaMA-60M) | - | Updating directly with preconditioned momentum lags Muon^2 by 2.55 |
| PolarExpress Variant | 30.01 / 29.42 | - | Optimizes polynomial coefficients without fixing ill-conditioned spectrum |
| Turbo-Muon Variant | 29.70 / 29.66 | - | Faster polynomial mapping; still trails Muon^2 significantly at Ns=3 |
| NorMuon Variant | 30.35 / 28.40 | - | Applies post-NS adaptive step sizes; fails to reduce required NS steps |
| AdaMuon Variant | 31.20 / 29.30 | - | Post-orthogonalization coordinate scaling; hypersensitive to learning rate |
Key Findings¶
- 3-step Muon^2 Outperforms 5-step Muon Everywhere: Across all model scales (60M to 13B) and architectures (GPT, LLaMA, MoE), Muon^2 with \(N_s=3\) achieves strictly better validation perplexity than standard Muon with \(N_s=5\) (e.g., 9.11 vs. 9.24 on LLaMA-13B), disproving the assumption that high-accuracy polar approximation requires 5 or more NS iterations.
- 25% Reduction in Pre-training GPU-Hours: Because single-step matrix multiplication cost drops by 40% and element-wise scaling overhead is negligible, reaching an identical validation loss (2.36 on LLaMA-1B) requires 25% fewer training steps, saving 25% wall-clock GPU-hours (from 1042 to 796 hours).
- Extreme Hyperparameter Robustness: Sweeping \(\beta_2 \in [0.90, 0.99]\) and \(\epsilon \in [10^{-10}, 10^{-6}]\) causes validation perplexity variations of less than 0.18, confirming that the benefits stem from the structural spectral transformation rather than fine-tuned hyperparameters.
Highlights & Insights¶
- Root-Cause Spectral Preconditioning: Instead of treating Newton-Schulz polynomial approximation as a black-box mathematical problem, the authors identify that the true obstacle is the ill-conditioned input spectrum. Grounding the solution in the Marchenko-Pastur theorem proves that second-moment division provably eliminates the singular value dead zone.
- Directional Alignment Over Frobenius Orthogonality: The critique of traditional \(\|Q^\top Q - I\|\) error reveals that scalar scaling deviations artificially distort orthogonality error while preserving directional fidelity. Proposing scale-invariant cosine similarity provides an interpretable and reliable metric for iterative matrix algorithms.
- Practical Deployability via Muon^2-F: Adapting Adafactor's rank-1 outer product factorization allows Muon^2-F to capture spectral preconditioning benefits while adding only 0.24 GB memory on a 7B model, making it immediately viable for large-scale production training clusters.
Limitations & Future Work¶
- Scale Evaluation Bound by Academic Budgets: Due to computational budget constraints, the largest model evaluated is 13B parameters on 26.8B tokens. Full-scale validation on frontier-scale models (70B+ parameters trained on trillions of tokens) remains unverified.
- Spectral Bounds on Square Weight Matrices: Proposition 1 requires rectangular matrices (\(\lambda < 1\)). For square layers (\(\lambda = 1\)), the theoretical Marchenko-Pastur lower edge rests at zero, though empirical mass concentration at zero is still absent.
- Further Optimizer State Compression: Investigating 8-bit block-wise quantization or gradient subspace projections (e.g., GaLore) to compress the second-moment state even further represents a promising future research path.
Related Work & Insights¶
- vs Muon (Jordan et al., 2024): Muon pioneered Newton-Schulz polar updates for hidden layers but applied it to raw, ill-conditioned momentum. Muon^2 adds upfront adaptive second-moment preconditioning, cutting NS steps by 40% while improving loss convergence.
- vs PolarExpress (Amsel et al., 2026) & Turbo-Muon (Boissin et al., 2025): Both modify polynomial coefficients to speed up polar convergence on unconditioned inputs. Muon^2 tackles the input spectrum directly, outperforming both methods across all iteration budgets.
- vs NorMuon (Li et al., 2025) & AdaMuon (Si et al., 2025): These variants integrate second moments after the Newton-Schulz step to scale coordinate learning rates, which alters the orthogonal update direction. Muon^2 places second-moment scaling before the NS iteration as an input preconditioner, preserving pure orthogonal update geometry.
Rating¶
- Novelty: โญโญโญโญโญ Establishes a principled synthesis of coordinate-wise second-moment preconditioning and matrix polar orthogonalization, backed by random matrix theory.
- Experimental Thoroughness: โญโญโญโญโญ Evaluated across GPT, LLaMA, and MoE architectures up to 13B parameters, featuring comprehensive learning rate sweeps, wall-clock timing, memory profiling, and downstream suites.
- Writing Quality: โญโญโญโญโญ Rigorous mathematical derivation, clear architectural flow, compelling empirical visualizations, and tight coherence between theory and practice.
- Value: โญโญโญโญโญ An immediate, impactful algorithmic advance for foundation model pre-training, delivering 25% training time savings and better convergence.