Skip to content

Muon^2: Boosting Muon via Adaptive Second-Moment Preconditioning

Conference: EMNLP 2026
Paper: ACL Anthology
Area: Information Retrieval
Keywords: Optimization Algorithms, Foundation Model Pre-training, Orthogonalized Updates, Second-Moment Preconditioning, Newton-Schulz Iteration

TL;DR

Muon^2 introduces Adam-style adaptive second-moment element-wise preconditioning prior to the Newton-Schulz polar orthogonalization step in Muon (together with a memory-efficient factorized variant Muon^2-F), eliminating singular value dead zones to reduce Newton-Schulz iterations by 40% while speeding up pre-training and boosting downstream task performance.

Background & Motivation

The relentless scaling of modern language models in both parameter capacity and pre-training corpora has imposed enormous computational burdens and distributed communication bottlenecks on pre-training infrastructure. Coordinate-wise adaptive first-order optimizers such as Adam and AdamW remain the de facto industry standard due to their empirical robustness and ease of tuning; nevertheless, they flatten multi-dimensional parameter tensors into unstructured coordinate arrays, overlooking the underlying two-dimensional matrix geometry of neural network weights. To surpass coordinate-wise diagonal scaling without paying the prohibitive cost of full Kronecker-factored second-order curvature estimates, matrix-structured optimizers such as Muon have emerged. Muon directly approximates the polar decomposition of the momentum matrix via iterative orthogonalization, effectively mitigating gradient rank collapse and accelerating training dynamics in large architectures.

However, the orthogonalization quality of Muon strictly hinges on the number of Newton-Schulz (NS) polynomial iterations performed per update. Standard Muon requires five iterations to approximate the polar factor, with each step executing high-order matrix polynomial multiplications that induce non-trivial local matrix multiplication (GEMM) workloads and distributed latency bottlenecks. Forcing the iteration count down to three leads to severe degradation in update direction alignment and catastrophic increases in validation perplexity. Prior variants either modify update step-size adaptation (e.g., NorMuon, AdaMuon) or explore alternative polynomial coefficients (e.g., PolarExpress, Turbo-Muon), yet none address the fundamental root cause: the severely ill-conditioned singular value spectrum of the raw input momentum matrix.

A rigorous empirical inspection reveals that during the early and intermediate training regimes, the momentum matrix's singular values span several orders of magnitude, with almost half of them concentrated in an near-zero "dead zone" that fails to converge under low-order polynomial mappings. If the momentum spectrum can be element-wise conditioned prior to orthogonalization, the convergence burden on the Newton-Schulz iteration can be drastically alleviated. Drawing on random matrix theory and empirical spectral analysis, this paper proposes integrating adaptive second-moment scaling as an upfront preconditioner for polar orthogonalization. Core idea: applying coordinate-wise second-moment preconditioning prior to Newton-Schulz orthogonalization reshapes the momentum singular value spectrum and eliminates the dead zone, enabling 40% fewer iteration steps to achieve higher directional fidelity and substantially faster pre-training convergence.

Method

Overall Architecture

Muon^2 retains the structural benefits of matrix orthogonal updates while systematically repairing the ill-conditioned input spectrum. The complete execution pipeline consists of four sequential stages: first, first-order momentum and uncentered second-moment statistics are accumulated from current gradients; second, the momentum is preconditioned via element-wise division by the square root of the second moment, shifting the singular values away from the dead zone into a compact, convergent spectrum; third, the preconditioned matrix enters a streamlined Newton-Schulz polynomial loop (requiring only 3 iterations) to compute a high-fidelity polar approximation; fourth, parameters are updated along the orthogonal direction scaled by an aspect-ratio-aware learning rate. To alleviate the GPU memory overhead of storing a full second-moment tensor, the architecture introduces a low-rank outer-product factorization variant termed Muon^2-F.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Gradient and Momentum Accumulation<br/>Compute Gt and first-order Mt"] --> B["Adaptive Second-Moment Preconditioning<br/>Element-wise scaling to reshape singular spectrum"]
    B --> C["Directional Alignment and Spectral Guarantees<br/>Eliminate dead zone and ensure cosine similarity"]
    C --> D["Accelerated Newton-Schulz Iteration<br/>K=3 polynomial matrix polar approximation"]
    D --> E["Factorized Memory-Efficient Variant Muon^2-F<br/>Row-column outer product optimizer states"]
    E --> F["Orthogonal Parameter Matrix Update<br/>Aspect-ratio-scaled update step to Wt+1"]

Key Designs

1. Adaptive Second-Moment Preconditioning: Eliminating Singular Spectrum Collapse Standard Muon directly feeds the first-order momentum matrix \(M_t = \beta_1 M_{t-1} + (1 - \beta_1) G_t\) into the Newton-Schulz loop. Because gradient variances across parameter coordinates are highly heterogeneous, the unconditioned momentum exhibits a large condition number, causing a significant fraction of singular values to fall into the dead zone (\(10^{-4} \sim 10^{-3}\)). Muon^2 maintains an exponential moving average of squared gradients \(V_t = \beta_2 V_{t-1} + (1 - \beta_2)(G_t \odot G_t)\) and applies element-wise preconditioning: $\(\widetilde{M}_t = M_t \oslash (\sqrt{V_t} + \epsilon I)\)$ This operation standardizes coordinate energy scales across the matrix. Under random matrix theory (the Marchenko-Pastur law), once coordinate-wise scales \(s_{ij}\) are divided out, the empirical singular value distribution of \(\widetilde{M}_t\) converges almost surely to the compact support \([1 - \sqrt{\lambda}, 1 + \sqrt{\lambda}]\) (where \(\lambda = m/n < 1\) is the layer aspect ratio), completely erasing the lower dead zone and providing a pristine spectrum for polynomial mapping.

2. Directional Alignment and Spectral Guarantees: Reconciling True Update Directions Prior studies evaluate polar approximation using exact orthogonality error \(\|Q^\top Q - I\|\). However, this metric is overly sensitive to any global scalar scaling factor \(c \in (0, 1]\): even when \(Q = c U V^\top\) is perfectly aligned with the exact polar factor \(Q^\star = U V^\top\), the orthogonality error evaluates to \(|c^2 - 1| \cdot \|I\|\). In deep learning optimization, scalar constants are absorbed into the tunable learning rate \(\eta\). Muon^2 therefore formalizes directional cosine similarity as the proper evaluation criterion: $\(\cos(Q_{\text{NS}}, Q^\star) = \frac{\sum_i \phi_K(\sigma_i)}{\sqrt{n} \sqrt{\sum_i \phi_K(\sigma_i)^2}}\)$ Under this scale-invariant formulation, the convergence of polynomial mapping \(\phi(x) = ax + bx^3 + cx^5\) (\(a=3.4445, b=-4.7750, c=2.0315\)) is governed purely by relative spectral concentration. Because preconditioning raises the effective rank from 48.9 to 142.5 and cuts the dead-zone fraction from 26% to 6%, Muon^2 with only 3 steps attains a cosine similarity of 0.916 (exceeding standard Muon's 5-step score of 0.902~0.931), reaching 0.975 at 5 steps.

3. Accelerated Newton-Schulz Iteration: Superior Optimization at Reduced Cost With a well-conditioned input spectrum guaranteed, the preconditioned matrix \(\widetilde{M}_t\) is processed through \(K=3\) Newton-Schulz iterations: $\(O_t = \text{NewtonSchulz}(\widetilde{M}_t, K=3)\)$ Reducing \(K\) from 5 to 3 eliminates 40% of the costly matrix-matrix multiplication FLOPs and latency-inducing synchronization barriers per step. The parameters are then updated using the dimension-aware scaling factor: $\(W_{t+1} = W_t - \eta \sqrt{\frac{m}{n}} O_t\)$ Empirically, 3-step Muon^2 yields lower pre-training perplexity than 5-step Muon across all scales, proving that preconditioning the spectrum is fundamentally superior to executing extra iterations on an ill-conditioned input.

4. Factorized Memory-Efficient Variant Muon^2-F: Outer-Product State Compression To eliminate the GPU memory overhead of maintaining the full \(n \times m\) matrix \(V_t\), Muon^2-F adopts an Adafactor-style row- and column-wise factorized second-moment accumulator: $\(r_t = \beta_2 r_{t-1} + (1 - \beta_2) \text{sum}_{\text{col}}(G_t \odot G_t), \quad c_t = \beta_2 c_{t-1} + (1 - \beta_2) \text{sum}_{\text{row}}(G_t \odot G_t)\)$ $\(\widehat{V}_t = \frac{r_t c_t^\top}{\text{sum}(r_t)}\)$ This reduces the auxiliary optimizer state memory footprint from \(\mathcal{O}(nm)\) to \(\mathcal{O}(n + m)\). Random matrix analysis demonstrates that as long as coordinate variances admit row-column separable structure, the effective rank and non-zero singular value margins are fully preserved. On a 7B model, Muon^2-F adds only 0.24 GB over standard Muon while retaining over 95% of the perplexity gains and the entire 40% NS reduction of full Muon^2.

Loss & Training

The optimizer is evaluated using standard autoregressive causal language modeling cross-entropy loss: $\(\mathcal{L}(W) = -\frac{1}{T} \sum_{t=1}^T \log P(w_t \mid w_{<t})\)$ Decoupled weight decay is applied directly to weights. Hyperparameters default to second-moment decay \(\beta_2 = 0.95\), numerical stabilizer \(\epsilon = 10^{-8}\), and first-order momentum \(\beta_1 = 0.95\). Non-2D parameters (embedding tables and 1D normalization vectors) are updated via standard AdamW.

Key Experimental Results

Main Results

Muon^2 was evaluated across GPT models (Small 124M, Base 362M, Large 774M), LLaMA models (60M, 350M, 1B, 13B), and a sparse Mixture-of-Experts architecture (MoE-7B-A1B).

Model & Scale Tokens Metric Muon (Ns=5) Muon^2 (Ns=3) Muon^2 (Ns=5) Gain (Ns=5 vs Muon)
GPT-Small (124M) 3.0B Val PPL (โ†“) 29.51 28.12 27.95 -1.56
GPT-Base (362M) 7.2B Val PPL (โ†“) 21.47 20.39 19.96 -1.51
GPT-Large (774M) 15.5B Val PPL (โ†“) 17.56 16.99 16.52 -1.04
LLaMA-60M 1.0B Val PPL (โ†“) 24.98 24.59 24.60 -0.38
LLaMA-350M 7.3B Val PPL (โ†“) 14.03 13.44 13.44 -0.59
LLaMA-1B 20.1B Val PPL (โ†“) 10.62 10.42 10.21 -0.41
LLaMA-13B 26.8B Val PPL (โ†“) 9.24 9.11 8.71 -0.53
MoE-7B-A1B 20.1B Val PPL (โ†“) 10.61 9.85 9.66 -0.95

Downstream zero-shot and fine-tuned benchmark evaluation on the pre-trained LLaMA-1B models reinforces these gains across diverse language understanding tasks:

Downstream Benchmark (LLaMA-1B) Muon (Ns=3) Muon (Ns=5) Muon^2 (Ns=3) Muon^2 (Ns=5)
ARC-Challengeโ€  (Acc %) 27.08 30.01 31.91 33.48
ARC-Easy (Acc %) 42.80 46.09 47.81 49.03
OpenBookQA (Acc %) 32.00 33.20 31.80 35.20
HellaSwag (Acc %) 44.37 49.75 50.23 52.61
PIQA (Acc %) 71.22 72.25 72.20 74.05
WinoGrandeโ€  (Acc %) 56.75 59.91 59.88 61.90
LAMBADA (Acc %) 36.52 40.99 42.50 45.08
MMLUโ€  (Acc %) 29.75 30.71 30.79 31.31
Unweighted Average (Acc %) 42.56 45.36 45.89 47.83

Ablation Study

The paper conducts ablations examining orthogonalization necessity, memory footprint, hyperparameter sensitivity, and alternative Muon variants.

Configuration Key Metric (GPT-Small PPL @ Ns=3 / Ns=5) Memory (LLaMA-7B) Mechanism Note
Muon^2 (Full Model) 28.12 / 27.95 60.66 GB Full second-moment preconditioning + polar orthogonalization
Muon^2-F (Factorized) 28.30 / 27.93 47.53 GB Outer-product factorization; minimal memory overhead vs Muon
Standard Muon Baseline 32.70 / 29.51 47.29 GB No preconditioning; suffers severe degradation at Ns=3 (+4.58)
Precondition Only (No NS) 27.14 (LLaMA-60M) - Updating directly with preconditioned momentum lags Muon^2 by 2.55
PolarExpress Variant 30.01 / 29.42 - Optimizes polynomial coefficients without fixing ill-conditioned spectrum
Turbo-Muon Variant 29.70 / 29.66 - Faster polynomial mapping; still trails Muon^2 significantly at Ns=3
NorMuon Variant 30.35 / 28.40 - Applies post-NS adaptive step sizes; fails to reduce required NS steps
AdaMuon Variant 31.20 / 29.30 - Post-orthogonalization coordinate scaling; hypersensitive to learning rate

Key Findings

  • 3-step Muon^2 Outperforms 5-step Muon Everywhere: Across all model scales (60M to 13B) and architectures (GPT, LLaMA, MoE), Muon^2 with \(N_s=3\) achieves strictly better validation perplexity than standard Muon with \(N_s=5\) (e.g., 9.11 vs. 9.24 on LLaMA-13B), disproving the assumption that high-accuracy polar approximation requires 5 or more NS iterations.
  • 25% Reduction in Pre-training GPU-Hours: Because single-step matrix multiplication cost drops by 40% and element-wise scaling overhead is negligible, reaching an identical validation loss (2.36 on LLaMA-1B) requires 25% fewer training steps, saving 25% wall-clock GPU-hours (from 1042 to 796 hours).
  • Extreme Hyperparameter Robustness: Sweeping \(\beta_2 \in [0.90, 0.99]\) and \(\epsilon \in [10^{-10}, 10^{-6}]\) causes validation perplexity variations of less than 0.18, confirming that the benefits stem from the structural spectral transformation rather than fine-tuned hyperparameters.

Highlights & Insights

  • Root-Cause Spectral Preconditioning: Instead of treating Newton-Schulz polynomial approximation as a black-box mathematical problem, the authors identify that the true obstacle is the ill-conditioned input spectrum. Grounding the solution in the Marchenko-Pastur theorem proves that second-moment division provably eliminates the singular value dead zone.
  • Directional Alignment Over Frobenius Orthogonality: The critique of traditional \(\|Q^\top Q - I\|\) error reveals that scalar scaling deviations artificially distort orthogonality error while preserving directional fidelity. Proposing scale-invariant cosine similarity provides an interpretable and reliable metric for iterative matrix algorithms.
  • Practical Deployability via Muon^2-F: Adapting Adafactor's rank-1 outer product factorization allows Muon^2-F to capture spectral preconditioning benefits while adding only 0.24 GB memory on a 7B model, making it immediately viable for large-scale production training clusters.

Limitations & Future Work

  • Scale Evaluation Bound by Academic Budgets: Due to computational budget constraints, the largest model evaluated is 13B parameters on 26.8B tokens. Full-scale validation on frontier-scale models (70B+ parameters trained on trillions of tokens) remains unverified.
  • Spectral Bounds on Square Weight Matrices: Proposition 1 requires rectangular matrices (\(\lambda < 1\)). For square layers (\(\lambda = 1\)), the theoretical Marchenko-Pastur lower edge rests at zero, though empirical mass concentration at zero is still absent.
  • Further Optimizer State Compression: Investigating 8-bit block-wise quantization or gradient subspace projections (e.g., GaLore) to compress the second-moment state even further represents a promising future research path.
  • vs Muon (Jordan et al., 2024): Muon pioneered Newton-Schulz polar updates for hidden layers but applied it to raw, ill-conditioned momentum. Muon^2 adds upfront adaptive second-moment preconditioning, cutting NS steps by 40% while improving loss convergence.
  • vs PolarExpress (Amsel et al., 2026) & Turbo-Muon (Boissin et al., 2025): Both modify polynomial coefficients to speed up polar convergence on unconditioned inputs. Muon^2 tackles the input spectrum directly, outperforming both methods across all iteration budgets.
  • vs NorMuon (Li et al., 2025) & AdaMuon (Si et al., 2025): These variants integrate second moments after the Newton-Schulz step to scale coordinate learning rates, which alters the orthogonal update direction. Muon^2 places second-moment scaling before the NS iteration as an input preconditioner, preserving pure orthogonal update geometry.

Rating

  • Novelty: โญโญโญโญโญ Establishes a principled synthesis of coordinate-wise second-moment preconditioning and matrix polar orthogonalization, backed by random matrix theory.
  • Experimental Thoroughness: โญโญโญโญโญ Evaluated across GPT, LLaMA, and MoE architectures up to 13B parameters, featuring comprehensive learning rate sweeps, wall-clock timing, memory profiling, and downstream suites.
  • Writing Quality: โญโญโญโญโญ Rigorous mathematical derivation, clear architectural flow, compelling empirical visualizations, and tight coherence between theory and practice.
  • Value: โญโญโญโญโญ An immediate, impactful algorithmic advance for foundation model pre-training, delivering 25% training time savings and better convergence.