Skip to content

Muon^2: Boosting Muon via Adaptive Second-Moment Preconditioning

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM (Other)
Keywords: LLM pretraining, Optimizer, MUON, Matrix orthogonalization, Second-moment preconditioning

TL;DR

Addressing the severe computational and communication overhead in MUON caused by the ill-conditioned momentum spectrum during Newton-Schulz iterations, MUON2 applies Adam-style adaptive second-moment preconditioning prior to orthogonalization, achieving superior directional alignment with 40% fewer iterations and cutting pretraining wall-clock time by 25%.

Background & Motivation

In large-scale foundation language model pretraining, exponential scaling of model parameters and dataset volumes has elevated optimization efficiency and convergence stability to paramount bottlenecks. While coordinate-wise adaptive optimizers like Adam and AdamW remain the de facto standards owing to their robust coordinate-wise learning rate adjustments, they fundamentally discard the underlying low-rank and spectral matrix geometry inherent to neural network weight tensors. Recently, the MUON optimizer emerged as a transformative breakthrough: by approximating a polar decomposition of the first-order momentum matrix via iterative Newton-Schulz (NS) polynomial updates, MUON enforces orthogonalized parameter updates, successfully mitigating gradient rank collapse and substantially improving training dynamics across several frontier foundation models.

However, the real-world efficiency of MUON is bottlenecked by the approximation fidelity of its polar decomposition, which hinges strictly on the number of Newton-Schulz iteration steps executed per parameter update. Standard implementations execute 5 NS iterations per step, requiring repeated full-matrix multiplications and inter-accelerator communication across distributed devices, incurring non-trivial latency and memory-bandwidth overhead. Existing endeavors to refine MUON predominantly focus on adjusting outer step sizes or discovering higher-order orthogonal polynomials. Yet they circumvent the core mathematical root: what makes polar orthogonalization computationally hard? Empirical spectral analysis reveals that raw momentum matrices during foundation model pretraining are severely ill-conditioned, featuring singular values spanning several orders of magnitude with high concentrations near zero. This ill-conditioned spectrum severely stalls the convergence of Newton-Schulz iterations.

This paper tackles the challenge from the input conditioning perspective: rather than forcing sophisticated polar iterations over an ill-conditioned momentum matrix, it reshapes the momentum spectrum prior to orthogonalization. Core idea: Apply element-wise Adam-style adaptive second-moment preconditioning to the momentum matrix before feeding it into the Newton-Schulz iterations, fundamentally repairing the singular value spectrum and shrinking the spectral dead zone, thereby outperforming 5-step standard MUON with only 3 iteration steps and accelerating training throughput.

Method

Overall Architecture

MUON2 follows a two-stage optimization pipeline: it first tracks running averages of the first-order gradient momentum and the coordinate-wise uncentered second moment, dividing the momentum matrix element-wise by the square root of the second-moment accumulator. The resulting preconditioned matrix is passed into a truncated Newton-Schulz iteration (\(N_s=3\)) to compute the approximate polar factor, which subsequently scales the parameter update. To address distributed memory overhead, the authors also provide a factorized variant, MUON2-F, which compresses second moments into marginal row and column vectors. The end-to-end workflow is illustrated below:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Gradient G_t & Weights W_t"] --> B["First & Second Moment Accumulation<br/>EMA updates of M_t and V_t"]
    B --> C["Adaptive Second-Moment Preconditioning<br/>Element-wise spectral normalization"]
    C --> D["Polar Orthogonalization<br/>K-step Newton-Schulz iteration (Ns=3)"]
    D --> E["Aspect-Ratio Scaled Update<br/>Parameter update with dimension factor"]

Key Designs

1. Adaptive Second-Moment Preconditioning: Eliminating Momentum Spectrum Collapse and Dead Zones

In vanilla MUON, the momentum matrix \(M_t = \beta_1 M_{t-1} + (1-\beta_1) G_t\) exhibits severe spectral decay during early pretraining and within deep layers, with a substantial fraction of singular values compressed into the \(10^{-4} \sim 10^{-3}\) interval. Under the standard quintic Newton-Schulz mapping \(\phi(x) = ax + bx^3 + cx^5\), these near-zero values lie within an intractable "Dead Zone" where finite polynomial iterations fail to push them into the targeted near-orthogonal singular interval. MUON2 tracks the second moment \(V_t = \beta_2 V_{t-1} + (1-\beta_2)(G_t \odot G_t)\) and preconditions the momentum matrix via element-wise division:

\[\widetilde{M}_t = M_t \oslash (\sqrt{V_t} + \epsilon \mathbf{1})\]

This operation effectively cancels coordinate-wise variance disparities across tensor dimensions. From a random matrix theory perspective, modeling momentum as \(M_{ij} = s_{ij} \xi_{ij}\) with deterministic scales \(s_{ij}\) and i.i.d. zero-mean unit-variance noise \(\xi_{ij}\), element-wise division cancels \(s_{ij}\), causing the empirical singular value distribution of \(\widetilde{M}_t / \sqrt{n}\) to converge almost surely to the Marchenko-Pastur law supported on \([1-\sqrt{\lambda}, 1+\sqrt{\lambda}]\) (where \(\lambda = m/n < 1\)). This spectral transformation drastically contracts the dead-zone fraction from 26% down to 6%, triples the effective spectral rank, and provides a well-conditioned baseline for rapid orthogonalization.

2. Directional Alignment Criterion: Transcending Rigid Orthogonality Metrics

Prior theoretical studies on polar approximation have traditionally relied on the matrix orthogonality error \(\|Q^\top Q - I\|\). However, this paper demonstrates that such a metric is fundamentally misleading in neural network optimization: if all singular values are mapped to an exact constant \(c \in (0, 1]\), the resulting matrix remains collinear with the exact polar factor \(UV^\top\), yet the orthogonality error explodes to \(|c^2 - 1| \cdot \|I\|\). In SGD-based optimization, any scalar factor \(c\) is seamlessly absorbed into the tuned step size \(\eta\). Consequently, MUON2 advocates measuring directional alignment via Frobenius inner-product cosine similarity:

\[\cos(Q, Q^\star) = \frac{\langle Q, Q^\star \rangle_F}{\|Q\|_F \|Q^\star\|_F} = \frac{\sum_i \phi^K(\sigma_i)}{\sqrt{n \sum_i [\phi^K(\sigma_i)]^2}}\]

Under this criterion, because the preconditioned singular values are concentrated away from zero into the active transition zone, MUON2 with merely \(K=3\) Newton-Schulz steps attains a directional alignment of 0.916, surpassing 5-step vanilla MUON (0.931); pushed to \(K=5\), MUON2 reaches an exceptional 0.975 alignment. This proves that preconditioning enables higher update directional fidelity with 40% fewer iterations.

3. Factorized Second Moments (MUON2-F): Negligible Footprint Preserving Preconditioning Gains

Maintaining a full second-moment tensor \(V_t \in \mathbb{R}^{n \times m}\) doubles optimizer state memory, which presents non-trivial pressure when scaling to multi-billion parameter foundation models. Since the second moment in MUON2 serves solely to condition the spectrum of the input matrix rather than directly dictating the exact coordinate-wise update magnitude, optimization dynamics exhibit remarkable tolerance to minor approximation errors in \(V_t\). Inspired by Adafactor, MUON2-F decomposes the squared gradient tensor \(R_t = G_t \odot G_t\) into rank-1 row and column marginal accumulators:

\[r_t = \beta_2 r_{t-1} + (1-\beta_2)\sum\nolimits_{\text{col}}(R_t), \quad c_t = \beta_2 c_{t-1} + (1-\beta_2)\sum\nolimits_{\text{row}}(R_t), \quad \widehat{V}_t = \frac{r_t c_t^\top}{\sum_i r_{t,i}}\]

This design slashes auxiliary optimizer state memory from \(O(nm)\) to \(O(n+m)\), rendering memory overhead negligible across distributed clusters. Theoretically, when entry-wise scales factorize as \(s_{ij} = r_i c_j\), the factorized preconditioned matrix still adheres to the Marchenko-Pastur spectral concentration guarantee. Empirically, MUON2-F maintains over 95% of the perplexity gains and iteration savings of full MUON2 while keeping peak GPU memory identical to vanilla MUON.

Loss & Training

Following preconditioning, the normalized matrix undergoes \(K\) steps of Newton-Schulz polynomial mapping using standard loose-target coefficients \((a, b, c) = (3.4445, -4.7750, 2.0315)\). The final parameter update integrates the dimension-aware step size scaling factor:

\[W_{t+1} = W_t - \eta \sqrt{\frac{m}{n}} O_t\]

Hyperparameter sensitivity analyses demonstrate that MUON2 is exceptionally robust: setting the default second-moment decay \(\beta_2 = 0.95\) and stability epsilon \(\epsilon = 10^{-8}\) yields consistent improvements across scales. Because Frobenius normalization washes out global scaling, varying \(\epsilon\) from \(10^{-10}\) to \(10^{-6}\) affects validation perplexity by less than 0.15 PPL, removing the need for tedious per-model hyperparameter tuning.

Key Experimental Results

Main Results

The authors evaluated MUON2 across GPT, LLaMA, and Mixture-of-Experts (MoE) architectures spanning 60M to 13B parameters, adopting compute-optimal training regimes on FineWeb and C4 datasets. Across all architectures and scales, MUON2 consistently outperformed vanilla MUON while reducing required Newton-Schulz steps from 5 to 3:

Model & Scale NS Steps (\(N_s\)) MUON Baseline PPL MUON2 (Ours) PPL MUON2-F (Ours) PPL Relative PPL Drop (MUON2)
GPT-Small (124M) \(N_s = 3\) 32.70 28.12 28.30 -4.58
GPT-Small (124M) \(N_s = 5\) 29.51 27.95 27.93 -1.56
GPT-Base (362M) \(N_s = 3\) 24.69 20.39 21.17 -4.30
GPT-Base (362M) \(N_s = 5\) 21.47 19.96 20.58 -1.51
GPT-Large (774M) \(N_s = 3\) 21.13 16.99 17.69 -4.14
GPT-Large (774M) \(N_s = 5\) 17.56 16.52 16.55 -1.04
LLaMA-60M \(N_s = 3\) 26.37 24.59 24.68 -1.78
LLaMA-60M \(N_s = 5\) 24.98 24.60 24.66 -0.38
LLaMA-350M \(N_s = 3\) 14.91 13.44 13.55 -1.47
LLaMA-350M \(N_s = 5\) 14.03 13.46 13.44 -0.57
LLaMA-1B \(N_s = 3\) 11.63 10.42 10.49 -1.21
LLaMA-1B \(N_s = 5\) 10.62 10.21 10.21 -0.41
LLaMA-13B \(N_s = 3\) 10.54 9.11 โ€” -1.43
LLaMA-13B \(N_s = 5\) 9.24 8.71 โ€” -0.53
MoE-7B-A1B \(N_s = 3\) 11.93 9.85 โ€” -2.08
MoE-7B-A1B \(N_s = 5\) 10.61 9.66 โ€” -0.95

Downstream zero-shot and fine-tuned evaluations on LLaMA-1B (across ARC-Challenge, OpenBookQA, HellaSwag, PIQA, WinoGrande, LAMBADA, and MMLU) mirrored these perplexity gains: MUON2 (\(N_s=3\)) attained an average score of 45.89%, outperforming MUON (\(N_s=5\)) at 45.36%, while MUON2 (\(N_s=5\)) advanced performance to 47.83% (+2.47% over baseline).

Ablation Study

To isolate the mechanism behind second-moment preconditioning and contrast against contemporary MUON variants, the authors conducted structural ablations on LLaMA-60M and comparative benchmarking on GPT-Small/Base:

Configuration / Variant Model Architecture Metric (Val. PPL) GPU Memory (1B) Core Mechanism & Analysis
MUON2 (Full Model) LLaMA-60M (\(N_s=3\)) 24.59 20.36 GB Combined preconditioning and orthogonalization
w/o Orthogonalization (No NS) LLaMA-60M (Direct Update) 27.14 20.36 GB Preconditioned momentum alone degrades by +2.55 PPL, proving gain is not merely Adam
MUON2-F (Factorized) LLaMA-60M (\(N_s=3\)) 24.68 18.57 GB Matches vanilla MUON memory while trailing full MUON2 by only 0.09 PPL
Vanilla MUON Baseline LLaMA-60M (\(N_s=3\)) 26.37 18.56 GB Severe polar truncation errors due to ill-conditioned spectrum
PolarExpress (Amsel et al.) GPT-Small (\(N_s=3\)) 30.01 (+1.89) 18.56 GB Modifies polynomial coefficients only; leaves input ill-conditioning unresolved
Turbo-Muon (Boissin et al.) GPT-Small (\(N_s=3\)) 29.70 (+1.58) 18.56 GB Heuristic acceleration still suffers from near-zero singular dead zones at low steps
NorMuon (Li et al.) GPT-Small (\(N_s=3\)) 30.35 (+2.23) ~20.3 GB Applies second moments post-orthogonalization; cannot reduce NS iterations
AdaMuon (Si et al.) GPT-Small (\(N_s=3\)) 31.20 (+3.08) ~20.3 GB Modifies update directions directly; highly sensitive to learning rate under low steps

Key Findings

  • 3-Step MUON2 Systematically Outperforms 5-Step MUON: Across all scales, 3-step MUON2 surpassed 5-step vanilla MUON (e.g., GPT-Base PPL 20.39 vs. 21.47). Preconditioning not only stabilizes the spectrum but enriches the directional trajectory with historical curvature information, yielding superior optimization geometry even with fewer iterations.
  • End-to-End Compute Acceleration: Pretraining LLaMA-1B on 64 supercomputing nodes (256 GPUs) to achieve a cross-entropy loss of 2.36 required 1042 GPU-hours with vanilla MUON, whereas MUON2 reached the same milestone in 796 GPU-hoursโ€”a 25% wall-clock reduction. The per-step computation latency rose by less than 0.5% (2971ms vs. 2958ms), demonstrating immense wall-clock efficiency.
  • Spectrum Reshaping as the True Driver: Discarding Newton-Schulz iterations and directly updating weights with preconditioned momentum resulted in severe degradation (27.14 PPL), underperforming even vanilla MUON. This decisively proves that MUON2's performance leap does not stem from coordinate-wise Adam mechanics, but specifically from supplying a well-conditioned matrix to the polar orthogonalizer.

Highlights & Insights

  • Rethinking Orthogonality Metrics: The paper establishes that rigid matrix deviation \(\|Q^\top Q - I\|\) can be deceptive in deep learning optimization due to learning-rate scale absorption. Introducing directional cosine alignment aligns optimization theory more accurately with empirical training realities.
  • Pareto-Optimal Optimization: While classical optimizer research often forces a trade-off between computational budget and convergence quality, MUON2 achieves both simultaneously: a 40% reduction in polar matrix iterations coupled with consistently superior empirical loss.
  • Orthogonal to Advanced Polar Polynomials: The preconditioning mechanism operates upstream of the iteration scheme. As confirmed in Appendix C, it seamlessly integrates with modern polar schemes like PolarExpress, universally accelerating convergence across diverse polynomial choices.

Limitations & Future Work

  • Frontier Industrial Scale Verification: Constrained by academic computational budgets, current empirical validations cap at 13B parameters and 27B tokens. Evaluating MUON2 on production-grade frontier models (e.g., 70B+ dense models or MoEs with tens of billions of active parameters) remains open.
  • Extended Memory Compression Exploration: Beyond Adafactor-style factorized statistics, future work could investigate 8-bit block-wise optimizer quantization or low-rank gradient projection (e.g., GaLore-style integration) to push memory efficiency even further.
  • vs. Vanilla MUON: MUON iterates over raw ill-conditioned momentum with 5 expensive NS steps, heavily penalized by near-zero singular values. MUON2 applies second-moment preconditioning beforehand, eliminating the dead zone and outperforming 5-step MUON in just 3 steps while cutting 25% training time.
  • vs. PolarExpress & Turbo-Muon: These approaches optimize the polar polynomial mapping itself while continuing to process raw ill-conditioned matrices. MUON2 addresses the root cause by reshaping input singular values, providing complementary and stackable benefits.
  • vs. NorMuon & AdaMuon: These methods introduce second moments after orthogonalization to rescale parameter update step sizes (acting as Adam-like post-processors), compromising pure orthogonalization geometry and failing to decrease NS iteration requirements. MUON2 preserves the clean geometry of orthogonal updates while slashing orthogonalization costs.

Rating

  • Novelty: โญโญโญโญโญ Formulates an innovative paradigm by solving polar approximation bottlenecks through pre-orthogonalization spectral reshaping.
  • Experimental Thoroughness: โญโญโญโญโญ Rigorously tested across GPT, LLaMA, and MoE architectures up to 13B parameters, backed by downstream evals and random matrix theory proofs.
  • Writing Quality: โญโญโญโญโญ Flawless narrative structure, deep mathematical rigor, and compelling geometric intuition.
  • Value: โญโญโญโญโญ An immediately actionable, drop-in optimizer enhancement offering 25% GPU-hour savings for LLM pretraining pipelines.